Adaptive Schemas: Letting the Data Model Evolve with the Work

Beyond the Chat Window · Article 4 of 8

The architecture developed so far has a conservative flavor: externalize the state, define its structure, govern mutations, commit progress, and resume from checkpoints. Each move reduces the workflow's dependence on ephemeral conversational context by giving important parts of the project a durable representation.

There is a risk, however, in carrying the software analogy too far. Open-ended knowledge work often begins before its ontology is fully known. The most useful types, relations, and constraints may emerge only after the work has started.

If the schema is treated as fixed infrastructure, it can become a model of yesterday's understanding rather than today's task.

This is where the inferential flexibility of the LLM becomes an architectural advantage rather than merely a generation capability. The model can recognize recurring semantic distinctions while it works, propose ways to formalize them, and help migrate existing objects into a richer representation. The goal is not loose structure. The goal is progressive formalization: start with enough structure to work reliably, then make the representation more precise as the task reveals what precision is useful. The development of this series is a concrete instance of that process.

The ontology emerged through use The first schema for the project was relatively simple. It knew that there would be five articles. It stored a series thesis, article purposes, dependencies, terminology, and status. That was enough for the initial planning problem.

Then I sharpened the objective. The series was not merely supposed to cover five topics; it was supposed to function as an instructive argument. Each article needed to begin from a plausible reader state and leave the reader justified in accepting a particular conclusion. That requirement exposed distinctions the original schema did not represent.

“Topic” and “purpose” were no longer enough. The model needed objects for prior beliefs, target beliefs, inferential moves, and the support required to license those moves.

Later, a second refinement made the same point more strongly. I asked for all five posts to be formalized one level deeper. The state acquired premises, warrants, objections, examples, and explicit belief transitions. Those were not decorative metadata. They changed what could be checked. A conclusion could now be tested against the path intended to justify it; an objection could be attached to the precise move it challenged. The schema had not failed. The project had learned what distinctions mattered.

This is common in difficult intellectual work. A research workflow may begin with generic “notes” and later need to distinguish source claims from derived claims. A software design process may begin with “requirements” and later discover that some are invariants while others are preferences. A curriculum model may begin with topics and later discover that prerequisite relations deserve first-class representation. The useful ontology is often discovered by doing the work.

An LLM can help because it is able to operate while the representation is still partly informal. It can notice that two recurring categories are being treated differently in practice and propose making the distinction explicit. Once formalized, conventional state-management rules can take over.

Flexibility without drift

Adaptive structure is useful only if change itself is disciplined. Otherwise every turn can invent a new field, rename an old relation, or quietly alter the meaning of a type. A schema that evolves without governance is not adaptive; it is unstable.

The protocol therefore treats structural change as a first-class event. A proposed schema extension should have a reason, an affected scope, a migration plan where necessary, and a lifecycle status. New constructs can begin as provisional rather than becoming permanent immediately. If they recur and improve the work, they can become stable.

If a stable construct is later superseded, it can be deprecated without erasing the history that older artifacts depend on.

Stable object identity is what makes this practical. Suppose the workflow introduces a new field called provenance_kind because it has become useful to distinguish source claims from synthesized claims. If existing claim objects already have persistent IDs, the model can backfill the new field selectively. The project does not have to be reinterpreted from an undifferentiated transcript.

The same is true for relations. A new supersedes edge can be introduced between existing objects without changing their identities. A new affected_by relation can connect a late requirement to the artifacts that require review.

Structure becomes richer while continuity is preserved.

This is also why the protocol now distinguishes recomposition from line revision. When I rejected the first prose as choppy and outline-like, the underlying argument objects were not automatically discarded. The flaw was localized to the transformation from argument state into prose. The protocol was extended with a composition orchestrator, helper roles, and adversarial editorial review. The project schema learned a new class of state - composition state - because the work demonstrated that it was necessary. That is progressive formalization at the process level as well as the content level.

A late idea as a live schema migration The clearest example arrived after much of the structural work was already complete. I suggested that the article series itself should become the central example of the method. The reader should not merely be told how persistent state, protocols, resumability, schema evolution, graphs, and provenance might work; each article should explain how that layer had actually been used to produce the series the reader was holding.

That requirement cut across every article. Article 1 now needed to show how the project had been externalized into state. Article 2 needed to show protocol changes becoming authoritative. Article 3 needed to use the project's interruptions and resumptions as evidence. Article 4 needed to treat the late idea itself as a schema-evolution event.

Article 5 needed to expose the dependency and change history as provenance.

A rigid workflow might have interpreted this as a reason to return to the beginning and redesign the entire series.

Instead, the requirement was added to the project model as a new central_demonstration object. A change record captured its origin, affected scope, and resolution. Each article received a specific obligation describing how the running example should appear in that article.

Crucially, the existing argument structure remained valid. The new requirement changed how examples were selected and how the prose would demonstrate the method, but it did not invalidate the central conclusions already assigned to the articles. The system therefore migrated only the affected layer.

This is the practical value of an adaptive schema. Late insight does not have to be treated as damage. If the project state is explicit enough, a new idea can be represented, its dependents can be identified, and the necessary transformations can be rerun without erasing unrelated work.

The current revision offers an even stronger example. I later judged the prose itself unacceptable. Rather than patching sentences indefinitely, the protocol was changed again: argument objects would no longer map directly to paragraph roles; a new composition layer would synthesize several structured objects into larger rhetorical movements; two independent revision agents would submit competing paragraph-level proposals; and a strict editor would referee them. That change did not require rediscovering what the articles were supposed to argue. It required improving how the established argument was materialized.

Progressive formalization as a design principle

It would be easy to describe all of this as weak typing, but that misses the direction of travel. The method is not trying to preserve ambiguity. It is trying to formalize the task at the moment formalization becomes useful.

Early in a project, an object may only need to be a “note.” Later, the workflow may discover that some notes are evidence, some are claims, some are constraints, and some are unresolved questions. Later still, the relations among those objects may become important enough to type explicitly. The state becomes more structured as understanding improves.

This is a particularly natural fit for LLMs because semantic inference can bridge the period between informal and formal representation. The model can work with partially specified concepts, detect patterns, and propose structure.

Once structure exists, deterministic operations can exploit it. The protocol controls the handoff between those modes.

The result combines advantages that are often treated as opposites. Persistent state gives continuity, stable identity, and inspectability. Formal relations give validation and analysis. LLM inference provides semantic flexibility when the task is not fully known in advance. Schema evolution allows the representational system to improve without repeatedly starting over.

By this point, however, the project contains more than a hierarchy of JSON objects. Articles depend on prior claims.

Claims are supported by examples. Change events affect multiple artifacts. Decisions supersede earlier rules.

Provenance links outputs back to transformations and sources. Once those relations are explicit, the state can be interpreted as a graph. That shift matters because graphs are not merely easier to visualize. They can be analyzed formally. The final article turns to that analytical layer and to the stronger kind of auditability that provenance makes possible.