Chapter 9 of 9 · 9 min read
Moving untyped fields into a shared file reproduces every handoff bug
On this page
The design document is four pages and the proposal is one sentence: every agent in the nine-agent underwriting system coordinates only through a single shared state.json, with no direct messages between agents at all. Two arguments are offered for it. The first is that on-call cannot currently reconstruct a failed run without correlating nine services' logs, which is true and is the argument chapter 6 was about. The second is the one to look at closely, because the document leans on it hardest: three incidents last quarter were caused by lossy handoff payloads, and moving to shared state removes handoff payloads.
Here are the three incidents as the postmortems recorded them, a postmortem being the write-up an on-call team produces after an incident to state what broke and why.
| Incident | What happened | Root cause as written |
|---|---|---|
| INC-441 | Renewal deadlines computed from parse time rather than signature date; 14 policies misfiled, two days to unwind | deadline carried as "30 days" by one producer and as an ISO date by another; consumer parsed whichever it got |
| INC-452 | 31 findings dropped from a risk summary | severity was free text; producer emitted "High" and "critical", consumer filtered on severity == "high" |
| INC-467 | Documents truncated to their first page on the OCR fallback path | pages omitted from the payload when the fallback ran; consumer defaulted the missing field to 1 |
Not one of those is a property of message passing. Each is a field with no typed constraint, a producer that wrote something reasonable, and a consumer that interpreted it differently. Put the same three fields into state.json and INC-441 is a document where deadline is sometimes "30 days", INC-452 is a document where severity is sometimes "High", and INC-467 is a document where pages is sometimes absent. The transport changed and the defect did not, because the defect was never in the transport.
The substitution, and why it is convincing
The proposal is doing something specific that is worth being able to name in a review: it answers a schema problem with a topology change, where topology means the shape of who exchanges what with whom — nine agents messaging each other, or nine agents reading and writing one file. Both are real categories of fix, and the topology change is the more visible one, so it absorbs credit for problems it does not touch. It is convincing because the three incidents genuinely did occur at handoff boundaries, and a boundary that no longer exists cannot host an incident — except that the boundary between a producer's meaning and a consumer's interpretation is not the handoff, and it survives the reorganisation intact.
The strongest argument in the proposal's favour is the one chapter 8 measured rather than either of the two the document makes: in a nine-agent run, a crashed worker currently takes the run's state with it, and shared state is what lets a replacement resume from the last recorded transition. Nine agents is enough that a worker dying somewhere in a run is a weekly event rather than a quarterly one. The readability argument is real too, provided the render exists and the document stays small, though it is worth saying plainly that the version of the argument in the document — that shared state is inspectable in a way messages are not — was shown to be backwards in chapter 1 and complicated further in chapter 7.
The strongest argument against is that the change makes the untyped-field problem worse by multiplying the readers. A handoff payload has exactly one consumer, so an ambiguous field has one interpretation and one place to fix it. A field in state.json that nine agents can read has up to eight consumers, each with its own parsing code written at a different time by a different person, and the set is open: the agent added next quarter reads severity with no contract with whoever writes it and no conversation with anyone who does. INC-452 dropped 31 findings through one consumer's exact-match filter. The same field on a board is one exact-match filter, one case-insensitive comparison, one ordered-severity ranking that treats an unknown value as lowest, and one agent that passes the string through into a prompt.
The remedy for all three incidents is the same in either topology, and it is cheap:
{
"$id": "https://underwriting/state.schema.json",
"type": "object",
"properties": {
"findings": {
"type": "array",
"items": {
"type": "object",
"required": ["clause", "severity", "pages", "deadline"],
"additionalProperties": false,
"properties": {
"clause": {"type": "string", "pattern": "^[0-9]+(\\.[0-9]+)*$"},
"severity": {"enum": ["low", "medium", "high", "critical"]},
"pages": {"type": "integer", "minimum": 1},
"deadline": {"type": "string", "format": "date-time"}
}
}
}
}
}Validation runs on write, in the coordination layer, before the conditional write is attempted, so a producer emitting "High" gets a rejection naming the field and the permitted values rather than writing a document that nine readers will disagree about. additionalProperties: false is the clause that stops the accumulation from chapter 6 arriving through the side door, since a field nobody declared cannot be written. The same schema, applied to the payload in the existing design, would have prevented all three incidents without moving anything.
Sequential dispatch is a fact about this quarter
The document's answer to concurrency is that it does not arise, because a scheduler dispatches the nine agents one at a time and only one is ever running. Taken at face value that is a claim about the current scheduler, and the response has three parts.
The first is that it expires. Sequential dispatch is the first thing anyone changes when the system needs to be faster, and a nine-agent run whose stages are independent is an obvious candidate for parallelism the moment someone looks at the wall clock. The concurrency design will then be retrofitted onto a document that nine agents already write, under time pressure, by whoever is on that project — which is the expensive order to do it in, because every existing writer has to be audited rather than written correctly once.
The second is that sequential dispatch is usually less sequential than the scheduler believes. An agent that times out at the scheduler's four-minute limit does not stop running; it keeps its model call open and may write its result after the next agent has started. A retried task runs concurrently with nothing on paper and concurrently with its own first attempt in fact, which is the unknown-outcome case from chapter 4. An operator releasing a stuck task with runctl release writes the document while an agent is mid-turn. Each of those is a second writer in a design that assumes one.
The third is that sequential dispatch removes only one of the four failure modes this topic covered, and the document treats it as removing all of them.
| Failure mode | Removed by sequential dispatch? |
|---|---|
| Lost update from two overlapping read-modify-writes | Yes, while dispatch stays sequential |
| Torn write from a process dying mid-write | No; a single writer can be killed mid-write as easily as one of nine |
| Retry after a write whose outcome was never learned | No; unrelated to how many writers exist |
| Acting on a view that has since moved | No; the scheduler itself can change the document between an agent's read and its write |
Atomic replacement and idempotent updates from chapter 4 are load-bearing on day one under this proposal, not deferrable. Only the compare-and-swap is arguably deferrable, and it is the cheapest of the four to add up front.
What to require, and what to refuse
A review of a proposal like this is more useful as a set of conditions than as a verdict, because the underlying idea is sound and the document is one schema section and one concurrency section away from being approvable.
| Requirement before approval | Why it is a gate rather than a nice-to-have |
|---|---|
| A declared schema with typed constraints, validated on write, closed to undeclared fields | It is the actual fix for the three cited incidents, in either topology |
| Per-field write ownership: which agent may write which path | Nine writers on one document need a rule that is checkable, not conventional |
| Conditional writes, atomic replacement, idempotent updates with recorded identifiers | Three of the four failure modes above are live under sequential dispatch |
| Leases with fencing on any task longer than a minute | Nine agents make a crashed holder a weekly event |
| The four on-call questions answerable from the document, and a render that answers them | The readability argument is only true if somebody builds this |
| A bounded appended event list for status transitions | Otherwise the representation invites the confident misdiagnosis from chapter 7 |
| A written rule for what stays a message | Request-reply values on the board are how the document becomes unreadable |
| The measurement from chapter 7, on existing failures, with a stopping rule | The justification is a claim about diagnosis and should be tested as one |
Per-field ownership deserves a sentence of its own, because it is the requirement most specific to nine writers. The rule is a map from JSON path to the agent permitted to write it, enforced in the coordination layer:
WRITABLE_BY = {
"tasks[*].owner": "the claiming agent, via claim() only",
"tasks[*].status": "the current owner",
"tasks[*].output": "the current owner",
"findings[*]": "agent-risk",
"policy.deadline": "agent-parse",
"run.status": "the scheduler",
}
def authorise(agent, changed_paths, doc):
for path in changed_paths:
rule = match_rule(WRITABLE_BY, path)
if rule is None or not rule.permits(agent, doc):
raise UnauthorisedWrite(agent, path)The value of this is not security; it is that a write touching a path its author has no business touching becomes a rejected write with a stack trace, rather than a silent overwrite discovered two days later. It also documents the design in a form that stays true, since it is executed rather than described.
Two things would make the proposal a rejection rather than a set of conditions. The first is a store with no conditional write, which chapter 3 established is a constraint that arrives before the design rather than after; if the team's plan is a shared Google Doc or a file on a network mount with no compare-and-swap, the concurrency story cannot be built and the answer is to change the store or keep messaging. The second is a design where state.json holds content rather than pointers — the extracted clauses, the model outputs, the drafted sections — because that makes the document the only copy of the work, which is the loss chapter 4 opened on, and it defeats the render, and it puts a multi-megabyte object on every agent's read path. Either of those is a no with a reason attached, which is a more useful review outcome than a list of concerns.
Standing it up on something small
The way to start is with one run type rather than nine agents. Take the pipeline with the most crash-related failures, define the schema for it first — ownership, status, inputs, outputs, completion, with the constraints typed and closed — and write the render before writing the agents, because a schema whose render is awkward is a schema that will be awkward at 02:14 and it is cheap to change on day one. Then write the four concurrency tests as tests rather than as intentions: two writers racing on the same entry, a reader acting on a version that moves under it, a retry after an unacknowledged write, and a process killed mid-write. Each of them is twenty lines and each of them fails against a naive implementation, which is the fastest way to find out whether the coordination layer is real.
Run it beside the existing design rather than instead of it, on the same task with the same agents, and collect the numbers chapter 8 collected: completion quality, coordination overhead, recovery after a killed worker, state freshness. Inject the crashes deliberately, because waiting for them is waiting a quarter for one data point. If recovery does not improve, check whether the stages write their outputs incrementally before concluding anything about the architecture. And keep the diagnosis claim separate from the durability claim, measuring the first with matched observers and an adjudicated ground truth, because the two claims have different evidence and only one of them is easy to establish.
Blackboard / shared-artifact architectures — read the current version at https://daviddominguez.dev/learn/en/blackboard-shared-artifact-architectures/moving-untyped-fields-into-a-shared-file-reproduces-every-handoff-bug