Shared state survives the worker that crashed holding it

The bake-off — two builds run side by side on identical work so that the only difference between them is the one under test — ran for four weeks against the same 2,000 contracts. Two builds of the contract pipeline, five agents each — parse, clause, risk, redline, report — the same prompts, the same model versions pinned to the same snapshot dates, the same object storage. Build M coordinates by handoff, in the shape topic 11.11 described: each agent finishes, invokes its successor with a structured payload, and exits. Build B coordinates through the artifact this topic has been building, with compare-and-swap writes, leases on stages longer than a minute, and the bounded event list added at the end of chapter 7.

The headline numbers were a disappointment to everyone who had argued for the migration on performance grounds:

Four of the five rows say nothing changed. The fifth is the entire result, and it is worth understanding precisely, because the mechanism behind it is narrow and the temptation is to credit it to the architecture in general.

Holding the task and the agents still#

A comparison of two coordination designs is only informative if coordination is the only thing that differs, and that is harder to arrange than it sounds. The two builds share a single repository with the agent implementations in one package and the coordination layer behind an interface, so parse, clause, risk, redline and report are literally the same code in both arms, receiving the same prompts and calling the same tools. What differs is where an agent gets its input from and where it puts its result. In build M an agent receives a payload as its invocation argument and returns one; in build B it reads the artifact, claims a task, and writes an output pointer back.

The corpus is fixed and replayed rather than sampled live, so the same 2,000 documents run through both arms in the same order, which removes the variance that comes from one arm happening to get more scanned faxes than the other. Quality is graded blind on a stratified sample of 120 outputs by two reviewers who do not know which arm produced which document, against the same rubric used for the production quality report.

Fault injection — deliberately breaking something in a running system to observe how it copes, rather than waiting for it to break by itself — is the part that has to be arranged, because in four weeks of normal running neither arm would produce enough crashes to say anything. A supervisor kills one randomly chosen agent process at a uniformly random point in its turn, in one run out of twelve, in both arms, at the same rate. Ninety-four runs in each arm took a kill.

The eight runs that did complete in build M are worth noting rather than rounding to zero: those are the cases where the supervisor happened to kill the report agent after it had written its final document but before its process exited, so the work was already durable in object storage and the run's only loss was its exit code. That is the same property build B has systematically, arrived at by luck.

The state was somewhere a replacement could read it#

In build M, when the clause agent is killed at minute three of a four-minute turn, the run's state at that instant consists of the payload it received, which exists only in its memory, and the twenty-eight documents it has processed so far, which exist only as a list it was accumulating to return. Both die with the process. The parse agent exited fifteen minutes ago and no longer holds the payload it sent. There is no component whose job is to notice, because in a handoff design the only thing that knows the clause stage was running is the clause process.

In build B the same kill destroys the same in-memory accumulation, and destroys nothing else. The task's ownership record, its lease, its fence number and its output pointer with a count of twenty-eight are in the artifact. Ten minutes later the lease expires, a replacement agent's poll finds a task in in_progress whose lease_expires has passed, and it claims it with a higher fence.

The transition that carries the whole result is in_progress → pending on expiry, because it is the one that exists only when state is durable. A handoff design has no arrow there; there is nothing holding a record of the claim, so there is nothing to expire.

Resumption is not automatic once the task is reclaimed. The replacement has to work out what is already done, and that is only possible because the output pointer and its count were written incrementally rather than at the end:

def resume(task, store):
    already = set()
    out = task.get("output")
    if out and out.get("uri"):
        already = {k.name for k in store.list(out["uri"])}   # 28 keys, ~120 ms
    inputs = store.list(task["inputs"]["from_task_uri"])      # 61 keys
    return [k for k in inputs if derived_name(k) not in already]

The remaining thirty-three documents take about two minutes, and the run finishes eleven minutes late rather than not at all. Median added wall clock across the ninety-three recovered runs was 38 seconds, which is low because most kills landed on short stages; the distribution has a long tail bounded by the lease duration.

Incremental output writing is the requirement that makes any of this work, and it is easy to get wrong in a way that produces exactly build M's behaviour with build B's architecture. An agent that processes all sixty-one documents in memory and writes one output pointer at the end has state that is just as much in flight as a payload, and killing it loses the same three minutes. The artifact bought durability for the coordination state; the work state is durable only to the extent the agent chose to write it down as it went. Teams migrate the coordination layer, see recovery numbers that barely move, and conclude the architecture did not help, when what happened is that every stage still does its work in one unwritten block.

What the extra reads cost#

Build B performs 2,740 storage reads per run against build M's twelve messages, which is a 228× increase in operations and, in this system, almost no increase in anything anyone bills for. Five agents polling a 4 KB object every five seconds across a 45-minute run is 2,700 GET requests, which on S3-class pricing is about $0.001, plus roughly 11 MB of transfer. The 186 writes are the ones with the interesting cost, since each is a conditional write that can be rejected and retried, and the p99 write latency of 210 milliseconds at five writers is dwarfed by everything else in a run where stages take minutes.

The overhead that does matter is one this design invites and does not require. If an agent's polling is done by code — read the object, evaluate the claim predicate from chapter 1, sleep — the polls cost nothing in tokens because the model is never involved. If instead the whole artifact is pasted into the agent's prompt on every poll so the model can decide whether there is work for it, the arithmetic changes completely: 4 KB of JSON is roughly 1,100 tokens, and 2,700 polls is about 3.0 million input tokens per run, or near $9 per run at $3 per million. That is more than the pipeline's entire model budget of 8.4 million tokens, spent on asking a model to notice that nothing has changed. The rule that follows is that selection is a predicate and belongs in code; the model is invoked once a task is claimed, with the one task's contents rather than the board.

The other overhead is human. Build B's coordination layer is about 900 lines including the conditional writes, the lease renewal path, the fencing check, the atomic replacement and the render from chapter 6. Build M's is about 120. That difference is permanent, has to be tested — the simultaneous-update, stale-read, retry and partial-write cases from chapters 3 and 4 each need a test that deliberately interleaves two writers — and is the honest price of the ninety-point recovery difference.

Where a direct message is still the right call#

The result above does not generalise to every exchange between agents, and the failure mode of over-applying it is specific. A value belongs on the board when more than one component needs it, or when someone will need it after the producer is gone. A value that has exactly one consumer and no reader after it should be sent to that consumer directly, because publishing it as shared state adds a place for it to go stale and a field every other reader has to decide how to interpret.

The clearest case in this pipeline is the redline agent's request to the OCR service for a re-scan of a single page. One producer, one consumer, an answer that is consumed in seconds and is meaningless afterwards. Routing it through the artifact would mean a write, a poll interval of latency, a field with a lifecycle nobody wants to own, and the question of when it gets cleared. It also means that if the same page is requested twice, the second request either overwrites the first or has to be given an identity, which is a coordination problem that did not exist until the value was published.

Teams that adopt the blackboard as a policy rather than as a tool tend to move every exchange onto it, including the request-reply ones, and end up with an artifact carrying twelve fields that are alive for four seconds each. The board's readability, which was the point, degrades with each of them, in exactly the accumulation pattern chapter 6 described.

A related limit is that shared state is a poor fit for anything with real-time expectations. The 2.5-second median freshness above is a consequence of a five-second poll interval, and shortening the interval trades linearly against read volume and contention. Where an agent needs to react within a hundred milliseconds, the board needs a change notification alongside it, and at that point there is a message in the system whether or not it is called one.

None of this settles the question a design review actually has to answer, which is not whether a shared artifact is a good mechanism but whether a particular proposal to adopt it fixes the problem it claims to fix. A nine-agent team is about to argue that moving everything onto one shared file will eliminate a class of incident, and the argument has a substitution in it.