Four defensible outputs and no way to name the fault

Invoice INV-88214 went out to Meridian Freight for $171,312.00. The correct figure was $169,825.06. The receivables service that produced it runs four agents — extract, validate, price, write — and the review that followed had all four prompts and all four outputs on a shared document within an hour of the customer's complaint. Two days later the thread had eleven participants and no conclusion.

Here is what each agent produced. The extractor pulled 340 billable lines from the metering export, subtotal $172,000.00, and every line reconciles against the raw export. The validator checked those lines against Meridian's contract and reported that volume rebate tier 3 applies at 9.5% and that the clause 7.2 freight surcharge applies at 9.1%; both are correct, and its report also carried the sentence "surcharge base is the rebated subtotal per 7.2(b)". The pricer applied the 9.5% rebate to get $155,660.00, then applied the 9.1% surcharge to $172,000.00 rather than to $155,660.00, producing $15,652.00 of freight instead of $14,165.06. The writer rendered exactly what the pricer handed it.

Read in isolation, the pricer is the obvious culprit and also the only agent whose output is arguably defensible on its own terms, because a freight surcharge quoted as a flat percentage with no stated base is conventionally applied to the subtotal. Two of the eleven people on the thread made that argument. Three others pointed at the validator for burying the base qualifier in prose. The rest wanted to know why the surcharge was applied at all on a rebated account.

Why reading each agent's output cannot settle it#

Every one of those positions is an inference about an input nobody has. The review had the pricer's prompt template — a file in the repo with {{pricing_brief}} in the middle of it — and the pricer's output. It did not have the rendered string that filled that placeholder on this run, because nothing persisted it.

That gap is not incidental. The orchestrator does not hand the validator's report to the pricer verbatim; it compresses. The validator's report ran 2,200 tokens on this invoice, and the pricer's context budget allocates 400 to the pricing brief, so a compression step in between — a formatting function wrapping a small model call, configured inline in the orchestrator rather than defined as an agent — reduced it. On this run the compressed brief read freight surcharge: 9.1% (clause 7.2). The base qualifier was gone.

So the pricer made a correct decision on a broken delegation, and the review had no way to see that, because the compression step had no prompt file to review, no output to read, and no place in the four-box diagram everyone was arguing over. Per-agent review is structurally blind to it: it examines nodes, and the fault was on an edge.

This is the distinction that attribution turns on, and it is worth stating plainly. When a stage produces a wrong result there are three possibilities: the stage reasoned badly on good input, the stage reasoned well on input that had already lost something, or the stage's output was correct and something downstream mangled it. Reading prompts and outputs can distinguish the first from the third. Nothing about it can distinguish the first from the second, because both look identical from the output side — an output that is wrong in a way the agent's own instructions do not forbid.

Recording the edges#

Eight handoffs, not four agents. The compression sits between hf-4 and hf-5 and is the only place in the run where information was lost. Diagramming the pipeline as four boxes and three arrows — which is how it appeared in the design doc and on the review thread — makes that step invisible, because it is not a box and it is not one of the three arrows people had in mind.

Once the edges are the unit, attribution becomes a diff. hf-4 contains the base qualifier and hf-5 does not; the fault is in the transformation between them; the pricer is exonerated by comparison rather than by argument. That took ninety seconds to establish on the next incident, once the records existed.

What a handoff record has to contain#

The record is small and its exact fields matter more than its format.

{
  "run_id": "run_01JQ8W3ZN4",
  "handoff_id": "hf-5",
  "seq": 5,
  "from": "orchestrator.compress_v3",
  "to": "price",
  "parent_handoff_id": "hf-4",
  "ts": "2026-03-11T09:14:22.481Z",
  "input_rendered_ref": "s3://rcv-traces/run_01JQ8W3ZN4/hf-5.in.txt",
  "input_sha256": "9f2c1e...",
  "input_tokens": 24312,
  "output_rendered_ref": "s3://rcv-traces/run_01JQ8W3ZN4/hf-5.out.json",
  "output_sha256": "0ab74d...",
  "config": {
    "model": "claude-opus-5",
    "effort": "high",
    "prompt_pack": "2026-03-r2",
    "template": "prompts/price.md@8c31a9f"
  },
  "attempt": 1,
  "retry_of": null
}

Four of those fields do the work and the rest are convenience. input_rendered_ref points at the exact bytes the model saw, after every template substitution and every compression — the template reference is useful for finding the code but is not a substitute, because two runs of the same template produce different rendered inputs and only one of them is the run that failed. run_id ties all eight records together, so a query returns a run rather than a scatter of unrelated spans. parent_handoff_id gives the ordering when stages run in parallel and seq no longer implies causality. And attempt / retry_of make retried stages distinguishable, which matters because chapter 2's 41% retry success rate means a substantial fraction of runs contain a stage that ran twice on identical input and produced different output — a fact that is invisible if retries overwrite each other's records.

Storage is not a reason to skip this. The receivables service processes about 4,100 invoices a month, the rendered inputs and outputs come to roughly 340 KB per run, and 90 days of full retention is about 4.2 GB. That is smaller than the service's application logs, which nobody proposed truncating.

What to do on the incident where the records do not exist yet#

Nobody has these records the first time they need them, and the review of INV-88214 could not wait for a quarter of instrumentation work. Three things were recoverable that week, in descending order of usefulness.

The application logs held the orchestrator's HTTP request bodies for 48 hours at DEBUG level, which had been switched on nine days earlier during an unrelated incident and never switched off. That was luck, and it is what produced the compressed brief. Checking whether some accidental retention exists is worth ten minutes before anything else, because it is the only path that recovers the actual bytes.

Failing that, the compression step is deterministic enough to re-run. Feed the validator's stored output back through orchestrator.compress_v3 at the same model and effort setting and see what comes out. This is weaker evidence — a second run is not the run that failed, and on a stochastic step it may not reproduce — but running it twenty times and finding that the base qualifier survives in three of twenty is itself a finding, and a much stronger one than an argument on a thread.

The weakest option, and still better than nothing, is to look for the same signature across other runs. Even without stored inputs, the outputs of every stage were retained, and a query for invoices where the validator's report mentions a surcharge base and the final total is consistent with the unrebated subtotal found nine more affected invoices going back six weeks. Discovering the blast radius — how many other runs the same fault reached before anyone noticed — does not identify the mechanism, and it converts one customer complaint into a known population, which is what turns a debate into a remediation.

What the records cost besides storage#

Persisting rendered inputs means persisting whatever was in them, and on a receivables pipeline that is customer names, contract terms, pricing, and in some jurisdictions personal data on the signatory. The 4.2 GB figure above is the easy part of the bill.

The retention policy that survived legal review on this service was 30 days at full fidelity in a bucket with the same access controls as the primary invoice store, then 12 months of records with the content references dropped and the SHA-256 digests, token counts and timings kept. The digests are what let you still answer "did handoff five change between these two runs" a year later without holding the text, and the timings are what let you compute boundary latency trends. Roughly 95% of the storage and essentially all of the exposure sit in the 30-day window.

The other cost is a failure mode: a trace store that holds rendered prompts is a place where a leaked credential or an injected instruction is durably retained and readable by anyone with trace access, which is usually a broader group than has access to the production data itself. Scrubbing on write is the standard answer and it is imperfect; scoping trace-store access to the same group as the primary store is the answer that actually holds.

What the records settle and what they do not#

The last two rows are the honest limits. Handoff records make a failure attributable; they say nothing about whether the boundary that failed should exist at all. And attribution is not blame allocation — establishing that the compression step dropped a qualifier does not settle whether the fix belongs in the compressor, in the validator's output format, or in the pricer's prompt, and reasonable people continued to disagree about that after the mechanism was known. What changed is that they were arguing about a fix rather than about what happened.

There is also a failure mode the records create rather than solve. A run of eight handoffs at 340 KB is a lot of material, and an on-call engineer paging through eight rendered contexts at two in the morning is not obviously better off than one reading four outputs. The records pay off when they are queryable — a stored diff between consecutive handoffs, an alert when a compression step reduces a payload by more than some ratio, a check that every numeric constraint present in hf-4 is still present in hf-5. Persisting the bytes is the prerequisite; the tooling on top is the deliverable.

The verifier buys detection and no attribution#

The proposal that came out of the incident review was a fifth agent that recomputes the total from the extracted lines and the contract, and rejects the invoice when its figure differs from the pricer's. It would have caught INV-88214. It is still the wrong first move.

A verifier detects. It does not attribute, which means every rejection it produces lands in a queue of cases with exactly the property that made this incident take two days: a discrepancy, four defensible outputs, and no record of what each agent was given. The team's throughput problem becomes a queue instead of an incident, and the queue is staffed by the same people making the same inferences from the same missing evidence.

It also adds a fifth agent and its two boundaries to a pipeline whose demonstrated problem is that nobody can reason about the three boundaries it already has — and, at the rates from chapter 1, a verifier that re-reads the line items, the contract and the priced output is not a cheap agent. It carries the invoice text, the contract terms and the full priced record, which on this workload is roughly 24,000 tokens of input per call before its own turns accumulate.

And a verifier that disagrees with the pipeline is one more defensible output nobody can adjudicate. When the verifier says $169,825.06 and the pricer says $171,312.00, something has to decide which is right, and on the first week of running it in shadow mode, alongside the live pipeline with its verdicts recorded and nothing acted on, the verifier was wrong on 31 of the impossible 96 disagreements it flagged — mostly by applying a surcharge base that the contract's clause 7.2(c) exempts for freight-forwarder accounts. The check needs a check.

None of that makes verification useless. It makes verification a second-order decision that depends on attribution existing first, because a detection mechanism whose disagreements cannot be adjudicated adds work rather than removing it. Recorded handoffs are what make the verifier's disagreements resolvable in minutes, and they are also what let you ask, over a few hundred runs, how often the verifier changed an outcome at all — which is a question chapter 7 turns into a general method for deciding whether a stage deserves to exist.

Attribution answers "which part of this design failed". It does not answer "should this design exist". Once the receivables team could name the failing edge, the next question they asked was the obvious one, and it was about the pipeline as a whole rather than any stage in it: the single loop this thing replaced never had a compression step, because it never had a boundary to compress across. Answering that question fairly turns out to be harder than instrumenting the boundaries, and it is where the next chapter goes.