Three unreliable agents downstream of one rewritten sender

Three requests arrived in the same planning cycle, from three teams who had not spoken to each other. The eligibility agent's team said their agent was misreading policy edge cases and wanted the frontier model. The document agent's team said theirs was extracting the wrong dates from uploaded evidence and wanted the same. The correspondence agent's team said theirs was producing letters that referenced the wrong scheme and had already run a small evaluation showing the stronger model doing better. Each request was written independently, each was well argued, and together they would take inference spend from about $19,000 a month to about $57,000.

All three agents receive their input from the same place. The case assembly agent sits directly above them, and it had been rewritten six weeks earlier, in a refactor whose stated goal was to simplify its output. What the boundary shape records from chapter 6 showed, once someone thought to look, was that the median constraints count on the assembly agent's outbound payloads had gone from 3.1 to 0.4 in the week of the rewrite, and that a field called summary had appeared in the unknown_fields counter at all three receivers, 4,100 times a week, ignored by every one of them because their schemas did not know it. The refactor had consolidated constraints, evidence and open_questions into one prose string, which is chapter 1's failure arriving through a tidy-up rather than through a design decision.

Why the common-cause hypothesis deserves the first day#

Three independent regressions in the same fortnight, in three components maintained by three teams that did not change their own prompts, is a coincidence with a low prior — a low probability assigned to it before any of the specific evidence is examined. One regression in a shared upstream, in a component that visibly changed six weeks ago, explains all three observations with a single event. That is not a proof, and the reason to prefer it is not elegance; it is that checking it is cheap and the alternative commits $38,000 a month before anyone has looked at what the agents are being handed.

The timing evidence is worth assembling before anything else, because it takes twenty minutes and frequently settles the direction. Complaints from the eligibility team began five weeks ago, from the correspondence team four weeks ago, and from the document team six weeks ago, which brackets the assembly rewrite rather than clustering after any of the three receivers' own deploys. Three teams noticing at different times is what you would expect from a common cause, since each team's detection depends on its own traffic mix and how visible its failures are, not on when the fault started.

Symptom shape points the same way. An agent that is genuinely at the edge of its capability fails on hard instances and succeeds on easy ones, so its failures correlate with instance difficulty. An agent starved of an input fails on instances where that input mattered, regardless of difficulty, which is why the eligibility agent was getting straightforward cases wrong while handling genuinely intricate policy interactions correctly — the intricate ones were intricate in ways the policy documents covered, and the simple ones turned on a constraint the customer had stated and nobody transmitted.

The experiment that cannot settle it, and the one that can#

The tempting move is to approve one upgrade as a trial: put the eligibility agent on the stronger model, and if it improves, the capability explanation is supported and the other two follow. That design has no diagnostic power, because a stronger model improves under both hypotheses.

A larger model is better at inferring the missing constraint from the surrounding prose, at noticing that a summary string contains something limit-shaped, and at asking for clarification instead of proceeding. Those are exactly the behaviours that partially repair a lossy payload. The experiment therefore produces the same qualitative outcome either way, and because the gain is real it reads as confirmation, which is the worst property an experiment can have: it terminates the investigation with the wrong conclusion and a receipt.

The correspondence team's own evaluation is worth examining, because it was competently run and still pointed the wrong way. They had assembled 80 test cases by hand, each one a payload written by an engineer to represent a scenario, and the stronger model scored 12 points higher on that set. Every input in it was well-formed, because a person writing a test case writes down the constraints they have in mind. The evaluation therefore measured the receiver's capability on good payloads, which was never the question, and it could not have detected a sender defect no matter how many cases it held. Evaluations built from hand-written inputs test the component in isolation and are silent about the boundary above it, which is a general property rather than a mistake this team made: an isolated evaluation and a production failure rate diverge exactly when the production input has drifted.

The check that does discriminate holds the receiver fixed and varies the sender, which is the axis the two hypotheses actually disagree about. Both versions of the assembly agent are in git, the failing runs are recorded, and the receivers are deterministic enough to replay against. Each arm below is one combination of sender version and receiver model, run over the same set of recorded cases, so that any difference in outcome between two arms is attributable to the one thing that differs between them.

ARMS = {
    "baseline":      (assembly_current,  MODEL_CURRENT),
    "old_sender":    (assembly_pre_v42,  MODEL_CURRENT),   # the discriminating arm
    "strong_model":  (assembly_current,  MODEL_STRONG),
    "both":          (assembly_pre_v42,  MODEL_STRONG),    # is the gain additive?
}

for run in failing_runs(agents=["eligibility", "document", "correspondence"],
                        window_days=14, n=300):
    for arm, (sender, model) in ARMS.items():
        payload = sender.emit(run.case_state)              # same case, two senders
        result  = receiver_for(run).run(payload, model=model, tools=SANDBOX)
        record(run.id, arm,
               correct=grade(run.ground_truth, result),
               constraint_digests_in=digests(payload),     # chapter 6's digests
               constraints_present=len(payload.constraints),
               tokens=payload.token_count,
               cost_usd=price(model, payload.token_count, result.output_tokens))

Three hundred failing runs across three receivers, four arms, replayed against sandboxed tools, is a few hours of compute and one engineer's day. The digest field is what connects the outcome to a mechanism: for every run that the old_sender arm gets right and baseline gets wrong, you can point at the specific constraint digest that was present in one payload and absent in the other, which converts the result from a bar chart into an explanation.

Restoring the sender recovers roughly thirty points across all three receivers; the stronger model recovers about twelve, and adds one point on top of a repaired sender for three times the cost. The uniformity of the old_sender gain across three unrelated tasks is the finding that closes the argument, because no story about three teams' agents independently declining in capability predicts that all three recover by the same amount when a fourth component is reverted.

The both arm earns its place by testing whether the two effects add, and the answer here is that they do not. If the stronger model were compensating for something other than the missing constraints, its twelve points would largely survive on top of the repaired payload and both would land somewhere near 99%. It lands one point above old_sender, which says the gain the strong model was producing was almost entirely reconstruction of information the sender had stopped transmitting. That is the quantitative version of the argument this chapter opened with, and it is the sentence to put in the decision document, because it converts "the upgrade is unnecessary" from an assertion into a measurement someone can re-run.

A note on how these numbers should be read: the runs are drawn from failures rather than from all traffic, so 61% at baseline is a recovery rate on known-bad cases and not the system's success rate. That sampling is deliberate, since failing runs are where the discriminating signal lives, and it means the arms are comparable to each other and to nothing else. Reporting them as absolute quality figures is how a useful diagnostic gets quoted back at you six months later in an argument it cannot support.

What changes, and what stops it happening again next quarter#

Reverting the assembly agent is the wrong fix, since the refactor did other things that were improvements, and a revert leaves the boundary exactly as vulnerable as it was. The change is to restore the typed fields — constraints, evidence, open_questions — as the contract, and to keep the summary string as the narrative field from chapter 2, explicitly documented as colour rather than as the place limits live. That is a day's work in the sender and no work at all in the receivers, which were never changed and had been quietly ignoring summary the entire time.

Stopping the recurrence is the more valuable half, because this argument regenerates every time an upstream changes. Three things carry that load, and none of them is a policy about being careful.

The first is a consumer-driven contract test, in the same spirit as the ones web service teams have run with Pact, the contract-testing tool that lets a consumer's expectations execute inside the provider's build, for a decade. Each receiver publishes the set of fields it reads and the invariants it depends on — constraints is present, constraints_elicited is true, every evidence item has an as_of — and the sender's CI replays fifty recorded case states through its own emitter and asserts every consumer's expectations hold. The assembly refactor would have failed on that suite in the pull request, six weeks and roughly $30,000 of misrouted debate earlier, and the failure message would have named the three consumers that broke.

The second is an alert on the shape metrics rather than a dashboard nobody opens. Median constraint count per boundary, share of payloads with constraints_elicited false, and unknown-field volume per receiver each want a threshold, because each of them moved sharply on the day of the rewrite and none of them was being watched. A step change from 3.1 to 0.4 in a median is not a subtle signal; it went unnoticed because it was not connected to anything that pages.

The third is procedural and does the most work per unit of effort. A request to change a model in response to a reliability complaint has to carry the boundary evidence with it: the field-population comparison between failing and succeeding runs, and the sender's recent deploy history. That requirement is cheap for a team whose agent is genuinely at the limit of its model, since the comparison shows complete payloads and the failures spread across difficulty rather than clustered on a missing field. It is expensive only for a request that has not looked, which is the request you want to make expensive.

There are boundaries where the capability explanation is the right one, and the same evidence identifies them. When the payloads on failing runs are complete and identical in shape to the payloads on succeeding runs, when failures track instance difficulty, when they are distributed across receivers with different upstreams rather than concentrated below one, and when the old_sender arm shows nothing, then the agent is doing the best its model can with a good input and a better model is what you are buying. Reaching that conclusion after the check costs a day; reaching it before costs a quarter's budget and leaves the sender broken underneath the upgrade.

The place to start is not the boundary that is currently on fire. Take the busiest boundary in the system, write down the fields the receiver actually reads, and compare that list against what the sender emits on a hundred recorded runs — the gap in that comparison is usually visible within an hour, and it is the same gap that produced every failure in this topic.