A loss concentrated at one handoff is a contract defect

The pipeline failed 26 of the 300 held-out runs, and the first four traces opened at random told the same story. The request attached to fixture H-014 said, in the customer instructions field, "the indemnity section was negotiated last quarter and is closed — do not propose changes to clauses 11.1 through 11.4." The redline the pipeline produced rewrote 11.2 to add a mutual cap. It was a good edit in the abstract, correctly reasoned against the policy list, and it violated an explicit instruction the customer had given, so the rubric scored it zero on the constraint-adherence criterion and the run counted as a failure. The loop, given the same document and the same instruction field, left clause 11 alone in all five trials.

Seventeen of the 26 failures had that shape. That concentration is the finding, because a topology that is genuinely worse does not fail in one place seventeen times out of twenty-six.

The constraint stops existing at the first handoff#

Reading the extraction stage's output for H-014 explains it without any inference. The stage receives the full request — document text, customer instructions, the policy list reference — and emits a list of clauses. That list is the pipeline's first boundary contract, the term for the agreed shape of what one stage hands the next, which in a typed pipeline is a schema and in a looser one is whatever the prompt says to expect. Its schema has fields for the clause number, its text, its type, and a confidence score. It has no field for anything the customer said.

The drafting stage is behaving correctly given what it was handed. It sees a normalised clause that deviates from the policy list and proposes the deviation be fixed, which is its entire job. Nothing in its context indicates that this clause is out of bounds, and nothing in the pipeline's design ever intended for the instruction to be dropped — the boundary schema was written early, when the extraction stage's only consumer needed clause text, and it was never revisited when the drafting stage arrived.

This is the same class of defect as a data transfer object that silently loses a field between two services because the mapper was written against an earlier version of the upstream type. Nobody responds to that by concluding that service boundaries are a bad idea. The response is to fix the mapping, add a test that fails when a field is dropped, and move on. An agent boundary is a mapping with the same failure mode and worse observability, since a missing field produces a plausible output rather than a null pointer.

Attributing failures to a boundary is worth doing mechanically rather than by reading, because reading twenty-six traces gives an impression and a script gives a distribution. Each stage's input and output are already in the trace, so the check is whether a property present in the request survives to the stage that needed it.

def constraint_survives(trace: Trace) -> dict[str, bool]:
    """Which stage inputs still contain the request's constraint text?"""
    needle = trace.request.customer_instructions
    if not needle:
        return {}
    return {
        step.name: any(needle_overlap(needle, m.content) > 0.6 for m in step.input)
        for step in trace.steps
    }

Run over the 300 held-out pipeline runs, that produced a clean picture: the constraint text is present in the extraction stage's input in 100% of runs where the fixture has one, present in the normalisation stage's input in 0%, and present in the drafting stage's input in 0%. Every failure of the seventeen sits downstream of that cliff.

The four at the second boundary are a second, smaller defect of the same kind, found only because the first one prompted a proper look. The five in-stage failures are the ones that would remain if both boundaries were perfect, and five out of 300 is roughly where the loop's own failure rate sits.

Fixing the contract and re-running, on tasks the audit has not seen#

The change is unglamorous. The boundary type gains a field, every stage that constructs the type is required to populate it, and the drafting stage's prompt gains one sentence instructing it to leave constrained clauses untouched and note them in its output.

class ExtractedClause(BaseModel):
    number: str
    text: str
    clause_type: ClauseType
    confidence: float

class ExtractionResult(BaseModel):
    clauses: list[ExtractedClause]
    constraints: list[str]          # verbatim from the request, never summarised
    out_of_scope: list[str]         # clause numbers the constraints exclude

    @model_validator(mode="after")
    def constraints_required(self) -> "ExtractionResult":
        if self.source_had_instructions and not self.constraints:
            raise ValueError("request carried instructions; none propagated")
        return self

Carrying the constraint verbatim rather than as a summary is the part that is easy to get wrong. An earlier draft of the fix had the extraction stage emit a paraphrase — "customer considers indemnity closed" — and two of the seventeen still failed, because the paraphrase lost the clause numbers and the drafting stage had to guess which clauses "indemnity" covered. Payloads that carry a model's restatement of an instruction inherit that model's error rate at every hop; payloads that carry the original text do not.

The validator raising rather than logging is deliberate. A boundary defect that produces a plausible output is invisible until somebody scores 300 runs, and a boundary defect that throws is visible in the first trial. Trading a silent quality loss for a loud crash is worth it in a component where the alternative failure is undetectable.

The re-run could not use the same 60 held-out documents, because the audit read their failure traces and the previous chapter's rule applies to the people doing the fixing as much as to anyone else. Forty fresh documents were labelled from the archive, sampled the same way, to serve as a confirmation set — a second held-out slice whose only job is to test whether a fix that was derived from failures holds up on tasks the fixing never saw — and both arms were run against them at five trials each under a new manifest that differed from the previous one only in the pipeline's implementation commit and fixture set.

The pipeline now wins on quality by a gap whose interval clears zero, at 2.07× the cost and 1.7× the tail latency. What that combination should lead a team to do is a question about thresholds and budgets rather than about measurement, and it is the subject of the decision record two chapters from now. The measurement question is settled: the pipeline's earlier loss was a bug in one payload, and it is gone.

Fixing until your preferred arm wins#

The obvious hazard is that this procedure, applied without rules, is a machine for producing whatever conclusion the auditor already held. Read the losing arm's traces, find something to fix, re-run, repeat until the numbers agree with your instinct. The constraints that keep it honest are procedural and cheap to state.

The fix must repair a violation of something the design already claimed. The pipeline's own design document said the drafting stage operates under the request's constraints; the payload did not carry them, so the implementation contradicted the design and the repair restores it. A change that improves scores without any such claim behind it is tuning, and it belongs to the previous chapter's accounting of surfaces rather than to this one's.

The audit budget is fixed in advance and applied to both arms. An audit round here means one pass of exactly what the previous section did — read a sample of that arm's failure traces, find the defects, repair them — and it is budgeted because it is the step that can be repeated until the numbers agree with whoever is repeating it. One round per arm, before any number is recorded, on the same held-out slice, with the same person doing both. The loop's audit found two failures caused by a tool response truncated at 8,000 characters, dropping the tail of a long schedule; raising the limit and paginating brought the loop to 86.2% on the confirmation set, which is the figure in the table above. Auditing only the arm you expect to win is how the procedure turns into advocacy, and running both audits costs a day.

The re-run happens on tasks the audit did not touch. Forty fresh documents cost one engineer a day and a half, which is the price of the confirmation set being able to mean anything.

Concentration is what distinguishes a defect from a verdict, and it can be checked rather than argued. Seventeen failures at one boundary, all with the same missing field, all disappearing when the field is added, is a defect. Failures spread evenly across every boundary, with different causes, that shrink only slightly when any one of them is addressed, is the shape of a topology that is losing on its merits.

Turning one audit into a standing check#

The seventeen failures were found because somebody read traces after a disappointing result, which is a discovery method that works once and depends on the result being disappointing enough to prompt a read. The same check runs as an assertion for a few lines of code, and once it does, the class of defect stops being something the evaluation has to catch at all.

What the assertion checks is a boundary invariant, a property that has to hold on every single run whatever the task was and whatever the answer turned out to be. The property being asserted here is that specified inputs reach the stage that consumes them. For this pipeline that is three invariants: request constraints appear verbatim in the drafting stage's context whenever the request has any, clause numbers in the drafting stage's input match the numbering in the original document, and the policy list reference passed to normalisation resolves to a document with a version the request agrees with. Each is checkable from the trace without a rubric, without a label, and without a model in the loop, so each can run on every request in production rather than on 300 fixture runs.

def boundary_invariants(trace: Trace) -> list[str]:
    failures = []
    if trace.request.customer_instructions:
        draft_ctx = "\n".join(m.content for m in trace.step("draft").input)
        if trace.request.customer_instructions not in draft_ctx:
            failures.append("constraints_not_propagated")
    if not numbering_preserved(trace.request.document, trace.step("draft").input):
        failures.append("clause_numbering_rebased")
    return failures

Running that over a week of production traffic on the shipped pipeline surfaced the constraint failure on 6.4% of requests, which is roughly the rate the fixture set showed, and it surfaced it in a dashboard rather than in an architecture debate. Instrumenting boundaries this way changes what an eval is for: the harness stops being the only place boundary defects can be detected, so a future comparison is more likely to be measuring topologies rather than the current state of everyone's payload hygiene.

The sentence that outlives the trace#

The cost of recording the pre-fix result as it stood is not the wrong architecture in one system. It is the sentence that would have gone into the decision record: "the multi-agent pipeline lost on constraint adherence in our matched comparison." That sentence is short, quotable, backed by an interval, and will be cited in design reviews for two years by people who will never open trace H-014. It survives its own evidence, and it becomes the reason a later team does not attempt a decomposition that would have worked for them, on a workload nobody re-measured.

Architectural beliefs acquired from bugs are unusually durable because they arrive with numbers attached and no obvious way to falsify them. The defence is to require, before any topology-level claim is written down, that the losing arm's failures have been attributed to a level — a payload, a stage's prompt, a tool, or the topology — and that the attribution is in the record beside the claim. A reader in six months can then check whether the loss was ever really about the shape of the system.

Nothing in this chapter says a topology cannot lose. It says this particular loss did not, and the discipline it introduces has a mirror image: a team that has learned to discount every multi-agent loss as somebody's broken payload will never accept a genuine one either. Establishing that coordination itself can be the cause requires measurements of a different kind — failure rates per boundary, tokens spent re-sending context, latency added per hop — and a prediction about how the gap behaves as boundaries are added.