The loop you kept in the repo has not been maintained

The contract-review service reads a supplier agreement, extracts the obligations it creates, and drafts a one-page summary for the legal team. Eight months ago it was one tool-using loop running five turns. Today it is three agents — classify, extract, draft — and the loop's code is still in the repository at legacy/review_loop.py, last modified 14 July 2025, with a comment at the top saying kept for comparison.

Somebody finally wanted the comparison. The pipeline costs $4.10 per contract against the loop's last recorded $1.35, its p95 is 71 seconds against 39, and the quality scores on the weekly regression suite are within a couple of points of each other. Running legacy/review_loop.py against the current 240-contract task set takes about twenty minutes of setup, and the numbers it produces will be worthless.

The two arms are not the same age#

A comparison of this kind has two arms, the term trials use for the conditions being run against each other — here the three-agent pipeline and the single loop, each given the same contracts, the same budget and the same scoring. Everything the team learned about reviewing contracts in the last eight months went into the pipeline, because the pipeline is what runs in production. The loop's prompt file has none of it. This is not a subtle confound to be controlled for statistically; it is six specific pieces of work, each of which improved the task rather than the topology, and each of which the loop is entitled to.

The third row alone is worth flagging. Switching the PDF tool from raw text to layout-preserving extraction cut input tokens by 31% and raised obligation recall — the share of the obligations actually present in a contract that the system finds — by an estimated four points, and the loop is still passing the model unstructured text with the table columns interleaved. A comparison run today would measure that tool change and report it as a property of having three agents.

Topic 11.10 established the discipline here for delegation trees: hold the model, the effort setting, the tool budget, the task set and the prompt pack — the versioned bundle of prompt files and shared task definitions both arms load — constant, back-port the accumulated repairs into the flat arm, and report both baselines so the size of the back-port effect is visible rather than laundered. That rule transfers to linear pipelines unchanged and is not the thing this chapter is about. What it leaves open is the practical question that kills the comparison in most organisations, which is that the back-port is three days of work nobody has scheduled, and the person who wants the comparison is not the person who has to do it.

An equal tuning budget, written down#

The fix is to treat tuning effort as a variable to hold fixed, in the same category as the model and the task set, and to give it a number before either arm runs. On this comparison the number was three engineer-days per arm, spent by the same engineer, in alternating half-days so that neither arm got the benefit of the other's fresh insight arriving last.

That sounds bureaucratic and it is the only mechanism that survives contact with a deadline. Without it, the sequence is always the same: the pipeline is production and gets touched constantly; the loop gets three hours on a Friday; the loop loses; the result is written down as "we tested this". Three days each is not a large budget — it is roughly the back-port table above, plus a day of headroom for whatever each arm turns out to need — and its value is that it makes the asymmetry a scheduling decision rather than an invisible property of who works on what.

The other half of the budget rule is a floor on what counts as a legitimate change. Anything that alters the task definition — the taxonomy, the constraint list, the return contract, the tool wrappers — goes into both arms and is checked into a shared pack. Anything that is genuinely about topology — how the loop sequences its turns, how the orchestrator briefs a stage — belongs to one arm and stays there. Disputes about which side of that line a change falls on are worth having out loud before the runs, because after the runs everyone's answer is determined by which arm won.

Choosing what the arms run on#

The 240-contract set the team reached for was the weekly regression suite, and it is the wrong task set for this question in a way worth being explicit about. A regression suite accumulates cases that once broke: it over-represents the pathological — scanned faxes, contracts with the obligations in an annexe, three agreements with clause numbering that restarts mid-document — and under-represents the ordinary contract that makes up most of production traffic. An arm that is better on hard contracts and worse on easy ones will win on the regression suite and lose in production, and the reverse is equally possible.

The set used here was 240 contracts drawn as a stratified sample of the last quarter's production traffic — stratified by page count and by whether the document was born-digital or scanned, because both drive difficulty — with the regression suite kept as a separate secondary arm reported alongside. On this comparison the two sets disagreed by 2.6 F1 points on the size of the gap — F1 being the single score that combines recall with precision, the share of returned obligations that are real, so that neither missed obligations nor invented ones hide behind the other — in the pipeline's favour on the regression suite, which is what you would expect from a design whose advantage is on long and awkward documents.

Scoring is the other half. Obligation F1 needs a ground truth, and the ground truth here was built once by two lawyers over three days and reused across every arm and every replicate, which is what makes 4,000 scored runs affordable. The scorer never sees which arm produced an output. That last condition is not paranoia about integrity; it is that obligation extraction has genuine judgement calls at the margin — is "Supplier shall use commercially reasonable efforts" an obligation — and a human who knows which arm they are looking at resolves those margins consistently in one direction without noticing.

Instrumenting the three mechanisms#

A comparison that produces two quality numbers and two cost numbers tells you which arm won and nothing about why, which means it cannot be acted on except by adopting or discarding the whole design. Three quantities turn a result into a diagnosis, and all three are properties of boundaries — so they are available for free once the handoff records from chapter 3 exist, and unavailable in principle without them.

Boundary latency is wall-clock between an upstream stage emitting its last output token and the downstream stage receiving its first input token. It captures orchestration, serialisation, queueing, and any compression call in between, and it is the term that explains p95 gaps. On this pipeline it came to 6.8 seconds per run at the median across two edges and 14.2 seconds at p95 — which is most of the 32-second p95 gap, and none of it is model time.

Duplicated context is the fraction of prompt tokens across the run that re-present material already presented to some other stage in the same run. It is the mechanism from chapter 1 expressed as a ratio rather than a bill, and it localises: knowing that 44% of the pipeline's prompt tokens are duplicates is useful, and knowing that 31 of those 44 points come from the draft stage re-reading the full contract text is actionable.

Coordination errors are runs where a constraint present in an upstream handoff is absent from a downstream one — the INV-88214 failure from chapter 3, counted rather than investigated. A single-loop arm has no boundaries and so has no coordination errors by construction, which would be a rigged metric if the loop did not have an exact analogue: a constraint present early in the conversation that is absent from the final output. Counting both under one definition keeps the comparison honest.

from hashlib import blake2b

def shingles(text: str, w: int = 12) -> set[bytes]:
    """Overlapping w-token windows, hashed. Cheap approximate content identity."""
    toks = text.split()
    return {blake2b(" ".join(toks[i:i + w]).encode(), digest_size=8).digest()
            for i in range(max(0, len(toks) - w + 1))}


def duplicated_context_ratio(handoffs: list[dict]) -> tuple[float, dict[str, float]]:
    """Share of prompt content each stage re-presents after some earlier stage saw it.

    Returns the run-level ratio and the per-stage contribution, so a high number
    points at a stage rather than at the topology in general.
    """
    seen: set[bytes] = set()
    total = dup = 0
    by_stage: dict[str, int] = {}
    for hf in sorted(handoffs, key=lambda h: h["seq"]):
        sh = shingles(read(hf["input_rendered_ref"]))
        overlap = sh & seen
        total += len(sh)
        dup += len(overlap)
        by_stage[hf["to"]] = by_stage.get(hf["to"], 0) + len(overlap)
        seen |= sh
    if total == 0:
        return 0.0, {}
    return dup / total, {k: v / total for k, v in by_stage.items()}


def boundary_latency_ms(handoffs: list[dict]) -> dict[str, int]:
    """Gap between an upstream stage finishing and the next one starting."""
    by_id = {h["handoff_id"]: h for h in handoffs}
    out = {}
    for hf in handoffs:
        parent = by_id.get(hf["parent_handoff_id"])
        if parent is None:
            continue
        edge = f'{parent["to"]}->{hf["to"]}'
        out[edge] = ms(hf["input_first_token_ts"]) - ms(parent["output_last_token_ts"])
    return out


def dropped_constraints(handoffs: list[dict], constraints: list[str]) -> list[str]:
    """Constraints stated upstream that no longer appear downstream."""
    lost = []
    for i in range(1, len(handoffs)):
        up = read(handoffs[i - 1]["output_rendered_ref"])
        down = read(handoffs[i]["input_rendered_ref"])
        lost += [c for c in constraints if c in up and c not in down]
    return lost

duplicated_context_ratio returning a per-stage breakdown rather than a single number is the design decision that matters in that listing. A run-level 44% is a fact about the architecture and invites an architectural response; "the draft stage contributes 31 of the 44 points because it re-reads the full contract to write a summary it could write from the obligation list" is a two-hour change.

Where each mechanism metric misleads

None of the three is a number to put on a dashboard without knowing how it lies.

Boundary latency undercounts when stages run concurrently, because the gap you measure on one edge overlaps with work happening on another and summing the edges double-counts. The fix is to report boundary latency as a share of the critical path — the chain of steps that has to happen in sequence, which is what sets a run's wall-clock — rather than as a total, which on this pipeline is 6.8 of 47 seconds at the median and 14.2 of 71 at p95. It also silently includes queue time, so a spike in the metric can mean the orchestrator is contended rather than that the design is slow.

Duplicated context overcounts on documents with heavy boilerplate. Two agents that both read a contract's standard indemnity language are duplicating in the arithmetic sense and not in any sense that matters, because the second one genuinely needs it. Shingle-based counting cannot distinguish "re-presented material the stage needs" from "re-presented material the stage ignores", which is why the per-stage breakdown is the useful output and the run-level ratio is a headline. Chapter 7 gets at the distinction properly by measuring what each stage's output actually influences.

Coordination errors depend entirely on the constraint list you check against, and a list of twelve constraints will find fewer violations than a list of forty for reasons that have nothing to do with the system. The list has to be fixed before the arms run and shared between them, and it should be derived from the contract's own requirements rather than from whatever the pipeline happens to emit — otherwise it encodes the pipeline's vocabulary and the loop scores well by never using those words.

Writing the decision rule before you have the numbers#

The last preparatory step is to decide, in advance, what each possible result means. This is unpopular and it is the difference between an experiment and a search for supporting evidence.

CONTRACT REVIEW — ARM COMPARISON, DECISION RULE
registered 2026-03-02, before any arm was executed

Primary:   obligation F1 on the 240-contract set, blind-scored, 8 replicates per arm
Secondary: cost per successful review; p95 wall-clock
Mechanism: boundary latency, duplicated-context ratio, coordination-error rate

IF   quality gap < 3.0 F1 points          -> ship the loop. Cost and p95 decide.
IF   quality gap >= 3.0 points            -> the gap must (a) survive replication and
                                             (b) fail to localise to one portable stage
                                             before it justifies the topology.
IF   the gap localises to one stage        -> port that stage into the loop, re-run.

Cost is a primary term in every branch. There is no "defer cost" branch.

The 3.0-point threshold is a judgement made when nobody knows which way it will go, which is the only time it can be made honestly. Set it after seeing a 4.0-point result and it becomes 3.5; set it after seeing a 2.0-point result and it becomes 2.5.

Why "cost is next quarter's problem" ends the argument permanently#

The tempting reading of a result where the pipeline wins on quality by four points and loses on cost by 3x and on p95 by six seconds is that quality decides, cost is an optimisation problem, and the optimisation gets scheduled. In practice that sequence is a one-way door, for reasons that have nothing to do with anyone's good faith.

Once the comparison is over, legacy/review_loop.py gets deleted, because keeping dead code that nobody runs is correctly regarded as a liability. From that point the loop is not a design the organisation can choose; it is a rewrite, and a rewrite competes against roadmap features rather than against the pipeline. Nobody schedules it.

The optimisation budget then goes where the code is. The engineer given a quarter to reduce cost will add caching breakpoints, move a stage to a smaller model, and trim the orchestrator's context — the chapter 1 work, which on the payables pipeline took the multiplier from 10.8x to 4.3x and was genuinely worth doing. What that engineer will not do is ask whether the three stages should exist, because that question is no longer answerable: the alternative has been deleted, and the measurement that would have supported the question was taken eight months ago under conditions nobody recorded.

The observable version of this is simple enough to check on any team. Ask when the cheaper alternative was last executed end to end. If the answer is "before the migration", the cost conversation has already been decided.

None of this says the four-point quality gap should be ignored. It says the gap has to be established before it is spent, and a gap measured once, on one pass over one task set, is not established — the same arm run twice against itself on this workload produced scores that differed by more than four points on several of the replicates. That is where the next chapter starts.