A four-point quality win can be run-to-run variance

The contract-review comparison ran eight times per arm, on the same 240 contracts, with the back-ported loop and the production pipeline both on claude-opus-5 at high effort and both loading the shared prompt pack. Obligation F1, blind-scored by two reviewers with a third resolving disagreements:

The four-point headline is the difference of the two means, and it is smaller than the spread of the loop against itself. Replicate 4 has the loop at 74.8 and the pipeline at 74.4; a team that had run each arm once and happened to draw that pair would have written "the loop matches on quality and costs a third as much" and been done. A team that drew replicate 6 against replicate 3 would have reported a 7.9-point pipeline win. Both write-ups would have been produced from the same system, the same task set and the same prompts, differing only in which afternoon the job ran.

Nothing about that variance is pathological. Adaptive thinking, where the model settles for itself how far to reason within the effort budget it was given, makes different depth decisions run to run, tool results arrive in different orders when calls run concurrently, and a 240-task suite contains perhaps thirty contracts sitting near a scoring boundary where a single obligation classified either way moves F1 by a third of a point. A single pass over a task set of this size gives you a number with a standard error of about 1.7 points — that being how far the figure would be expected to move if you simply ran the same arm again — and 1.7 points of noise is a substantial fraction of most architecture claims.

Pairing by task, and bootstrapping over tasks#

Treating the sixteen replicate means as sixteen independent observations throws away the most useful structure in the data, which is that both arms saw the same contracts. Contract difficulty is by far the largest source of variation in the raw scores — some agreements have four obligations and some have ninety — and pairing removes it.

The unit you want to generalise over is contracts, not replicates, because the question is "would this gap hold on the next 240 contracts", not "would it hold on the next Tuesday". That makes a cluster bootstrap over contracts the right resample. A bootstrap estimates how far a number would move under a different sample by recomputing it many thousands of times over draws taken with replacement from the data already in hand, and the cluster version draws whole contracts: keep all of a drawn contract's replicates together, and recompute the paired difference.

import random
from statistics import mean

def paired_gap(scores: dict[tuple[str, str, int], float], tasks: list[str]) -> float:
    """scores[(arm, task, replicate)] -> per-task F1. Mean over tasks of the
    within-task difference, each task's difference averaged over its replicates."""
    diffs = []
    for t in tasks:
        a = [v for (arm, tt, _), v in scores.items() if arm == "pipeline" and tt == t]
        b = [v for (arm, tt, _), v in scores.items() if arm == "loop" and tt == t]
        diffs.append(mean(a) - mean(b))
    return mean(diffs)


def cluster_bootstrap(scores, tasks, n=10_000, seed=7) -> tuple[float, float, float]:
    """Resample TASKS with replacement, keeping each task's replicates intact."""
    rng = random.Random(seed)
    point = paired_gap(scores, tasks)
    draws = sorted(
        paired_gap(scores, [rng.choice(tasks) for _ in tasks])
        for _ in range(n)
    )
    lo = draws[int(0.025 * n)]
    hi = draws[int(0.975 * n)]
    return point, lo, hi

On this data the paired point estimate is 4.00 F1 points with a 95% interval of [2.6, 5.3]. The gap is real — the interval excludes zero comfortably — and it straddles the 3.0-point threshold the team registered in chapter 4 before running anything. The pre-registered rule therefore does not resolve, which is the correct outcome for evidence in that shape and is exactly why writing the rule down early is worth the awkwardness. Nobody got to pick the threshold after seeing where the interval landed.

There is a cheaper diagnostic worth running alongside the bootstrap, and it costs nothing: split each arm's own replicates into two halves and compare them. The loop's replicates 1–4 average 72.6 and 5–8 average 71.7, a within-arm gap of 0.9 points from a system compared against itself. That number is the resolution floor of the harness. A reported effect smaller than it is not an effect.

How many replicates buy how much resolution#

Eight replicates per arm cost about $2,900 in model spend and most of a day of wall-clock, which is a real budget item and prompts the obvious question of whether four would have done. The standard error of a mean scales as one over the square root of the replicate count, so the resolution of a comparison improves slowly: going from four replicates to eight narrows the interval by about 29%, and going from eight to sixteen narrows it by another 29%.

That relationship makes the sensible planning question "what gap do I need to be able to see" rather than "how many runs can I afford". With the loop's per-replicate standard deviation of 1.72 and the pipeline's 1.17, four replicates each gives a 95% interval on the difference of roughly ±2.5 points, eight gives ±1.6, and sixteen gives ±1.1. If the decision threshold is 3.0 points, four replicates leaves an interval wide enough that almost any observed gap is ambiguous, and sixteen buys half a point of resolution for another $2,900.

There is a cheaper lever than replicates, which is the task set. Doubling the number of contracts from 240 to 480 narrows the interval by about the same 29% as doubling replicates, and costs one extra pass rather than eight. It also improves the thing you actually want to generalise over, since the population of interest is contracts. On a fixed budget, more tasks and fewer replicates is usually the better trade — with a floor of at least three replicates per arm, because without them you cannot estimate the within-arm variance at all and the resolution floor stays unknown.

Where the four points came from#

An interval says the gap exists. It does not say what produced it, and "the topology" is not an answer — it is the absence of one. The next step is to build the set of tasks where the arms actually disagree and look at what differs there.

Disagreement has to be defined so that noise does not enter the set. A contract counts as a disagreement when the two arms' per-task scores differ by at least 5 F1 points and the sign of that difference is the same in at least seven of the eight replicates. That second condition is what keeps out contracts that happen to sit on a scoring boundary; by construction, a boundary contract flips sign across replicates.

Of 240 contracts, 61 met both conditions. The pipeline was better on 44 and the loop on 17. Tabulating the 44 by what was actually missing from the loop's output:

Thirty-one of the 44 are the same failure: an obligation in a late clause, in a contract long enough that the loop reached it with a context already holding several hundred extracted obligations and forty pages of source text. The pipeline's extract stage does not have that problem, because it starts fresh with the classify stage's clause map rather than the accumulated conversation.

That is a mechanism, and it is one of the mechanisms instrumented in chapter 4: the loop's advantage on duplicated context is also, on long documents, a disadvantage on attention. Notice that it is not an argument for three agents. It is an argument for not carrying four hundred prior obligations into the turn that reads page 47.

The 17 contracts where the loop won are worth the same treatment and are usually skipped. Eleven of them are cases where an obligation in section 3 modifies one in section 12, and the loop caught the interaction because both were in the same context — the exact property that hurt it on the other 31. This is the shape a real trade-off has, and a summary that reports only "the pipeline is four points better" has averaged it away.

Porting the mechanism instead of the topology#

If 31 of 44 disagreements come from context load at extraction time, the intervention is to extract in chunks with a compacted running state, inside one loop, and re-run the comparison. That took a day and a half.

The chunked loop scores 75.6 mean over eight replicates, standard deviation 1.3. The paired gap against the pipeline is now 0.55 points with a 95% interval of [−0.8, 1.9] — indistinguishable at this harness's resolution, and below the 0.9-point within-arm floor measured above. Cost per successful review is $1.52 against the pipeline's $4.10, and p95 is 44 seconds against 71.

Nothing about the chunked loop is a compromise design. It is the same single conversation with a different extraction strategy, and it inherits none of the boundary latency, none of the cross-agent duplication, and none of the coordination-error surface. The four points were never a property of having three agents; they were a property of one thing the three-agent design happened to do, which any design can do.

The general form of this move is worth stating, because it is where most multi-agent quality claims end up. A stage whose advantage is isolation of context is almost always portable, because a loop can clear or compact its own context between phases. A stage whose advantage is a different model, tool set or permission scope is portable at the cost of a mid-conversation switch, which invalidates the prompt cache and is a real trade. A stage whose advantage is genuine concurrency — several independent sub-tasks running at once — is the one case where the boundary is doing work no single sequential conversation can do, and that case is about latency rather than quality.

A statistically real gap and a decision are different things#

The 4.00-point gap with an interval of [2.6, 5.3] is a real effect by any reasonable standard, and on its own it does not tell the team to keep the pipeline. Effect sizes and decisions live in different units, and translating between them requires the cost side.

Four F1 points on obligation extraction, on this service, means roughly one additional missed or spurious obligation per eleven contracts. The legal team reviews every summary before acting on it and catches most extraction errors, so the cost of a miss is a few minutes of a lawyer's time rather than a contractual exposure — measured over a quarter of tracked review sessions, about 4.5 minutes per affected contract. At 3,800 contracts a month, four points is roughly 26 hours of lawyer time a year. The cost gap is $118,000 a year. Those two numbers are not close, and no amount of statistical confidence in the four points changes that.

On a different service the same four points would decide the other way. If the summaries were acted on without review, or if a missed indemnity obligation surfaced in litigation rather than in a lawyer's inbox, the expected cost of a miss would be several orders of magnitude larger and $118,000 would be cheap. The arithmetic is the same; only the consequence term changes. This is why the pre-registered rule in chapter 4 put a numeric threshold on the quality gap rather than a significance test: significance answers "is this gap real", and the decision needs "is this gap worth 3x".

What localisation cannot promise#

Thirteen of the 44 disagreements did not localise, and three had no pattern at all. A residue like that is normal and it is not evidence for the topology; it is evidence that the analysis has resolution limits, the same way the harness does. Reporting "31 of 44 localise to context load at extraction" is honest. Reporting "the gap is explained" is not.

The slicing itself is a hazard. A disagreement set of 61 contracts can be cut by clause type, page count, contract value, supplier, jurisdiction, and length, and cutting it six ways will produce a striking-looking pattern by chance alone. The defence used here is that the pattern was stated as a hypothesis with a mechanism attached — late clauses, loaded context — and then tested by making a change and re-running both arms. A pattern that survives an intervention is a finding. A pattern found by slicing is a hypothesis.

And a gap that fails to localise is not thereby attributed to the topology. It may be a mechanism nobody instrumented, which on this comparison would have included anything about how the orchestrator phrases its briefs, since that was never varied. "We could not explain it" and "it is architectural" are different statements, and the second is a claim that costs money.

What remains after all of this is the argument that does not go away when the quality gap does. The pipeline's supporters on this team did not, in the end, argue that it was more accurate — the chunked loop settled that. They argued that each agent has its own trace, that debugging the pipeline is far easier than debugging a five-turn conversation, and that they were not willing to give that up. That claim is true, and it is the subject of the next chapter.