Move one thing at a time or the comparison means nothing

The message in the channel reads: "Three-stage chain beats the single prompt by six points on the eval set — 87 vs 81. Proposing we migrate everything." The evaluation set is the same 240 cases in both runs, the grader is the same, and the numbers are not made up. Reading the pull request that produced the chain, two other things moved. The chain runs on a larger model, because someone upgraded the default while building it. And the shared system prompt was rewritten during the experiment, picking up a new output schema and four worked examples that the old prompt did not have.

Six points were measured. What produced them is unknown, because three things changed at once and the evaluation cannot tell them apart. The six could be entirely the model, entirely the prompt, entirely the shape, or any mixture, including a mixture where one factor helped by eight and another hurt by two.

What has to be held, and what cannot be#

The response people reach for first is that the comparison was controlled, since both arms ran on the same suite. That belief is worth taking apart slowly, because it is the characteristic mistake in this area and it is not carelessness — something really was held fixed, and holding it fixed really did remove a source of error. Holding the evaluation set constant controls for exactly one thing: the difficulty and mix of the inputs. It guarantees the two systems were asked the same questions. It says nothing about which property of the system produced the difference in answers, and every property on the system side moved simultaneously. The quantity that was pinned sits on the input side of the experiment; the quantity that decides what the number means sits on the system side and was free to vary. A control that holds one side steady while the dangerous variation happens on the other is not a control for the claim being made.

A comparison that supports a conclusion about topology needs everything except topology pinned:

Three rows in bold is three candidate explanations for one number. The fix is an ablation: run the intermediate arms, each arm being one version of the system that differs from the version below it in a single factor, so that each factor is priced separately. This is the same manoeuvre used to attribute gains in a model paper, where removing one component at a time is the only way anyone can say which component did the work, and it costs about what one extra evaluation run costs.

# experiments/chain-vs-single.yaml
arms:
  - id: A0            # baseline as it shipped
    topology: single
    model: small
    prompt: v11
  - id: A1            # + model only
    topology: single
    model: large
    prompt: v11
  - id: A2            # + model + prompt
    topology: single
    model: large
    prompt: v14
  - id: B             # the proposal
    topology: chain-3
    model: large
    prompt: v14
common:
  eval_set: sets/240-v3.jsonl
  grader: rubric-v3
  temperature: 0.2
  repeats: 3          # per case, to see spread not just the mean
  pairing: by_case_id # compare per-case outcomes, not just aggregate scores

Read the four arms as a ladder rather than as four systems. Each rung differs from the one below it in exactly one factor, so the difference between two adjacent scores has only one thing in it that could have caused it. That is the whole mechanism, and everything the ablation can and cannot say follows from it.

Four arms rather than two, and on this project each arm was roughly 240 cases × 3 repeats × 4 model calls, which came to about 45 minutes of wall clock and $11 of inference. The result:

The information in that table is in the gaps between consecutive rows and not in the four levels. Subtracting each score from the one above it, the six points decompose into 3.8 from the model, 1.4 from the prompt, and 0.7 from the topology, and the 0.7 arrives with three times the model calls. Those three differences have to add back to the top-to-bottom gap, and checking that they do is worth thirty seconds: the middle terms cancel, since the score of A1 is added once and subtracted once, and the same for A2, leaving B's score minus A0's. Three point eight plus one point four plus nought point seven is five point nine, and B minus A0 is eighty-seven point one minus eighty-one point two, which is also five point nine. The headline six points came from the rounded scores in the channel message.

What that identity licenses is a decomposition, and it is worth being precise about the residual, because the same arithmetic works for any ordering of the rungs and the answers are not the same. Each rung measures its factor in the presence of the factors below it. The one point four for the prompt is what the prompt rewrite is worth on the large model, and the nought point seven for the chain is what the chain is worth on the large model with the rewritten prompt. A ladder that added the chain first and the model last would also sum to five point nine and would attribute the terms differently.

Two lines in the common block are doing quiet work and are worth reading before the results. The three repeats exist because a run is not deterministic even at a low temperature: the same case can pass on one draw and fail on the next, so a single run reports one sample from a distribution and three runs report a mean plus some idea of how far apart the draws sit. The pairing line says that the comparison is made case by case rather than between two aggregates, which is the property from chapter 2 that lets a difference be read as a list of specific cases.

What the team wanted to migrate everything for was the model upgrade, which they could have shipped in an afternoon by changing one line, on a design with a third of the calls. That is the ordinary outcome of these ablations rather than a contrived one; prompt and model effects are usually large and topology effects are usually small, which is precisely why bundling them flatters the topology.

That asymmetry has a mechanism behind it rather than being a rule of thumb to memorise. A model upgrade changes what happens inside every call the system makes, and so does a prompt rewrite, since both alter the instructions and the capability applied to each unit of work. A topology change alters none of that. The same model reads the same instructions; what moves is how the work is divided between calls and what context each one is given. So a topology change can only help where the division itself was the problem — a task too long for one context, a stage whose output the next stage needed in a stricter form — and it can only help by as much as that division was costing. On a task where one call already does the job adequately, there is nothing for the rearrangement to recover.

One thing genuinely cannot be held constant, and pretending otherwise is worse than naming it. A three-stage chain has three system prompts and a single prompt has one, so "same prompt" can only mean same instructions, same examples, same output schema, same tone constraints, distributed across the stages. There is judgement in that distribution, and it is a residual confound — a factor that moved alongside the shape and cannot be separated from it afterwards — which you disclose rather than eliminate.

The mitigation that matters most is fairness to the simple arm, and the mechanism behind that is worth spelling out because it is easy to read as a fairness slogan. Prompt iteration is a search: you look at outputs that are wrong, form a theory about why, and edit. When the outputs you look at come from a chain, the wrong answers you see are the ones a chain produces, so the edits you make are repairs to a chain's failure modes — an instruction that keeps stage two from restating stage one, an output schema shaped by what the third stage needs to parse. Those edits are not neutral improvements that happen to have been discovered on the chain; they are fitted to it. A prompt that was tuned while staring at chain traces will contain assumptions that suit the chain, so give the single arm the same tuning effort before you declare it beaten. On the run above, A2's prompt was given two rounds of iteration by the same engineer who tuned B's, which is why A2 is the honest comparison point and A0 is only history.

The confounds that are not in the diff#

Model, prompt and shape are the three that a pull request makes visible. Three more move without appearing in any diff, and each of them has cost a team a conclusion.

The retrieval index is the most common. If the corpus was re-ingested with better chunking during the three weeks the candidate was built, the candidate is reading better context than the baseline ever did, and the effect can be several points on a retrieval-backed task. The check is a one-line assertion that both arms resolve the same index snapshot ID, and the fix when they do not is to re-run the baseline against the current index rather than to argue about how much it mattered.

The grader is the second. Rubrics get edited while people look at outputs, usually for good reasons, and a rubric that gained a clause about citation formatting halfway through the experiment scores the two arms by different standards. Version the rubric, record the version in the report, and treat a rubric change as invalidating every score taken under the old one.

The evaluation set itself is the third, and the failure is subtle. Cases get added while the candidate is being built, and they get added because someone noticed the candidate failing them, which means the set has been enriched with cases selected for being hard for the candidate — a bias that pushes against the arm you are trying to promote — or, more often, cases the candidate was then tuned on, which pushes the other way and is worse. Freeze the set for the duration of the comparison and keep new cases in a separate file that the next comparison uses.

The common property of all three is that they are state outside the experiment's control which both arms resolve by name at run time. That is also what makes them cheap to catch: a name resolved at run time can be recorded at run time, and two recorded names can be compared before anybody reads a score.

def assert_comparable(a: RunReport, b: RunReport) -> None:
    for field in ("eval_set_sha", "grader_version", "index_snapshot", "tool_schema_sha"):
        if getattr(a, field) != getattr(b, field):
            raise Incomparable(f"{field}: {getattr(a, field)} != {getattr(b, field)}")

Twelve lines of that in the harness turn a class of unfalsifiable claim into a failed assertion, and the assertion fires often enough in the first month to justify itself.

Where one factor at a time is not required#

It is worth being clear about where this discipline applies, because applied everywhere it would stop a team from shipping. If your goal is simply to make the product better this month, bundling a model upgrade with a prompt rewrite and a new shape is a perfectly good way to work: you get the sum of the effects and you did not have to pay for four evaluation arms. One-factor-at-a-time is required only when the conclusion you want to draw is about one factor — when someone is about to standardise on a topology, or rewrite three other features to match, or defend the coordination cost in a review. The cost of the ablation is small; the cost of migrating four features onto a shape that contributed 0.7 points is not.

The ablation above also has a known blind spot, and it is the residual named earlier arriving as a practical limit. Running arms as a ladder — baseline, plus model, plus prompt, plus shape — measures each factor only in the presence of the ones below it, so it cannot see interactions. It is entirely possible that the chain is worth four points on the small model, where splitting a hard task into narrow stages compensates for weaker instruction-following, and worth 0.7 on the large one. Detecting that needs the cells you skipped, and the full factorial for three factors is eight arms rather than four, because each factor has two settings and every combination has to be run: two settings of the model times two of the prompt times two of the topology. Running eight is not usually worth it, but knowing which conclusion you are not entitled to is: the honest claim from the ladder is "on the large model with prompt v14, the chain is worth 0.7 points", and not "chains are worth 0.7 points".

Even a clean 0.7 leaves the decision open, because a point of quality has a price and nobody has quoted it yet. The price is not only the extra inference. It is what the coordinated design does at three in the morning when one of its parts returns something unexpected, and that bill has line items most teams have never written down.