The churn baseline

A clean ablation gives you a difference. It does not tell you what difference is distinguishable from zero, and the two are separate measurements. Nine cases worse and six better on a 500-case set is a difference; whether it is a finding depends on how many cases that same harness moves when you change nothing at all.

The name for that quantity here is the churn rate: the fraction of an eval set that changes its mind between two runs of an identical configuration. It exists because a language model's sampling is not reproducible even at temperature 0, and because most eval harnesses have three or four other sources of movement layered on top of that. Measuring it costs one extra run and turns every per-case delta from a story into arithmetic.

Running arm A against arm A#

The measurement is one command and no new machinery. Run the identical arm twice, join the two per-case outputs on case_id, and count the cases whose score differs.

import json
from collections import Counter

def load(path: str) -> dict[str, float]:
    rows = (json.loads(line) for line in open(path))
    return {r["case_id"]: r["score"] for r in rows}

def churn(run_a: str, run_b: str) -> Counter:
    a, b = load(run_a), load(run_b)
    assert a.keys() == b.keys(), "runs cover different cases; the set moved"
    c = Counter()
    for case_id in a:
        if b[case_id] > a[case_id]:
            c["up"] += 1
        elif b[case_id] < a[case_id]:
            c["down"] += 1
        else:
            c["same"] += 1
    return c

# support-eval-v3, arm A run twice, 2025-08-04 and 2025-08-05
# Counter({'same': 489, 'up': 6, 'down': 5})

The measurement has to be a per-case join and not a difference of two aggregates, and the reason is the same one the next chapter is entirely about. A difference of aggregates would be near zero here, because the six cases that went up and the five that went down almost cancel; the count of cases that moved is eleven regardless of direction. Those two numbers answer different questions, and the one you need is the second. The assert on the two key sets is guarding the same property: if the two runs cover different case IDs, some case is present on one side and absent on the other, and the loop would silently compare a case against nothing. That is not a small run difference — it means somebody edited the set between the two runs, and the floor you were about to measure would be a measurement of the edit.

Without that floor, the readings in the rest of this topic are unfalsifiable in the strict sense: any per-case result at all can be told as a story about the removed block, because there is no quantity that the story could have failed against. The floor is what a claim can fail against.

The name for this in web experimentation is an A/A test, and it was invented for a related purpose: before trusting an A/B result, you run two identical variants through the splitter and check that the reported difference is around zero. If your A/A comes back with a significant lift, the fault is in the experiment infrastructure — a broken randomiser, a logging skew, a cache that serves one arm faster. LLM evaluation borrows the idea and uses it differently. Here the A/A is not expected to come back at zero, because model sampling is genuinely non-reproducible, and the useful output is not a pass or a fail but a number: the size of the noise floor you must clear. On the support set above it is 11 cases out of 500, or 2.2%, and everything that follows is arithmetic against that number.

Two conversions make that figure usable in the chapters that follow, and the second is the one people skip. Eleven cases out of five hundred is eleven divided by five hundred, which is the percentage quoted. Going the other way matters more, because eval tables report score differences rather than case counts: with a judge that returns a pass or a fail, one hundredth of the zero-to-one scale is one hundredth of five hundred cases, which is five cases. So a floor of eleven cases is a shade over two points of aggregate score, and any reported difference smaller than that is a difference this harness produces against itself.

Where the eleven cases come from#

The churn is not one phenomenon. Instrumenting the eleven cases individually — re-running each one ten times in isolation and watching which stage varies — splits them into sources with quite different fixes and quite different costs.

The judge being the largest single source surprises people, and it is the cheapest to attack. Half of those five cases sat on rubric language like "the answer is appropriately hedged", which is a judgement the judge itself makes inconsistently on borderline text. Rewriting that criterion as two checkable sub-questions and running the judge three times with a majority vote took judge-driven churn from five cases to one, at a cost of two extra judge calls per case — about $4 on a 500-case run, since the judge reads roughly 1,200 tokens per case and writes 80.

The last two rows are worth attacking even though they account for two cases, because both are bugs wearing the costume of noise. A case that truncates at the answer budget is not producing a worse answer, it is producing an answer your harness cut off, and if the ablation you are about to run removes 1,500 tokens of prompt it may quietly change how many cases land near that boundary. Logging a truncated boolean on every per-case row and reporting the count alongside the score turns that from an invisible source of movement into a number you can look at. The retrieval retry is the same shape: an ablation is not supposed to be measuring your vector index's tail latency, and caching retrieval per case removes the variance and the confusion at once.

Repeats buy variance reduction at a price you can compute#

The remaining churn is model sampling, and the only lever there is repetition. Run each arm n times, take the per-case majority or the median for graded scores, and the noise shrinks roughly as the square root of n while the cost grows linearly with it.

from statistics import median

def majority_score(scores: list[float]) -> float:
    """Binary judge: majority vote. Graded judge: median of the repeats."""
    if set(scores) <= {0.0, 1.0}:
        return 1.0 if sum(scores) * 2 > len(scores) else 0.0
    return median(scores)

The square root is not a rule of thumb, it is where the arithmetic lands. Treat each repeat of a case as an independent draw around the same underlying quality. Averaging several such draws produces a quantity whose variance is the original variance divided by the number of draws, because independent variances add and the average divides by the count twice over — once in the numerator and once in the squaring. Spread is the square root of variance, so dividing variance by the number of repeats divides the spread by the square root of it. Four repeats halve the spread. Nine repeats cut it to a third. The cost, by contrast, is one full run of the set per repeat, so it multiplies straight up, and the ratio of what you buy to what you pay gets worse with every repeat added.

Two things stop that being a formula you can predict the table from. The first is that a majority vote is not an average: on a binary judge, three repeats do not shrink the wobble on a case, they correct it outright whenever the true verdict wins two of the three, which is a stronger move than averaging and beats the square root at small counts. The second is that only part of the floor is model sampling at all. The rows above that are truncation and retrieval retries do not shrink with repeats, because they are not random draws around a stable answer — they are the same bug firing on the same case each time. That is why the table below has a floor it approaches rather than a curve toward zero.

The numbers on the support set, with the judge already fixed at three votes:

The cost per arm comes from the prompt size: 500 cases at roughly 14,000 input tokens each is 7.0M input tokens, which at $3 per million is $21, plus 500 answers of about 250 output tokens at $15 per million, which is $1.88, plus the three judge calls. Going from three repeats to five costs another $54 per arm and buys one case of floor reduction, which is the point at which the curve stops being worth walking. Three repeats is the setting most teams land on, and the reason is visible in the table rather than in anybody's preference.

There is a shortcut that looks like it drives churn to zero and does not. If you cache the model's response per case and replay it for both runs, the A/A comparison is exactly identical every time. What you have measured is the cache. The moment either arm changes the prompt — which is the entire point of an ablation — the cache key changes and you are back to a live call with its original variance, now unmeasured. Response caching is a legitimate tool for saving money on unchanged arms; it is not a way to make the floor go away.

Discordant pairs carry the signal#

With a floor in hand, the comparison between two arms becomes a specific question about the cases that disagree. Cases scoring the same in both arms contribute nothing either way. The information is entirely in the discordant pairs: on the classifier ablation the next chapter takes apart, 420 of 500 cases scored identically, 41 improved when the block came out and 39 got worse.

Why the four hundred and twenty carry nothing is worth being explicit about, because it is counter-intuitive that most of the data is discarded. Ask what a case scoring the same in both arms could tell you about the removed block. If it passed both times, either the block was irrelevant to it or the block helped in a way that made no difference to the verdict; if it failed both times, the same in reverse. Either way the case is compatible with the block helping, hurting and doing nothing, so it cannot move the answer in any direction. The cases that changed are the only ones whose behaviour depends on the block being there. The question therefore reduces to a much smaller one: among the cases that moved, did more of them move down than up, by more than a coin would manage?

The standard test for exactly this shape of data is McNemar's test, which asks whether the split between the two kinds of discordant pair is more lopsided than chance would produce and reports the answer as a p-value: the probability of seeing a split at least this uneven if the removed block made no difference at all. With a continuity correction, the statistic is (|b − c| − 1)² / (b + c), where b and c are the two discordant counts.

from math import sqrt
from statistics import NormalDist

def mcnemar_p(improved: int, worsened: int) -> float:
    n = improved + worsened
    if n == 0:
        return 1.0
    chi2 = (abs(improved - worsened) - 1) ** 2 / n
    z = sqrt(chi2)
    return 2 * (1 - NormalDist().cdf(z))

mcnemar_p(41, 39)   # 0.911 — the split is what a coin would give you
mcnemar_p(62, 18)   # 0.0000015 — the split is not

Both calls are worth walking by hand, because the formula is three operations and the two results could hardly be further apart. In the first, the two discordant counts differ by two; the continuity correction takes one off that, leaving one; squaring leaves one; dividing by the eighty cases that moved gives a statistic close to zero, whose square root is about a tenth of a standard deviation. A departure of a tenth of a standard deviation from an even split is what you get most of the time by flipping eighty coins, which is what the reported p-value says. In the second, the counts differ by forty-four, the correction leaves forty-three, squaring that gives a number in the low thousands, and dividing by eighty puts the statistic in the twenties. Its square root is near five standard deviations from an even split, and five standard deviations is where p-values acquire leading zeros.

The subtraction of one before the squaring is the continuity correction, and it is there because the formula compares a count, which moves in whole steps, against a continuous curve. Without it, small samples produce statistics that are slightly too large and p-values that are slightly too confident. Subtracting one shrinks the gap by the smallest step a count can take, which is the conservative direction to err in.

Both of those calls describe 80 cases changing behaviour. The first says nothing about which direction the block pushes the system; the second says the block is clearly harmful on the cases it touches. The test is only about direction, which is why it does not replace the reading the next chapter does: 80 cases moving is a large behavioural effect in both examples, and the p-value is silent about that.

Stability is not correctness#

The churn baseline is a statement about stability, not about correctness. A judge that is reliably wrong produces a beautifully low churn rate, because it is wrong the same way every time; a set that under-samples an important query shape churns no more than a set that covers everything. Neither defect shows up here, and both will hurt you more than sampling noise does. The last chapter of this topic comes back to the coverage half of that.

The floor also has a shelf life, which is why it belongs next to the arms rather than in somebody's memory. A model snapshot change, a judge rubric edit, a new answer budget and a larger eval set all move it, and each of those happens on a normal week. Re-measuring quarterly, or whenever the model pointer moves, is enough; the number is stable between those events in a way the individual case outcomes are not.

It is also specific to the arm you measured it on. Churn scales with how many cases sit near the judge's decision boundary, and an arm that produces longer, hedgier answers pushes more cases into that region. When a proposed variant changes the answer style substantially — a different output format, a much larger answer budget — the honest move is to measure the floor again on that arm rather than assume the number transfers. Measuring it on both arms costs one extra run and occasionally reveals that the variant is not just different but noisier, which is itself worth knowing before it ships.

Finally, the floor sets a minimum detectable effect and nothing beyond it. Knowing that 11 cases move for free tells you that a 9-case swing is unreadable and a 60-case swing is not; it does not tell you what a readable swing means. A floor is the precondition for reading a delta, not a substitute for reading one, and the reading is where the interesting failures live.

The ablation this chapter opened with had a clear resolution once the floor existed: run more repeats, or accept that the block's effect is smaller than this harness can see. The case in the next chapter has no such escape, because 80 of 500 cases changed behaviour and the aggregate did not move at all.