A held-out set too small cannot resolve two points

A noise band is the range a measurement wanders over when the thing being measured has not changed. Everything in this topic that looks like a decision — whether memory helps, whether a tuning quarter paid off, whether a slice is really different from the rest — is a comparison between two numbers, and a comparison between two numbers means nothing until the wandering underneath them has a size. The previous chapter's comparison, 152 successes against 148 out of 200 held-out tasks, is four tasks wide. This chapter is about establishing what four tasks is worth.

Two quantities do that work. The first is the run-to-run standard deviation, written SD, which is the typical distance between one run's score and the average of several runs of the same thing. The second is the confidence interval around the difference between the two arms, which is the range of true differences that would not be surprising given what the run produced. They answer different questions — the first is a property of the apparatus, the second is a statement about the effect — and a team that has one and not the other will still argue in circles.

Where the movement comes from#

Someone puts the 76-versus-74 result in a slide and writes "memory improves task success by 2 points" under it. Before that slide goes anywhere, run the memory-on arm twice more on the same 200 tasks, changing nothing at all. The three runs give 152, 149 and 154 successes, or 76.0%, 74.5% and 77.0%. The memory-off arm, run three times, gives 148, 150 and 147, or 74.0%, 75.0% and 73.5%. Each arm moves across a range of about two and a half points while the code, the data and the model are identical, so a two-point gap between arms is well inside what the harness produces when nothing happens.

Turning that spread into a single number is the standard deviation, and it is worth computing once by hand rather than accepting from a library. Take the three memory-on scores, find their average, and measure how far each one sits from it: one lands about a point and a third below, one about a point and a sixth above, and the third almost exactly on it. Square those distances, so that a miss in either direction counts as a miss and large misses count for more than small ones. Average the squares, then take the square root to get back to the units the scores were in. The standard deviation (SD) of a single arm across those three runs — the typical distance between one run's score and the average of the runs — is about 1.3 points. That number is a property of the measuring apparatus, and it has three separable sources.

Each source can be isolated by freezing it and re-measuring the spread of the remaining ones. Freezing sampling means dropping temperature — the parameter governing how much randomness enters the model's choice of each token — to 0 and fixing the random seed, so that the model takes its highest-scoring option every time; freezing retrieval means replacing the approximate-nearest-neighbour (ANN) index, which trades a little accuracy for speed by not comparing the query against every record, with exact search and a deterministic tie-break on record id; freezing the grader means scoring frozen traces once and reusing the verdicts. The contributions below are given in percentage points (pp), which are absolute differences in the score rather than relative ones, so a move from 74% to 76% is 2 pp.

Those three contributions do not add up to the observed 1.3, and the way they do combine is the check that the accounting is complete. Add them arithmetically and they come to two full points, which is more movement than the harness actually shows. The reason arithmetic addition overshoots is that these three sources vary independently of each other: on any given run, sampling might push the score up while the grader pushes it down, so they cancel as often as they reinforce. Independent sources add by their squares rather than by their sizes — square each contribution, add the squares, take the square root of the total, which is called combining in quadrature — and that gives 1.2 points. The observed 1.3 sits next to 1.2 and a long way from 2.0, which says the three measured sources are very likely the whole story rather than three of five.

The retrieval row is the one that surprises people, because index lookups feel like the deterministic part of the system. An HNSW index — hierarchical navigable small world, the graph structure most vector stores use for approximate search — walks a graph from an entry point towards the query, and which candidates it visits depends on where it entered. Two candidates whose scores are nearly identical can therefore come back in either order on two runs of the same query. That reordering is harmless while both candidates are inside the injected set and decisive at the edge of it: when the difference is between rank 3 and rank 4 with k=3, one of them enters the prompt and the other does not, and the model sees a different context on the second run.

The grader row is a reminder that the model-graded portion of a success criterion carries its own variance on top of the system under test. The verdict on a trace is itself a model output, so re-grading the same trace can return a different verdict, and that movement is added to whatever the system did. This is why the deterministic checks described in the previous chapter are worth keeping deterministic even when a model could do the same job: a comparison against a stored expectation returns the same answer every time, so it contributes exactly the 0 pp in the last row.

Reducing these is possible up to a point and not always desirable. Temperature 0 removes most of the sampling contribution and also removes the behaviour you actually ship, so an eval at temperature 0 measures a system nobody uses. Exact search is fine on a 4-million-record store in an offline harness and unaffordable online. Freezing the grader is free and worth doing. What remains after the cheap fixes is the floor, and the sensible move is to measure that floor rather than to keep chasing it.

Pairing is what makes 200 tasks worth anything#

Comparing 76% against 74% as two independent proportions throws away the fact that both arms ran the same 200 tasks. Most of the variance in a task set is between tasks — some are hard, some are trivial — and that variance is shared by both arms, so subtracting within each task removes it.

The algebra behind that sentence is short and worth doing. The variance of a difference between two quantities is the variance of the first, plus the variance of the second, minus twice how much the two move together. When the two arms run different tasks, they move together not at all, so nothing is subtracted and both task-difficulty variances stay in the total. When they run the same tasks, a task that is hard in one arm is hard in the other, so the two arms move together strongly, and the subtracted term is large. Everything the two arms share cancels, and what is left is the part where they differ, which is the only part carrying information about memory.

The paired view of the same data is a 2×2 table of outcomes.

The 176 concordant tasks carry no information about the difference between arms, because they came out the same way both times. Write each task's contribution as its memory-on outcome minus its memory-off outcome, counting a pass as one and a fail as zero. A task in either concordant cell contributes zero exactly: one minus one, or zero minus zero. A task in the top-right cell contributes plus one, and a task in the bottom-left contributes minus one. Summing over all 200 tasks therefore gives 14 minus 10, and dividing by 200 gives the paired difference of +2.0 pp. The 176 tasks never entered the arithmetic at all. All the evidence lives in the 24 discordant tasks, split 14 in memory's favour and 10 against. McNemar's test, the standard test for paired yes-or-no outcomes, asks whether a 14-to-10 split of 24 coin flips is surprising, and it is not.

The uncertainty comes out of the same picture. Suppose memory changes nothing, and that the 24 tasks whose outcome differs between the arms differ for reasons that have nothing to do with which arm they were in. Then each of those 24 lands plus one or minus one with equal chance, like a coin, and the concordant tasks stay at zero however the coins fall. A sum of that many independent plus-and-minus-one draws has a variance equal to the number of draws, so its typical size is the square root of the discordant count. Divide by the number of tasks, because the difference is reported as a share of the set rather than as a raw count, and the standard error of the paired difference is the square root of the discordant count over the total.

from scipy import stats

b, c = 14, 10                      # on-pass/off-fail, on-fail/off-pass
n = 200

result = stats.binomtest(b, b + c, 0.5)          # exact McNemar
diff = (b - c) / n                                # +2.0 pp
se = ((b + c) ** 0.5) / n                         # 2.45 pp, paired
lo, hi = diff - 1.96 * se, diff + 1.96 * se

print(f"diff={diff:+.1%}  p={result.pvalue:.3f}  95% CI [{lo:+.1%}, {hi:+.1%}]")
# diff=+2.0%  p=0.541  95% CI [-2.8%, +6.8%]

Every line of that block is now readable rather than incantatory. se is the expression derived above. The 1.96 is the number of standard errors that covers the middle 95% of a bell-shaped distribution, so multiplying it by the standard error and stepping that far either side of the estimate produces the interval printed in the comment. A 95% confidence interval, which is the range of true differences that would not be surprising given what the run produced, running from a 2.8-point loss to a 6.8-point gain is compatible with memory being mildly harmful and with it being very helpful. That is the honest statement of what 200 tasks bought, and it is worth writing on the slide instead of the 2 points.

What the expression licenses is a shortcut worth carrying: two cells of the table give you the whole interval, and the two large cells are irrelevant to it. A team that doubles the number of trivially passing tasks in its set has not narrowed anything. The residual is that the square-root argument treats the discordant split as a fixed number of coin flips and leans on a bell-shaped approximation, both of which get thin when the discordant count is small — which is why the code also runs the exact binomial test, and why the exact p-value is the one to quote when the two disagree.

Sizing the set against the difference you would act on#

The set size follows from two quantities: the smallest difference that would change a decision, which the sizing formula calls the minimum detectable effect, and the discordance rate — the fraction of tasks whose outcome memory changes in either direction. Discordance is the paired equivalent of variance, and it is the one you have to measure rather than assume; on this set it was 24 in 200, or 12%.

def paired_n(mde: float, discordance: float, power: float = 0.80) -> int:
    z_a, z_b = 1.96, stats.norm.ppf(power)        # two-sided 5%, 80% power
    return int(((z_a + z_b) ** 2 * discordance) / mde ** 2) + 1

paired_n(0.02, 0.12)     # 2352 paired tasks
paired_n(0.05, 0.12)     # 377

That formula is the standard error rearranged, and following the rearrangement is what stops it being a black box. Substituting the discordance rate for the discordant count, the standard error becomes the square root of the discordance rate divided by the number of tasks. An effect is detectable when it is a certain number of standard errors away from zero, and that number is z_a + z_b: z_a covers the false-positive rate you will tolerate, and z_b covers the false-negative rate. Setting the minimum detectable effect equal to (z_a + z_b) times the standard error and solving for the number of tasks moves the square root to the other side, which squares everything on the right: the sum of the two z values gets squared, the discordance rate comes up from under the root, and the minimum detectable effect lands squared in the denominator. That is the expression in the code, line for line.

Two consequences fall straight out of the shape. The set size scales with the inverse square of the effect you want to see, so halving the effect quadruples the set — which is why 5 points is cheap and 2 points is not. And it scales linearly with discordance, so a set where memory changes more outcomes is a cheaper set to run.

The 80% power the formula assumes is a choice rather than a law: it says the experiment should detect a difference of that size, when one genuinely exists, on four runs out of five. Discordance has to come from somewhere before the full set exists, and a pilot of 100 tasks estimates it well enough for sizing, since the formula depends on it linearly rather than sharply. A pilot that comes back with a discordance rate near zero is worth pausing on: it means memory changes the outcome of almost nothing in the set, which is either a true statement about the feature or a sign that the tasks do not depend on stored facts. Distinguishing those two readings is exactly the subject of the next chapter, and a near-zero discordance rate in a pilot is the earliest warning you will get.

Every row of that table is the same formula read backwards, with the number of tasks fixed and the effect solved for, and the inverse-square relationship is visible in it: the jump from 200 to 5,000 is a factor of twenty-five in tasks and a factor of five in resolution.

The unpaired version of the same calculation asks for about 7,350 tasks per arm, or 14,700 in total, to detect the same 2 points. The gap comes from what sits where the discordance rate sits in the paired formula. Unpaired, each arm contributes the variance of a pass-or-fail outcome at its own success rate, which is the success rate times the failure rate; at roughly three-quarters passing that is a little under two-tenths per arm, and two arms make a little under four-tenths. Paired, the only thing in that position is the discordance rate of 12%, which is about a third as large as one arm's variance and about a sixth as large as both arms' together. Pairing buys a factor of six, and it costs nothing except the discipline of running both arms on identical tasks and analysing them as pairs. Teams that build two separate task sets, one "with memory" and one "without", give this up before they start and usually never learn that they did.

Two thousand three hundred and fifty tasks is a lot when each one takes half an hour to build. That number is the argument, not an obstacle to it: if the effect you are hoping for is 2 points and you can afford 300 tasks, the experiment cannot answer the question you are asking, and it is better to know that before spending three weeks on it. The options at that point are to enlarge the set, to accept a larger minimum detectable effect and decide in advance that anything smaller is not worth shipping, or to concentrate the set on the slice where the effect should be largest. The third option is usually the right one, and it works because a set restricted to tasks that genuinely depend on stored facts has a much higher discordance rate — on this product, 31% within the explicit-preference slice against 12% overall — and discordance sits in the numerator of the sizing formula.

When flat is an answer rather than a shrug#

An interval of [-0.2, +4.2] does not establish that memory helps, and it also does not establish that memory is useless, because a true gain of 4 points is entirely consistent with it. Treating "not significant" as "no effect" is the mirror image of the mistake in the first slide, and it matters here because the decision on the table later in this topic is whether to remove a feature that costs real money. Removing something on the grounds that its benefit was not proven is weak when the experiment was too small to prove anything.

What converts a flat result into a decision is an equivalence bound: an interval whose upper end sits below the smallest gain that would justify the cost. The logic is that an interval names the range of true effects the data is compatible with, so if every value in that range is below the threshold, the decision comes out the same way whichever of them is true, and the size of the effect stops mattering. Suppose the added latency and tokens are worth paying only if memory buys at least 3 points. If the set is enlarged to 2,350 paired tasks and the observed difference comes out at +0.4 points with an interval of [-1.0, +1.8], the entire interval lies below the break-even threshold, and the conclusion is not "we failed to find an effect" but "whatever effect exists is smaller than the one we would need". That is a positive finding, obtained with the same machinery, and it requires deciding the break-even threshold before looking at the result rather than after. An interval that straddles the threshold, however narrow, still settles nothing.

The alternative that gets proposed instead is another quarter of tuning, on the grounds that the concept is sound and the implementation is young. That proposal is reasonable exactly once, and it needs a number attached to it: which threshold, ranker change or prompt revision is expected to move the result, by how much, and measured against which pre-registered interval, pre-registered meaning written down before the run rather than chosen once the result is visible. Without those, the quarter passes, the eval comes back inside the band again, and the same argument is available for the quarter after that, while the store keeps growing and the per-task cost keeps being paid. Framing the extension as an experiment with a stated expected effect and a stated stopping rule turns an indefinite deferral into a decision with a date on it.

Two habits undermine all of this quietly, and both are arithmetic rather than dishonesty. The first is re-running the eval until a run comes out favourable. With a 1.3-point run-to-run SD, a flattering number is one that lands in the upper part of a distribution that is more than a point wide, and if that is roughly one run in three, the expected wait for the first such run is about three attempts. Nobody has to intend anything for this to work; fix the number of runs in advance and report their mean.

The second is slicing after the fact until some subset reaches significance. A slice tested at the usual level clears by accident about one time in twenty. With a dozen slice labels, the chance that no slice clears by accident is one-in-twenty's complement multiplied by itself a dozen times, which comes to a little over a half — so a little under half of runs will produce at least one apparent finding with nothing behind it. Slices are worth analysing, and the earlier chapter set them up deliberately, but a slice-level result found by searching is a hypothesis for the next run rather than a finding from this one.

The noise band is now a number, the interval is a number, and both were computed from two cells of a table that the harness produces for free. What none of this arithmetic checks is whether the tasks measure memory at all. A confidence interval can be tight, the pairing correct and the power adequate, and the whole apparatus can still be reporting on an eval where the answer was sitting in the transcript the entire time — in which case the discordance rate collapses towards zero, every interval clamps around it, and the machinery reports with great precision that nothing is happening. That failure is common enough, and quiet enough, that it deserves its own chapter.