An identical rerun moves the metric, and the best of 360 moves further

Run one pipeline twice with every setting held identical and the two evaluation scores are not the same number. Take the manifest from the last chapter — the artifact that records everything a run depended on, from corpus snapshot to decoding parameters — and run it five times, changing nothing. Same corpus snapshot, same pinned embedding version, same index parameters, same 300 queries, same judge prompt. The five nDCG@10 values come back as 0.5941, 0.5903, 0.5972, 0.5918 and 0.5960 — a standard deviation of 0.0028 and a spread of 0.7 points between the best and worst run of a configuration that is identical to itself.

Two conventions and one term carry the rest of the chapter. nDCG@10 is a per-query number between zero and one that rewards a relevant passage more the nearer the top of the first ten results it appears, averaged across the query set to give the score a run reports. A point is a hundredth of that score. A sweep is one pass over a grid of configurations, each scored end to end and compared against the others. So the gap between 0.5903 and 0.5972 is the 0.7 points quoted, and the standard deviation of 0.0028 is a shade under a third of a point.

That number is the scale on which every result from the sweep has to be read. A configuration that beats another by 0.4 points has produced a difference smaller than the difference a configuration produces against itself, and no amount of care in the experimental design changes that. Most teams discover the noise floor by accident, usually when a rerun of a shipped configuration comes back worse and someone spends two days looking for the regression. Measuring it deliberately, once, before the sweep, costs five runs and is the cheapest thing in this topic.

Where the 0.0028 comes from#

Nothing in the pipeline is exactly deterministic, and the contributions are unequal enough to be worth separating. Running the same five-fold repetition while pinning one source at a time gives a decomposition:

That column has to be read cumulatively — each row is what is left once that source and every source above it is held fixed — and the reason is arithmetic rather than convention. Independent perturbations combine through their variances, which are the squares of the standard deviations, because the cross terms in the square of a sum have expectation zero when the two sources know nothing about each other. Squares add; standard deviations do not. Read the rows as four separate one-at-a-time pinnings and the squares of the implied contributions come to more than three times the square of the live figure, which is impossible for parts of a whole. Read them cumulatively and each contribution is the difference between consecutive squares, those differences and the square of the last row add back to the square of the first, and the generator comes out largest — which is what the paragraph below the table says.

The same reading explains why the last two rows are printed identically. Pinning the index build fixes the graph, and tie-breaking by internal id only has something to vary when a near-tied pair could have been ordered either way, so once the graph is frozen the tie-break has nothing left to move. That row is not reporting a source of its own size. It is reporting that pinning it changes nothing further, which makes the figure beside it a residual rather than a contribution — and a residual roughly a third of the live figure in standard-deviation terms. A fifth source of run-to-run variation is therefore present, unnamed, and not much smaller than the index build. The four listed mechanisms do not account for the noise floor; they account for most of it.

Setting the generator to temperature 0 removes the largest single term and is usually the right call for an evaluation harness even when production runs warmer, because the question being asked is about retrieval rather than about generation diversity. The judge is more stubborn: temperature 0 on a hosted model is a request, not a guarantee, since batching and mixture routing make the arithmetic non-reproducible at the token level. Taking three judge samples and using the majority verdict brings its contribution down to about 0.0006 at three times the judging cost, which for a 360-configuration sweep is exactly the line item chapter 2 identified as dominant — so this is a real trade rather than an obvious improvement.

Index-build variation is the one people find surprising, and it is also the one that matters least here, because a sweep comparing chunk sizes rebuilds the index anyway and a sweep comparing k values does not rebuild it at all.

One comparison the table cannot support is worth naming before somebody makes it. Every figure in it is a spread estimated from five runs, and an estimate of a spread from five observations is itself loose to the tune of tens of percent. The two smallest rows are printed identically and the residual sits at the same magnitude, so nothing here licenses a ranking among the bottom three. The top of the table is a different matter: the generator term is several times the smallest ones in variance, and a gap that large survives the estimation error.

Subtracting before averaging, and where the interval comes from#

The instinct after seeing 0.0028 is to run each configuration five times, take means, and compare with error bars. That works and it wastes most of the information available, because the comparison you care about is paired: both configurations answered the same 300 queries, and a query that is hard for one is usually hard for the other. Differencing per query cancels the query-difficulty variance that dominates the absolute score.

That cancellation is worth writing out, because everything downstream rests on it. Model one query's score under a configuration as three terms added together: a difficulty term belonging to the query, an effect term belonging to the configuration, and a noise term. The difficulty term is the same number for both configurations, since it is a property of the question and its gold passage. Subtract the two per-query scores and it is removed exactly, leaving the configuration effect plus the difference of two noise terms. Nothing was approximated there — the same quantity appears on both sides of the subtraction with the same sign, which is the only condition a cancellation needs.

The size of that effect is the difference between an inconclusive study and a decisive one. Per-query nDCG@10 on this golden set has a standard deviation of about 0.30, since a query either retrieves its gold passage near the top or it does not, so the standard error of an absolute mean over 500 queries — the standard error being the spread of the estimate itself, as opposed to the spread of the per-query scores it averages — is 0.30/√500 = 0.0134 and a 95% interval on any single configuration's score is roughly ±2.6 points. Two such intervals will overlap for almost any pair of configurations you compare. The per-query difference between two configurations, by contrast, has a standard deviation of about 0.12, because on most queries both configurations do the same thing and the difference is exactly zero. Its standard error over 500 queries is 0.0054, giving a 95% interval of about ±1.05 points on the delta — two and a half times tighter, from the same data, by subtracting before averaging rather than after. The interval on that delta comes from a bootstrap, which draws a fresh set of 500 per-query differences by sampling the real ones with replacement, takes its mean, repeats that ten thousand times, and reads the interval off the middle 95% of the resulting means, so that no assumption about the shape of the distribution is required.

Both spreads in that paragraph are per-query quantities, which is what lets them be divided by the square root of any set size you like. The 0.30 is a property of the metric and this corpus; the 0.12 is a property of the pair of configurations being compared. Neither is a property of how many queries were labelled, and that is why the noise-floor runs above can use a 300-query set while the figures here are quoted for the 500-query evaluation set the sweep proposal named.

The two spreads are also not independent of each other, and the relation between them is checkable. The variance of a difference of two correlated quantities is the sum of their variances minus twice their covariance, which for two arms with the same per-query spread reduces to twice that spread's square times one minus the correlation. Put the figures in: 0.12 divided by 0.30 is 0.4, squaring gives about a sixth, halving gives about a twelfth, so the per-query correlation between the two configurations is around nine-tenths. That is the arithmetic behind "on most queries both configurations do the same thing", and it is computable from your own logs before you build a harness around it.

One further identity settles a question people otherwise try to answer by labelling more data. The ratio of the two intervals is the absolute spread over the square root of the set size, times the multiplier, divided by the difference spread over the square root of the same set size, times the same multiplier. Multiplier and square root both cancel, leaving 0.30/0.12 = 2.5. The two-and-a-half-fold tightening is exactly the ratio of the per-query spreads and does not depend on the set size at all, so pairing buys a fixed factor that no amount of extra labelling substitutes for.

import numpy as np

def paired_bootstrap(a: np.ndarray, b: np.ndarray, resamples: int = 10_000, seed: int = 7):
    """a, b: per-query metric values for two configs, same queries, same order."""
    assert a.shape == b.shape
    rng = np.random.default_rng(seed)
    n = a.shape[0]
    d = a - b
    idx = rng.integers(0, n, size=(resamples, n))
    boot = d[idx].mean(axis=1)
    return {
        "delta": float(d.mean()),
        "ci95": (float(np.percentile(boot, 2.5)), float(np.percentile(boot, 97.5))),
        "queries_better": int((d > 0).sum()),
        "queries_worse": int((d < 0).sum()),
        "queries_tied": int((d == 0).sum()),
    }

# winner vs current champion, 500 queries
# {'delta': 0.0151, 'ci95': (0.0043, 0.0259),
#  'queries_better': 96, 'queries_worse': 71, 'queries_tied': 333}

The resampling is over queries, not over runs, because the population being generalised to is the population of questions users ask. The last three fields are worth reporting alongside the interval: a delta of +1.5 points built from 96 improved queries and 71 degraded ones is a different result from the same delta built from 30 improved and 2 degraded, and the mean hides the distinction completely. Two-thirds of the query set being untouched also tells you that the evaluation is spending most of its statistical power on queries that cannot discriminate.

What gets resampled decides what the interval means. Sampling queries with replacement treats the labelled set as a sample from the questions users ask, so the interval answers whether this difference would survive on other questions of the same kind. Had the golden set been the entire population of interest — every query the system will ever be asked, which no golden set is — there would be nothing to generalise to and the interval would be zero wide. Reading the width as an error bar on the labelled set itself gets the direction of the inference backwards.

The stumble here costs teams a great deal of time. The two absolute intervals of ±2.6 points overlap heavily while the paired interval of ±1.05 points on the same comparison can sit entirely above zero, and it is tempting to read the overlap as proof that the study found nothing. Two intervals drawn separately discard the covariance between the two arms, and it was exactly that covariance — the nine-tenths correlation above — that the subtraction removed. Overlapping intervals on two individual means are compatible with a decisive difference between them, and the paired interval — the one built on the standard error of the difference — is the one that answers the question that was asked.

That standard error is also a power calculation, which is the question almost nobody asks before running a sweep. To detect a true difference of size δ with 80% power — power being the probability that a difference that genuinely exists comes back as statistically significant instead of being missed — at the conventional 5% level you need roughly δ ≥ 2.8 × SE, so at SE = 0.0054 the smallest difference this 500-query set can reliably resolve is 1.5 points. Resolving 1 point needs 500 × (1.5/1.0)² ≈ 1,125 queries; resolving half a point needs about 4,500.

The 2.8 is not a convention to be memorised; it is two distances added together. The first is how far the observed difference must sit from zero before the test declares anything at all, which at the conventional two-sided 5% level is a shade under two standard errors. The second is the further distance the true difference must sit above that threshold so that four-fifths of its sampling distribution lands beyond it rather than under it, which is a bit over four-fifths of a standard error. Change the power you are willing to accept and only the second distance moves.

Rearranging that inequality produces the query counts. Substituting the standard error gives δ ≥ 2.8 × s/√n, where s is the per-query difference spread, and solving for n gives n ≥ (2.8 × s/δ)². The set size scales with the inverse square of the difference you want to see, so halving the difference multiplies the labelling bill by four — which is where the jump from 500 to 1,125 and then to 4,500 comes from.

That table is not five measurements. It is one relation evaluated at five sizes: the middle column is the per-query difference spread over the square root of the left column, and the right column is the middle column times the multiplier just derived, converted into points. Every cell reproduces to the printed precision, which licenses the strong version of the claim — a reader can price any set size at all, including one not in the table, without running a single query.

A sweep whose plausible gains are around a point, run on 300 queries, is not underpowered by a little. It cannot see the effect it is looking for, and every configuration it ranks is being ranked mostly on noise.

The maximum of 360 noisy estimates#

Everything so far concerns a single comparison. A sweep makes 360 of them and then reports the best, and that final step introduces a bias that no amount of careful interval construction removes.

The reason is visible before any simulation is run. Write each configuration's measured score as its true quality plus an error term with mean zero, which is what the noise floor established. Suppose every configuration has the same true quality, so the measured scores differ only in their error terms. The reported winner is then that common quality plus the largest error term, and the largest of several draws from a zero-mean distribution is not itself zero-mean: it is at least as big as their average, which is near zero, and strictly bigger whenever any two draws differ — which for continuous noise is always. The winning score therefore exceeds the truth in expectation for any number of candidates above one, whatever the size of the noise, and it does so upward only, because nobody reports the minimum.

That is why the operation is a maximum over a noisy sample rather than a measurement of the best configuration. The same numbers do two jobs: they choose which configuration to call the winner, and they report how good it is. Whatever error term happened to be largest is exactly the one that gets kept and quoted, so the quoted figure is a draw conditioned on having come out on top rather than a fresh look at the thing that won.

Simulate the extreme case in which the parameters do nothing at all, so all 360 configurations have identical true quality, and the only thing separating them is measurement noise at the SE computed above:

import numpy as np
rng = np.random.default_rng(0)

TRUE = 0.560          # every config identical
SE   = 0.0054         # paired SE from a 500-query set
trials = rng.normal(TRUE, SE, size=(20_000, 360))
winners = trials.max(axis=1)

print(round(float(winners.mean() - TRUE) * 100, 2))          # 1.57 points
print(round(float(np.percentile(winners - TRUE, 5)) * 100, 2))  # 1.16 points
print(round(float(np.percentile(winners - TRUE, 95)) * 100, 2)) # 2.06 points

A sweep over 360 configurations that are all exactly as good as each other reports its winner at +1.6 points on average, and reports at least +1.2 points 95% of the time. The expected maximum of n draws from a normal distribution grows like σ√(2 ln n), which is 2.9σ at n = 360, and the growth is slow enough that trimming the grid barely helps: even 30 configurations still buys you +2.2σ of free improvement.

Those three printed lines are three different quantities and are easy to read as one. The first is the mean excess over many simulated sweeps, and it is the one the σ√(2 ln n) expression is about. The second is the fifth percentile of that same distribution, the value exceeded in ninety-five sweeps out of a hundred — which is why it, and not the ninety-fifth percentile on the third line, supports "at least +1.2 points 95% of the time". A reader who takes the average as a floor will understate how often a sweep does worse than it.

The closed form deserves one caveat and one consequence. It is a leading-order asymptote, exact only as the number of candidates grows without limit, and at a few hundred candidates it overshoots the true expected maximum by an appreciable margin — evaluate it at 360 and it lands well above the 2.9σ the simulation returns. The simulated value is the one to trust here, and the expression is useful for how the quantity scales rather than for what it equals. Its consequence is the important part: the number of candidates enters through a logarithm, and then under a square root. Raising the grid to a power multiplies the inflation by the square root of that power, so cutting the inflation in half means taking the fourth root of the number of configurations. Trimming a grid is not a defence against this.

Two conditions keep that argument from being over-read. The simulation assumes every configuration is equally good, which is the worst case rather than the usual one; where real differences exist, the argmax usually does land on a genuinely good configuration, and what is corrupted is the number attached to it rather than the choice. And the inflation is proportional to the noise scale, so halving the noise floor halves the phantom gain — the second reason the five runs at the start of this chapter pay for themselves.

This is why the winner's measured score is not a usable estimate of what it will do next. The selection step chose, among other things, whichever configuration got lucky, and its score contains that luck. The same argument was made in topic 3.5 about an evaluation set that has been selected against over months; the sweep compresses six months of that into a week, and 360 comparisons made in one afternoon carry the identical optimism as 360 made one at a time over two quarters. The mechanism does not care how long it took.

The correction has one form: re-measure the winner on data that took no part in the selection. Hold out a slice before the sweep starts, never look at it during, and score exactly one configuration — or at most the top three — on it at the end. The reason the number of configurations you re-score matters is the same logarithm as before: a maximum over three candidates carries a small fraction of the inflation a maximum over 360 does, so a top-three re-measurement is close enough to clean, while re-scoring the whole grid on the held-out slice would simply move the selection onto it.

Differences the study cannot resolve#

Measuring the noise floor and bootstrapping the delta does not make a small effect real. If the true difference between two configurations is 0.3 points, no interval-construction technique will let a 500-query evaluation resolve it, and the honest report says the study could not distinguish them. That is a different statement from saying they are equivalent, and the difference matters when a stakeholder wants to know whether to spend on the more expensive one.

The bootstrap also does nothing about selection bias, which is the part that surprises people who have done the statistical work carefully. Resampling gives a correct interval for the comparison you specified in advance; the winner of a 360-way search was not specified in advance, and its interval is conditional on having been selected for being high. There are corrections for this — Bonferroni on 360 comparisons is brutally conservative, and the more useful move is simply the held-out re-measurement above, which sidesteps the correction entirely by generating a fresh estimate.

The belief worth naming here is that a correctly computed interval is a defence against selection, and it is tempting for a good reason: the interval genuinely is correct, computed from real per-query differences with no approximation anywhere. It is also identical in two worlds that are not remotely alike. Run one pre-registered comparison and the bootstrap over the 500 differences returns some width; run 360 comparisons, keep the best, and the bootstrap over that winner's 500 differences returns essentially the same width, because it is the same arithmetic over the same kind of data. The quantity being watched holds still while the property that matters — how much of the observed delta is the maximum of a search — has moved from none of it to most of it. A check that cannot move when the danger moves is not a check.

The null arm, and a noise floor that drifts#

There is a cheap habit that catches most of what goes wrong here, borrowed from online experimentation: run the null comparison. Put the current champion into the sweep twice under two different run ids and let the harness treat them as separate configurations. The paired delta between those two arms should come out near zero with an interval that comfortably contains zero, and if it does not — if the same configuration beats itself by 0.9 points — the harness has a bug, usually a cached index shared across arms or a query order that leaks into sampling. That check costs one extra arm out of 360 and it validates the measurement apparatus with the same machinery that measures the result.

What makes that arm worth its cost is that its expected answer is known in advance. Every other arm produces a number nobody can independently predict, so a broken harness shows up there as a result rather than as an error. The null arm has one correct answer, and any apparatus fault large enough to matter pushes it off zero in a way no interpretation can absorb.

Everything in this chapter is also the reason a sentence like "we improved quality by 6%" cannot be acted on. Six percent of what metric, against which baseline configuration, with what interval, measured on which split — those four questions are not pedantry, they are the difference between a 6% that is four times the noise floor and a 6% that is the expected maximum of a large search over a flat surface. Chapter 7 sets out what a sweep report has to publish; the point here is that the numbers it publishes are the ones this chapter tells you how to compute, and a report cannot contain an interval that nobody produced.

The noise floor is also specific to a pipeline and a metric, and it drifts. A sweep that switches the generator model or the judge prompt has a new noise floor, and reusing last quarter's 0.0028 as the yardstick is the same category of error as reusing last quarter's parameter values. Re-measuring it costs five runs.

Three numbers, then, decide whether a sweep result means anything, and each is cheap next to the sweep itself. The noise floor says how large a difference has to be before it is a difference at all, and five identical runs produce it. The paired standard error says which differences the query set can resolve, and it comes from per-query differences you already have. The gap between the selection score and a held-out score says how much of the winner's advantage was the act of choosing it, and it costs one sealed slice and one extra scoring run.

None of this touches the deeper version of the problem, which is what happens when a team does the held-out re-measurement correctly, ships the winner, and then repeats the whole exercise next quarter against the same golden set. Each individual sweep is clean. The sequence is not, and after a few rounds the set that was supposed to measure quality has become the thing the pipeline is fitted to.