Attributing a retrieval gain

Retrieval changes arrive bundled. In the fortnight it takes to swap an embedding model, an ingestion bug gets fixed, a golden set grows, a library is upgraded, and somebody publishes a documentation page that happens to contain nine of your evaluation queries. Each of those moves the headline metric, none of them leaves a mark that says by how much, and the request to ship arrives as one sentence with two numbers in it.

Attribution is the work of turning that sentence into a claim about one change, and it has three parts: making the two numbers commensurable, separating changes that landed together, and establishing that what remains is larger than the movement the system produces on its own. The third is the one most often skipped and the cheapest of the three.

The 2×2 that has to be run#

Start with the commensurability problem, because it invalidates most comparisons before any of the interesting work begins. A 0.61 computed over 80 queries and a 0.68 computed over 140 are different measurements of different populations, and the difference between them is not a delta in any sense. Nothing can be concluded until the archived baseline configuration from the previous chapter is replayed against the new query set and the current qrels — a forty-minute job when the ranked lists and the manifest were stored. Replayed that way, one old configuration scored 0.598 on the 140-query set, so the comparison the team thought they were making had a different starting point than the one they quoted.

With a common starting point, two changes means four configurations, and the only way to learn what either change did is to run all four against the same query set and the same judgments.

Read the rows and columns. Holding the corpus at its old state, the new model is worth +0.024. Holding the model old, the ingestion fix is worth +0.066. If the effects were additive, the bottom-right cell would read $0.598 + 0.024 + 0.066 = 0.688$; it reads 0.652, so there is an interaction of −0.036 between the two changes. On the corpus that will actually be in production, the new model is worth −0.012 relative to the old one.

The interaction can be computed a second way, and the two answers agreeing is what makes it a property of the pair rather than an artefact of the order they were subtracted in. Take the model's effect on the new corpus, which is 0.652 against 0.664, or −0.012. Take its effect on the old corpus, which is +0.024. The difference between those two effects is −0.036. Now do it the other way: the corpus fix is worth +0.066 under the old model and 0.652 against 0.622 — three points — under the new one, and the difference between those is again −0.036. One number, reached from two directions, and it means the same thing in both: switching the model is worth thirty-six thousandths less when the tables are present than when they are absent, and fixing the tables is worth thirty-six thousandths less under the new model than under the old. Those are the same sentence. An interaction is not a correction term bolted onto two main effects; it is the statement that neither change has an effect that can be quoted without naming the state of the other.

The mechanism is legible once you look at which queries moved. The new model's advantage on the old corpus was concentrated in 19 queries about limits, quotas and pricing tiers — exactly the content that lived in tables the HTML parser had been dropping. With the tables gone, answering those queries meant matching against the prose paragraph near where the table used to be, and a stronger embedding model is much better at that kind of loose semantic match than BM25 or the older model. Once the tables came back, the numbers, units and column headers were in the index as literal tokens, and a lexical match on rate limit 10000 requests beat any semantic approximation of it. The word concentrated there is doing quantitative work worth unpacking. The new model's advantage on the old corpus was 0.024 averaged over 140 queries, which is about three and a half points of nDCG spread across the whole set. Nineteen queries carrying all of it means each of those nineteen moved by roughly two tenths — an enormous per-query change, the difference between finding the answer near the top and not finding it. That is the signature of a change that fixes a specific failure rather than one that lifts everything slightly, and the two are worth telling apart, because a change of the first kind stops paying the moment its failure is fixed by other means. The new model's contribution was largely a workaround for a bug, and fixing the bug consumed the benefit.

This is why subtracting the corpus delta is not a repair. Subtraction is the arithmetic of additive effects, and the whole reason to run the 2×2 is that you do not know in advance whether the effects are additive. The number you would have produced by subtraction — crediting the model with $0.68 - 0.066 = 0.614$ worth of improvement over the 0.598 baseline — is off by more than the entire effect being measured, and it points the decision in the wrong direction.

Running four configurations instead of two costs something real. Re-embedding 61,200 chunks with the old model to reconstruct the old-model-new-corpus cell took 25 minutes and about $1.20 of API spend, and re-indexing the old corpus snapshot took eleven minutes. Under an hour of work bought the difference between shipping a change worth +0.024 and shipping one worth −0.012. The 61,200 chunks in that sentence are the corpus from the previous chapter at its September size, which is the check that the missing cells were built from the same snapshot as the ones already in the table — attributing a gain across four cells only works if all four were scored against one corpus, one query set and one qrels version, and the manifest is where that gets verified rather than assumed.

Two quieter rivals for the same seven points#

A confounded change is the explanation people reach for because it is visible in the git log. Two others produce movements of similar size and leave no trace anywhere.

Leakage is the first, and it has three routes into a retrieval evaluation. Golden queries can reach the corpus: on one project the docs team published a "questions people ask" page assembled from the search logs, ingestion picked it up, and 9 of the 140 golden queries appeared in it verbatim. BM25 ranked that page first for all nine, and because a labeller had graded it 2 as a genuinely useful landing page, nDCG@10 across the set rose by 0.021 with no retrieval change at all. Judgments can reach a tuning loop: fitting RRF fusion weights, or $k_1$ and $b$, or a reranker's score threshold, against the same 140 queries you then report on will produce a gain of two to four points that vanishes on any other query set. And golden documents can reach a prompt, when few-shot examples for a query-rewriting step are drawn from the same corpus region the golden queries target.

# Route 1: does a golden query appear verbatim in the indexed corpus?
$ python -m eval.leakcheck --queries golden-v5.jsonl --corpus corpus-2024-09-14 \
    --mode exact,trigram-jaccard --threshold 0.85
9/140 queries matched a corpus chunk above threshold
  q_0031 "how many webhooks per project"  -> faq/common-questions#q14 (exact)
  q_0087 "rate limit 429 retry after"     -> faq/common-questions#q31 (exact)
  ...
Recommendation: exclude faq/common-questions from the snapshot or re-judge.

The defence against the tuning route is the split every other part of machine learning takes for granted and retrieval teams routinely skip: 90 queries for tuning, 50 held out for reporting, with the held-out set never once used to choose a parameter. On a 140-query set that leaves 50 queries to report on, and the standard error on 50 queries is large enough that only substantial gains will clear it, which is an honest description of what a small golden set can support rather than a reason to skip the split.

The size of the tuning leak is measurable on any stack that has fusion weights. Sweeping the RRF constant and the lexical-to-dense weight over all 140 queries and reporting the best cell gave nDCG@10 of 0.692; fitting the same sweep on 90 and reporting the chosen configuration on the held-out 50 gave 0.661. Three points of the apparent gain were the sweep discovering the particular queries in the set rather than anything about retrieval, and with a grid of 24 cells that is exactly what a maximum over a small sample does. Why a maximum over a grid inflates is worth one sentence of mechanism, because it is the part that feels like cheating and is not. Each of those 24 cells is an estimate carrying its own sampling noise, and the cell you report is not a cell you chose in advance — it is the one whose noise happened to point upward furthest. The more cells you look at, the further up the luckiest one reaches, so the reported maximum is biased high by an amount that grows with the size of the grid rather than with the quality of the retrieval. Nothing about the sweep is dishonest; the bias is a property of taking a maximum over a sample, and it is there whether or not anybody intends it.

Cross-validation across five folds recovers most of the statistical power lost to the split, at the cost of five times the compute, and is worth doing when the golden set is small enough that a hold-out of 50 leaves nothing detectable. The trade is worth pricing before choosing. Splitting removes about three points of upward bias, and reporting on fifty queries instead of a hundred and forty widens the band on the reported figure by roughly the square root of that ratio — to something wider than the three points the split was removing. So a hold-out on a set this size buys an unbiased estimate and pays for it with an interval too wide to act on, which is the honest reason cross-validation is offered here rather than mentioned in passing.

Leakage detection belongs in the evaluation pipeline rather than in someone's memory, because the routes open up on their own. The FAQ page was published by a different team with good intentions, and nobody involved was thinking about the golden set at all. A check that runs on every evaluation and reports the count of queries whose text appears in the corpus, plus the fraction of each run's top-10 that is unjudged, costs seconds and catches both this and the pooling drift from the first chapter of this topic.

Nondeterminism is the second rival, and it is the reason to know your run-to-run spread before interpreting any delta. Approximate nearest-neighbour search, which finds vectors close to the query without comparing against every vector in the index, returns different neighbours depending on ef_search and on the order in which the HNSW graph — the layered proximity graph most vector indexes are built on, and the subject of topic 8.10 — was constructed, so re-indexing the identical corpus produces a slightly different index. BM25 scoring in a sharded cluster computes document frequencies per shard by default, so the same document can score differently depending on how documents were distributed across shards. Ties are broken by internal document id, which changes after a segment merge. Reranker and embedding calls over a hosted API are not bit-identical across batches. Each of these is small; together they are not.

Measuring the spread is one loop. Re-index and re-run the identical configuration ten times, changing nothing:

run  1  0.6612    run  6  0.6701
run  2  0.6668    run  7  0.6644
run  3  0.6571    run  8  0.6659
run  4  0.6683    run  9  0.6602
run  5  0.6640    run 10  0.6672
mean 0.66452   sd 0.00405   range 0.0130

A change that moves nDCG@10 by less than about 0.012 on this stack has not been shown to move it.

That threshold has two independent derivations from the same ten runs, which is why it can be quoted rather than argued about. Three standard deviations of 0.00405 is a little over 0.012. And the observed range — the best run minus the worst, with nothing changed between them — is 0.0130. One figure comes from the spread of the distribution and the other from its extremes, and they agree, so the number is not an artefact of assuming the runs are normally distributed. The cheaper of the two is the range, which needs no arithmetic at all: run the same configuration ten times, subtract the smallest from the largest, and refuse to interpret anything smaller. Ten runs is an afternoon of unattended compute, and it is the single cheapest thing in this chapter.

Testing the delta properly rather than eyeballing it#

Comparing two means against a run-to-run band is a blunt instrument, because it throws away the fact that both runs were scored on the same queries. Query difficulty is the dominant source of variance in any golden set, and it cancels exactly in a paired comparison, which is why a paired test detects differences several times smaller than a comparison of independent means would.

import numpy as np

def paired_bootstrap(a, b, n=10_000, seed=7):
    """a, b: per-query metric arrays for two runs over the SAME queries."""
    d = np.asarray(b) - np.asarray(a)
    rng = np.random.default_rng(seed)
    idx = rng.integers(0, len(d), size=(n, len(d)))
    means = d[idx].mean(axis=1)
    return d.mean(), np.percentile(means, [2.5, 97.5]), (means <= 0).mean()

delta, ci, p_le_zero = paired_bootstrap(old_per_query, new_per_query)
# delta = -0.0121   ci = [-0.0308, +0.0079]   p(delta <= 0) = 0.88

The interval crosses zero, so the data do not support a claim in either direction, and the per-query differences are worth reading directly as well.

A mean of −0.012 across the whole set is a redistribution, not a flat result: the new model is clearly better on entity-attribute and migration queries and clearly worse on navigational and error-string ones, which is the behaviour you would expect from a model that generalises well semantically and matches literal tokens less sharply. That table is a decision-support artefact in a way the scalar is not, because navigational queries are 22% of traffic and error strings 18%, and a change that damages 40% of the traffic to improve 21% of it is a bad trade regardless of what the headline says.

Those two percentages come from the traffic shares fixed when the golden set was sampled, and putting all six of them against the sign of each stratum's delta accounts for the whole of the traffic with nothing left over. Navigational at 22 and error strings at 18 are the two that got worse, which is the 40. Entity-attribute at 15 and migration at 6 are the two that got better, which is the 21. How-to questions at 31 and the tail at 8 are the two whose means are within a rounding of zero, which is the remaining 39. Four tenths damaged, two tenths improved, four tenths untouched — and that sentence is a decision, where "−0.012 overall" is a fact nobody can act on. The query counts close the same way: the six strata add to 140, and the improved column adds to 61, which is the figure quoted a paragraph below as the reason a wins-only report proves nothing.

The win-loss counts also warn against a common shortcut. Reporting only the queries that improved, which is what a screenshot of the biggest wins amounts to, will find a compelling story in any change whatsoever, because 61 of 140 queries improved even in a run whose overall effect was slightly negative.

Where isolation stops scaling#

None of this establishes that the new embedding model is bad. It establishes that the evidence offered does not distinguish it from the old one on the corpus that matters, which is a different and weaker claim than "do not ship it". There may be good reasons to ship it anyway — cost per million tokens, latency, vendor consolidation, better behaviour on a query type the golden set under-samples — and those are arguments the number neither supports nor refutes.

There is also a limit to how far isolation scales. Two changes require four runs; five changes require thirty-two, which nobody will do. The growth is a doubling per change, since each additional change adds a state that every existing configuration has to be run in both of, and doubling is what makes the practice fail at a size that arrives quickly. The practical discipline is to land changes one at a time when they are individually worth measuring, and when a bundle has to ship together, to label the result as a bundle and resist the urge to apportion credit inside it.

Attribution comes down to three questions asked in order, and the order matters more than any individual technique. Are the two numbers measurements of the same thing — same queries, same judgments, same cutoff — and if not, replay until they are. Did more than one thing change, and if so, what does the full grid say rather than the subtraction. And is what remains larger than the movement this stack produces when nothing changes at all, which is a question with a numeric answer that takes ten runs to obtain.

Every technique here depends on there being an archived before-number to replay. The situation where that does not exist is more common than any of this, and it has a cost that is usually described in the wrong terms.