Selecting forty times against one golden set makes it the objective

Two quarters of work sit behind the slide. Forty pipeline configurations have been compared against the same 300-query golden set, each round adopting whatever won, and the current champion is +9.0 nDCG@10 over where the team started in January. Then somebody labels 200 queries sampled from last month's production traffic — a genuinely new sample, drawn after the champion was chosen — and scores both the January configuration and the champion on it. The champion is +1.1.

Three terms in that paragraph carry the argument. A golden set is a fixed list of queries, each with the passages a human judged relevant, scored the same way every time so that two runs are comparable. nDCG@10 is the score, one number per query between zero and one, higher when a relevant passage sits nearer the top of the first ten results; a point is a hundredth of it, so +9.0 means the champion's average rose by nine hundredths on those queries.

Nine points of progress on one set, one point on another, from the same two configurations. Nothing was faked and the golden set was not corrupted in any obvious way. What happened is that the team ran a fitting procedure and did not notice, because each individual step looked like an experiment.

Forty rounds of keeping the winner is optimisation#

Chapter 4 covered what happens inside one sweep: the maximum of many noisy estimates is biased upward, and a search over 360 configurations on a 500-query set can manufacture 1.6 points from nothing. The failure here is the same mechanism applied across time, and it is worse in two ways.

The first is compounding. Each round starts from the previous round's champion, which was itself selected for scoring high on these 300 queries, so the optimism does not reset. Forty rounds of selection at a paired standard error of 0.0069 on a 300-query set accumulates a substantial share of the +9 even if every parameter change was genuinely inert. The second is that the search is not random. A person looked at which queries the champion was failing, formed a hypothesis about chunking or rewriting from those specific failures, and proposed the next configuration. That is a far more efficient optimiser against those 300 queries than any grid, and it is exactly what a good engineer does, which is why the failure recruits the team's competence rather than their carelessness.

"Substantial" is worth replacing with a size, because the two mechanisms are not the same size and the remedy differs. Chapter 4's argument bounds the first one: a running maximum over forty rounds inflates the retained score by a modest multiple of the standard error, and the standard error here is the 0.0069 above. A little over two of those is about a point and a half, which is substantial against a single round's resolution and is still the most that selection on noise alone can manufacture over forty rounds. The observed +9 is more than a dozen standard errors. Luck did not produce the bulk of it, and that is what makes the second mechanism the important one.

The second mechanism does not need luck. A hypothesis formed from the queries the champion is failing on is fitted to those queries, and the change it suggests can be perfectly real — a chunk size that genuinely does suit the documents those questions are answered from, a rewriting rule that genuinely does help questions phrased the way these are. The score moves for a reason, and it moves again next round for another reason, and nothing about the mechanism requires any of the changes to be spurious. What it does not do is generalise, because the property being fitted belongs to those 300 queries and their documents rather than to retrieval in general. The largest part of the +9 is therefore not noise at all. It is real improvement on a target that stopped standing in for anything else.

The end state is that the golden set stopped measuring quality and started defining it. Its 300 queries have particular characteristics — the average one is 14 words long, phrased as a complete question, in English, answerable from a single document, and written by four members of this team in the same fortnight — and two quarters of hill climbing have fitted the pipeline to that shape. The +9 is real in the strict sense that the pipeline is genuinely nine points better at those queries.

The stumble underneath all of this is a belief about what a fixed set of queries guarantees. Freezing the contents does hold one thing still: the same questions, the same labels, the same scoring code, so a difference between two runs is not a difference between two instruments. Everything about that is true, and it is why the belief survives forty rounds. What it does not hold still is the number of times the set has been used to choose something, and that count is the property that decides how much of a score is quality and how much is fit. The team watches a number that keeps rising and reads it as evidence; the quantity that would tell them what the number now means is not in it. A rising golden-set score is the golden set being compared with itself, and a check that compares a value with itself cannot fail.

The gap has four candidate explanations, and they leave different fingerprints#

Announcing "we overfitted the golden set" from the +9/+1 gap alone is a guess. There are at least four ways to produce that gap, they have different remedies, and each leaves evidence you can go and collect.

The first two are separated by one four-cell table, and it is the highest-value measurement in the whole investigation:

The absolute levels are close — 0.512 against 0.498 for the January configuration — so the production queries are not markedly harder in any way this metric can see. What collapsed is the improvement, and an improvement that exists on one set of queries and not on an equally difficult set of queries is fit to the first set. Had the January configuration scored 0.512 on golden and 0.31 on production, the story would be difficulty rather than fit, and the response would be about coverage instead of about methodology.

That table closes on itself, which is what lets it carry the diagnosis. Both stated deltas are the difference of the two cells above them, so the bottom row adds no information and cannot disagree with the body; the argument rests entirely on four measured numbers and one subtraction each. What it licenses is narrow and strong. Because the January configuration scores at the same level on both sets, the two sets are of comparable difficulty for a fixed pipeline, so difficulty is eliminated as an explanation of the collapse and fit is what remains standing.

The table cannot do more than that, and the boundary matters. Selection fit and the fourth row — shallow labelling — predict the same four cells: in both worlds the January configuration scores alike on both sets while the champion's advantage evaporates on the new one. The difference is what happens to the champion's newly retrieved passages, which are judged on the golden set and unjudged on the production sample, and no arrangement of these four numbers can see that. Pooling the top-10 of both configurations and labelling the union is not an optional extra test; it is the only one of the four that separates the two survivors.

The archived manifests from chapter 3 give a second, independent read. Because every one of the forty rounds recorded a full run, the team can rescore a handful of the historical champions on the new production sample and plot the two trajectories against each other:

A golden-set curve rising 9 points across those rounds while the production curve wanders inside 0.6 points from the second recorded round onward is the signature of fitting, and it is far more persuasive to a sceptical room than a single endpoint comparison, because it shows the divergence opening progressively as selection pressure accumulated rather than appearing in one suspicious jump. It also dates the divergence: rounds 1 through 9 moved both curves, and everything after round 18 moved only one, which suggests the early work was real and the later work was not.

Two arithmetic checks on that table are worth making, since the argument rests on it. The production column's first and last entries are the same two numbers as the production column of the four-cell table, so the trajectory and the endpoint comparison are one measurement seen twice rather than two that happen to agree — the production curve's total movement across all five rounds is the champion's +1.1. That is also why the 0.6 has to be quoted from the second recorded round onward: taken across all five rounds the production column moves about twice that, and the whole of the extra movement is the step from the first recorded round to the second — which is exactly the stretch the paragraph above dates as real work. The narrow reading of the 0.6 and the dating claim stand or fall together.

Taking the representativeness objection seriously#

A teammate looks at the production sample and objects that it is not a fair comparison: it contains follow-up questions that depend on conversational context, it contains out-of-scope questions the system is not meant to answer, and roughly a fifth of it is neither well-formed nor answerable. That objection is often correct. Production traffic really does contain "and what about the second one" and "is anyone else seeing this", and a retrieval metric computed over those is measuring something nobody designed for.

The way to handle it is to convert it into a filter specified in advance and then re-run the comparison, rather than to argue about it. Stratify the 200 queries by type — sorting them into groups defined in advance and scoring each group on its own, rather than reporting one average over all of them — before looking at any per-stratum score: single-turn in-scope, follow-up requiring conversational context, out-of-scope, malformed. Then compute the paired delta on the in-scope single-turn stratum alone, which is the stratum the golden set was built to represent.

strata = classify(production_200)   # by an annotator, blind to which config wins
for name, qs in strata.items():
    r = paired_bootstrap(champion[qs], january[qs])
    print(name, len(qs), round(r["delta"] * 100, 2), r["ci95"])

# in_scope_single_turn 118  +1.4  (-0.4, +3.2)
# follow_up             41  +0.6  (-2.9, +4.1)
# out_of_scope          27   0.0  ( 0.0,  0.0)
# malformed             14  -0.3  (-4.8, +4.2)

Three properties of that output are worth reading before its headline. The four stratum sizes add up to the 200 queries in the sample exactly, so the partition is complete and there is no unclassified fifth bucket in which an inconvenient result could be hiding. The two strata the objection is about — out-of-scope and malformed — together come to the "roughly a fifth" the teammate quoted, so the stratification reproduces the objection's own figure before answering it. And the out-of-scope row reports an interval of exactly zero width, which is only possible when every per-query difference in that stratum is exactly zero: with no relevant passage to retrieve, both configurations score the same, and resampling values that are all zero can only ever return zero.

That last property is also why stratifying is not cosmetic. A pooled delta over all 200 queries is the size-weighted average of the four stratum deltas, so 27 queries that cannot move contribute nothing to the numerator while still counting in the denominator. Including them dilutes the measured effect by the ratio of the two counts, and the same is true of any stratum whose delta is structurally near zero. The pooled number is smaller than the in-scope number for an arithmetic reason rather than a substantive one.

That weighted average also reconciles with the four-cell table only approximately, and the residual should be stated rather than left for a reader to find. Weighting the four stratum deltas by their sizes reproduces most of the +1.1 but lands a little under it, by something under a fifth of a point. That is more than the two-decimal rounding in the printed output can account for on its own, so the two figures were probably not produced by the same scoring pass. It changes no conclusion here — every reading of these numbers puts the champion's production gain at about a point — but it is the kind of gap that should be closed before the table goes into a report.

On the 118 queries that are directly comparable to what the golden set contains, the champion is +1.4 rather than +9.0. The objection was factually right about the sample's composition and it does not rescue the result, which is the outcome to expect and the reason to run the test rather than debate it. Two other things follow from that output. The interval on the in-scope stratum spans zero, so 118 queries cannot resolve a gain this size — chapter 4's power table says as much — and the follow-up stratum, at 41 queries and 20% of this sample, has never been evaluated by this team at all, which is a finding in its own right and probably a larger opportunity than any chunking parameter.

That share belongs to the sample, and carrying it across to production traffic as a whole is a step the sample supports only loosely. Forty-one queries out of 200 estimate how common follow-ups are in real traffic, and an estimate built on that many observations carries an interval several points wide in either direction. The finding does not depend on the exact share — an unevaluated stratum is worth attention at any plausible size — but the sentence to write down is that this sample puts follow-ups at around a fifth of traffic, and the sentence to avoid is one that treats the sample's count as the traffic's count.

Demoting the set, and the rotation that replaces it#

The golden set is not worthless and it is not fixable by addition. Its value as a selection instrument is spent: forty rounds of selection pressure means any future comparison on it is measuring fit as much as quality, and that damage does not wash out. Its value as a regression suite is intact, because catching a change that drops the score by 15 points does not require an unbiased estimator. So it gets demoted rather than deleted — run it on every pipeline change, alert on large drops, never use it to choose between two candidates that are within a few points of each other.

That split in what the set is still good for is worth deriving rather than memorising, and it follows from the size of the bias relative to the size of the signal. Selection inflates a score by something on the scale of a few standard errors, which here is a point or two; a regression alert fires on a drop of 15 points. A bias roughly an order of magnitude smaller than the threshold cannot flip that decision, so a contaminated instrument is still a perfectly good alarm. The same bias is decisive when the question is which of two candidates a few points apart is better, because there it is the same size as the thing being measured.

Merging the 200 production queries into it is the move to refuse, and it is the one most teams reach for, because it appears to fix the composition problem the teammate raised. The production sample's entire value came from never having been selected against. Fold it in, run the next 40 comparisons against the merged 500, and within a quarter the merged set gives an inflated number again — with the difference that there is now nothing left in the building that was not part of the search. The clean measurement is a consumable, and merging spends it to improve the composition of a set whose composition was not the binding problem.

There is also an asymmetry worth naming in how filters get proposed. When an unwelcome number arrives and someone proposes a filter that would remove it, the filter has to be justified by a property of the queries rather than by its effect on the result. "Out-of-scope questions should not count towards retrieval quality" is a rule you would have written down in advance. "These 30 queries are unusual" discovered after seeing which 30 the champion loses on is not a rule, it is the fitting procedure again, one level up.

What replaces it is a rotation with an explicit budget. Label a fresh slice of production traffic each quarter, seal it, and use it to score at most the top three candidates once, at the end of a selection process that ran entirely on the working set. The manifest from chapter 3 gains one more field, because the number of times a split has been selected against is a property of the split that has to travel with it:

{
  "split_id": "golden-v1-300@sha256:e5b2…",
  "role": "regression_guard",
  "selections_against": 40,
  "sealed": false,
  "retired_for_selection_at": "2026-07-02"
}
{
  "split_id": "prod-2026Q2-200@sha256:9c44…",
  "role": "confirmation",
  "selections_against": 0,
  "sealed": true,
  "unseal_policy": "one scoring run, top-3 candidates, results published whatever they say"
}

Writing selections_against into the artifact makes the resource visible, which is the practical difference between a team that knows its confirmation set has been used twice and a team that discovers after the fact that it has been used eleven times.

The field is doing something more specific than bookkeeping, and it is the whole argument of this chapter in one line of JSON. The dangerous property of a split is not any measurement taken on it; it is how many times it has been used to choose. That count leaves no trace in the score, cannot be recovered from the data later, and only ever increases. Recording it beside the split is the only place it can live, and once it is recorded the demotion rule writes itself — a split past its budget is a regression guard, whatever its contents.

This is not an argument that golden sets are a bad idea, and topic 3.5 already established the general principle that a set you select against stops estimating generalisation. What is specific here is the size of the leak in a RAG pipeline — retrieval-augmented generation, where the answer is written from passages fetched out of a document collection rather than from what the model remembers — and how fast it opens: two quarters and forty comparisons, which is an ordinary amount of work for one team, was enough to move eight of nine points into the set. That figure is the one to carry: the champion's honest gain is the point that survived on data nobody had selected against, and the other eight measure how hard the team tuned rather than how much better the system got.

The obvious defence is the one every engineer proposes at this point: stop using the whole set for selection, split it, tune on one part and confirm on the other. That instinct is correct and the standard implementation of it does not work for retrieval, because splitting the queries leaves the documents on both sides of the line.