Held-out sets and eval-set overfitting

Read these first

• Building a golden dataset

What you will know

• Keep a held-out set you do not look at while iterating

  Reserve access for scheduled checks or release decisions so prompt changes cannot silently fit its examples.

• Explain why iterating against a visible set is train/test leakage

  Repeatedly selecting changes on the same cases turns those cases into indirect training data, even when model weights never change.

• Notice when a gain on the dev set does not transfer to the held-out set

  A widening dev-to-holdout gap is a practical overfitting signal, especially when the examples are small or hand-curated.

• Refuse to ship a difference smaller than the run-to-run noise

  Sampling, model-service variation, and evaluator randomness can make an apparent win disappear on the next repeated run.

• Measure how quickly a visible eval set overfits by logging prompt variants and comparing dev-to-held-out transfer across iterations

  There is no universal iteration count; the burn rate depends on set size, prompt search breadth, and how aggressively changes target failures.

Chapters

1. 1

   Selection leakage in prompt tuning

   Keeping or discarding each prompt variant on the strength of the same labelled cases routes information from those cases into the prompt, one accept-or-reject decision at a time. No weights move and it is still leakage. This chapter shows the tuning loop as a training loop, counts the bits a score returns, simulates thirty identical variants producing four points of free gain, and separates this failure from the two kinds of contamination teams already guard against.

   Read chapter
2. 2

   Held-out sets and read budgets

   A held-out set is a second eval set that is independent only because of a rule about who reads it and when — nothing in the files makes it one. This chapter counts the information a single read returns, shows why an aggregate score leaks far more than it feels like it should, prices what a fortnight of daily plotting costs, and moves the rule out of people's heads and into a budgeted runner with an append-only log.

   Read chapter
3. 3

   Paired comparison and the winner's curse

   Two corrections sit between a two-point score difference and a real improvement: run-to-run noise from sampling and evaluator randomness, and the upward bias of taking a maximum over many noisy measurements. This chapter compares two prompts case by case rather than score against score, discounts a winner for the number of variants it beat, prices the cases a given difference would need, and separates all of that from the case where a flat number means the eval is not scoring the axis that changed.

   Read chapter
4. 4

   The dev-to-holdout gap

   When two sets start at the same score and one of them is consulted thirty times, the distance that opens between them is the fit component and nothing else. This chapter reads a six-week trajectory, rules out the four explanations that are not fitting, replaces a seventeen-point claim with a three-point estimate and its interval, shows why averaging the two numbers produces a quantity nothing corresponds to, and sets out which questions a burned set can still answer.

   Read chapter
5. 5

   Burn rate and transfer ratio

   There is no iteration count at which an eval set expires. How fast one burns is set by its size, the breadth of the search run against it, how failure-driven the edits are and how granular the scoring is — and the four factors span a factor of four across three real campaigns. This chapter shows what a point is worth on a small set, logs every variant tried, and turns the share of dev movement that reaches an independent set into an observed quantity rather than folklore.

   Read chapter
6. 6

   Governing and retiring a held-out set

   Governance exists here because the failure it prevents leaves no trace: no commit, no alert, no test that turns red at the moment a set stops being independent. This chapter reads a burn out of a read log, merges the retired cases into daily use, writes a manifest with size, provenance, owner, budget and retirement conditions, sizes a replacement against the smallest transfer it has to resolve rather than as a percentage of a scarce golden set, and sets out the three checks that say six months later whether any of it worked.

   Read chapter