Public benchmarks

Read these first

• Why traditional testing fails

What you will know

• Run two named models on a benchmark’s released examples, inspect its scoring protocol, and record the capability and slices it actually tests.

  Aggregate scores can hide failures on long inputs, rare labels, adversarial cases, or the particular output format your system needs.

• Reproduce a leaderboard result and audit its dataset, scoring harness, and plausible contamination paths; record what evidence would invalidate the comparison.

  Training exposure and harness changes can raise scores without improving generalization, while small prompt or metric differences can make rankings incomparable.

• Use a benchmark to shortlist models and then stop using it

  After narrowing the candidates, continued optimization against a public score rewards leaderboard behavior rather than performance on your inputs.

• Compare a benchmark winner and a runner-up on a task-specific evaluation, using the same prompts, data, and scoring.

  A public ranking often reverses when correctness, abstention, structured-output validity, or domain-specific edge cases receive the weight they have in production.

• Build the private benchmark that actually decides

  It needs representative production cases, deliberately sampled hard failures, a stable rubric or oracle, and enough examples to distinguish candidates rather than amplify noise.

Chapters

1. 1

   Reading a benchmark score

   A leaderboard percentage is a property of three things at once: a capability someone chose to measure, a slice of items they chose to measure it on, and a protocol that turns a generated string into a verdict. Fetching the released examples and running two named models yourself costs an afternoon and converts a rank into a per-item record you can slice — which is how a three-point gap established on 400-token items turns out to say close to nothing about traffic running 8,000 to 12,000 tokens.

   Read chapter
2. 2

   The evaluation harness

   Everything between a benchmark item and a percentage — prompt assembly, few-shot count, reasoning elicitation, answer extraction, normalisation, aggregation — is the harness, and its knobs move a score further than swapping the model does. Run one model on one item set through four configurations and the totals span 33.5 points, which is why a reproduction landing at 61% against a published 74% is almost never the model, and why numbers lifted from different sources cannot be ranked against each other.

   Read chapter
3. 3

   Benchmark contamination

   A test set published two years ago has had two years to reach a training corpus, by scraping, reposts, forum solutions, translations, derivative instruction mixes and synthetic data — and each route defeats a different filter, so a decontamination claim is evidence of effort rather than evidence of absence. You can raise suspicion cheaply and never lower it to zero, which makes two afternoon probes, perturbation and completion, the practical instruments: option shuffling alone cost one model 16.5 points against another's 2.8.

   Read chapter
4. 4

   Screening candidate models

   A leaderboard cheaply narrows twenty models to three worth integrating — an hour of reading against fifty engineer-days of trying them all — and that is the job it does genuinely well. It ends when the shortlist does. Six weeks later, tracking the public score weekly while iterating on your own prompts tunes your system for somebody else's distribution: one variant gained 3.1 points on the benchmark and lost 12.4 on the product eval. Which decisions a six-point public gap can carry, and which it cannot.

   Read chapter
5. 5

   Rubric weights and abstention

   When a private eval reverses a public ranking, the explanation is almost always that your rubric scores something the benchmark does not score at all. Abstaining when the source material has no answer is the usual candidate: two models four points apart on a public board were 28.9% and 88.1% on correct abstention, which is 17.8 of a 9.2-point reversal. Writing the weights down turns a disagreement into a claim someone can argue with, and explains why four correlated public benchmarks are close to one measurement repeated.

   Read chapter
6. 6

   Auditing your own eval

   Running two models on the same prompt is an identical comparison, not a fair one: a prompt iterated for six weeks against model A encodes A's quirks as if they were task requirements. Three threats have to be cleared before a private-eval result decides anything — an unequal adaptation budget, a rubric that scores one model's formatting convention as correctness, and a point estimate with no spread. On one support eval they took a 9.2-point gap to 6.0, with a paired bootstrap interval of ±4.3.

   Read chapter
7. 7

   Building a private benchmark

   A selection benchmark is not a survey. Cases drawn uniformly from last month's traffic are dominated by routine work every candidate handles, so the aggregate compresses toward a tie while the rare cases carrying the risk appear once each. Covered: allocation by stratum rather than by frequency, an oracle whose annotator agreement you have measured, sizing worked backwards from the smallest gap you would act on, keeping the set from going stale, and scoring when a fabrication costs eight times what a miss costs.

   Read chapter