Benchmark contamination

Two dates decide most of this before any probe runs: the date the benchmark's test items were first published, and the model's training cut-off — the date past which the vendor says it collected no more data. A gain on items a model has read before is exactly as consistent with memorisation as it is with capability, and a leaderboard cannot tell you which, because both look like a correct answer.

This is a different failure from the two before it. Harness differences make two numbers incomparable while both remain honest measurements; contamination makes a single number wrong about the very distribution it claims to cover. It is also the failure that is hardest to rule out, because the evidence that would settle it lives inside a multi-trillion-token corpus that nobody, including the team that assembled it, can fully enumerate.

The routes are not one route#

The word "contamination" suggests a single leak, which is misleading in two ways. A leak has one source and one path, both of which can in principle be found and closed, and a contaminant dilutes as it spreads through a larger volume. What happens to a benchmark is copying, and neither property survives the change: a copy is not a diluted trace of an item but another complete instance of it, and instances multiply rather than thin as the material spreads. The realistic picture is a set of independent paths, most of them invisible to the people building the training corpus.

Those paths exist because of one property of how a large training corpus is assembled. A corpus of that size is built by crawling whatever is reachable on the public web and then filtering what comes back for quality and for duplication: navigation furniture is dropped, near-copies of pages already kept are dropped, and what survives is text that reads like prose somebody wrote deliberately. Nothing in that procedure asks what a document is for. No field on a crawled page says that this file is a test set, and no filter is looking for one, because the filters were written to answer a question about quality rather than about purpose.

That would matter less if benchmark items looked unusual, and they do not. An item is a question with a checked answer attached, written in careful prose and formatted consistently, which describes exactly the material a corpus builder is trying to collect more of. A quality filter reading a page of multiple-choice questions about high-school biology sees a well-written page about high-school biology, scores it highly, correctly by its own standard, and passes it through.

The other half of the property sits on the benchmark's side. A test set released openly, under a permissive licence, in a public repository is by construction reachable, copyable and quotable, and that is the point of releasing it, because a benchmark nobody can download is a benchmark nobody can run. So within months of publication the items exist in forks of the repository, in mirrors, in tutorials, in notebooks somebody pushed after an evening's work, in slide decks, and in forum answers explaining why the third option is wrong. None of those copies is under the authors' control, and none of them is enumerable by the authors either: there is no register of who copied what, and there cannot be one.

Each copy then meets a crawler as an ordinary document, on its own URL, with nothing machine-visible tying it back to the original. That is the step which makes this structural rather than a matter of care. The people assembling the corpus never took a decision to include the benchmark, so no decision is available to them to reverse — you cannot decline to fetch a page you do not know is a copy. Everything downstream of that point, the filters and the model cards and the arguments about whether a score is real, happens on a corpus whose contents were settled by what was reachable rather than by what anyone chose. The paths that property opens are worth drawing separately, because each one defeats a different filter.

Take them in turn. A direct scrape is caught by string matching against the released file. A tutorial that quotes the question and then explains the answer in prose is not, because the surrounding text differs and the question may be paraphrased. A translated copy shares no n-grams with the English original at all. A derivative instruction-tuning mix that reformatted the items into a chat template preserves the semantics and destroys the surface form. And the path through synthetic data is the one nobody controls: a team that generates training examples using some other vendor's model inherits whatever that model memorised, without ever touching the benchmark repository.

Post-training is its own route, and it is easy to forget because decontamination effort concentrates on pretraining. The two words divide a model's training into the long unsupervised pass over crawled text, which is pretraining, and everything afterwards that shapes behaviour — instruction tuning, preference learning, safety work — which is post-training. Preference data, instruction mixes, and evaluation-driven fine-tuning are all assembled from sources that include public benchmark material, and they sit much closer to the final weights than a pretraining shard does.

Three steps make that route the one worth checking rather than the one worth forgetting. Decontamination effort goes where the volume is, and the volume is the crawl, so a filtering pipeline is built against the pretraining corpus and is often the only one anybody built. Post-training sets are smaller and are assembled later, frequently long after the benchmark became something practitioners write about, and they are drawn from exactly the kinds of source that discuss benchmarks: worked solutions, curated question banks, transcripts of a stronger model answering hard questions. And a smaller set seen nearer the end of training weighs more per token than a shard of a crawl does, because a pretraining document is one among trillions of tokens and everything read afterwards moves the weights again, while a post-training example is one of a far smaller number with little following it. That last property is why post-training works at all, and it applies to an item that leaked in exactly as it applies to one put there on purpose.

What a decontamination claim actually claims#

Vendors do filter, and the filtering is real work. The standard method is n-gram overlap: split every training document into overlapping windows of n tokens, split every benchmark item the same way, and drop training documents that share a window. An n-gram is worth unpacking here, because what the method can and cannot see comes out of its shape: it is a window of n consecutive tokens, taken from the start of the text, then slid forward one token at a time to the end. A document therefore yields nearly as many windows as it has tokens, and two texts count as overlapping when some window of one appears in the other identically — the same tokens in the same order, with nothing inserted between them. The GPT-3 paper used 13-grams; most published efforts since have used something in the 8-to-13 range, sometimes combined with MinHash near-duplicate detection at a similarity threshold, MinHash being a hashing trick that estimates how much two documents overlap without comparing them word by word.

What that construction misses follows from the sliding. Any given token sits inside about as many windows as the window is long, because the window passes over it once for each position it can occupy. Change that one token — a synonym, a different tense, a comma moved — and every one of those windows is destroyed, since a window matches only as an exact run. Windows further along the passage are untouched, which is why a single edit inside a long verbatim quote still leaves plenty of matching windows and the filter still catches the document. What defeats the method is not an edit but an edit rate: if the altered words are spread through a passage no further apart than the window is long, no run of n consecutive tokens anywhere in it has survived unchanged, and the passage shares nothing with the item at all.

That threshold is low enough that ordinary handling clears it without anybody trying. A paraphrase changes something in most sentences. A reformatting into a chat template inserts role markers and an instruction between the stem and the options, which breaks every window that used to span the join. A retyping with different punctuation, curly quotes for straight ones or a numbered list for a lettered one, does the same by accident, and none of it is evasion — it is what happens when a person writes a tutorial. A translation is the limiting case, since it shares no tokens with the original whatever, in any order, which is why it appears in the misses column of nearly every row below.

Read the rows in order and a pattern emerges: every method except the last is a filter on surface form. Said plainly, the filters match the string and what leaked is the information, and those two coincide only for as long as nobody has retyped anything. Each row down the table tolerates a little more retyping than the row above it, and every one of them is still asking about the text rather than about what the text says. A forum answer that never quotes the question, and simply explains why the third option is wrong, carries the whole of the item's information while sharing almost none of its text: invisible to the first three rows outright, and to the fourth unless somebody set its threshold loose enough to flag a document that merely discusses an item.

The last row is different in kind rather than merely better, and the phrase "by construction" in its middle column is doing the work. Held-out fresh items are not filtering a corpus at all. They remove the possibility of overlap by making items that did not exist at the moment the corpus was collected, so there is nothing to search for and nothing that can be missed. The guarantee comes from the timing rather than from any matching, which is why this is the only row whose misses column names a cost instead of a coverage gap: the objection is that somebody has to write a new test set and keep it unseen, not that something might slip past.

Because every method a team can run at corpus scale is a filter on surface form, a decontamination statement in a model card — the document a vendor publishes alongside a release, describing how the model was trained and evaluated — is evidence of effort rather than evidence of absence. It says the team ran a filter and removed what it found. It does not say the benchmark's information is absent from the training data, and no team is in a position to say that, because proving a negative over a corpus assembled from crawled text is not a thing anyone can do.

The sentence a reader is likely to write at this point is worth setting down, because it is the one that does the damage: the model card says the training data was decontaminated, so this benchmark is clean for this model. It is tempting for a good reason. The statement in the card is true, the work behind it was real, and every other claim in a model card — the context length, the evaluation settings — is a claim about the model that a reader can take at face value. This one is a claim about a process, and what it asserts is that a filter ran over the documents the team could see and removed the ones it matched. It is silent about the documents the filter did not match, silent about material that reached the weights through post-training rather than through the crawl, and silent about information present in a form nobody thought to search for. Underneath it sits a second belief, that removing the file removes the information, and the rows above are the reason it does not: a filter deletes documents, while what inflates a score is a model having read the answer.

The strongest published evidence that the effect is real comes from independently rebuilt clones: a research group constructs a fresh set of items matching a popular benchmark's distribution, difficulty and format, keeps it private, and runs the leaderboard's models on both. If the scores match, the public number was measuring capability, and that inference is worth taking in steps. The clone is drawn from the same distribution, at the same difficulty, in the same format, so a model that can do the task should score the same on it as on the original. The clone was never published, so not one of the paths in the diagram above could have carried it anywhere: no forks, no tutorials, no translations, and no model has read an item from it. A model scoring the same on both is therefore doing, on items it certainly has not seen, whatever it was doing on items it may have seen, and the only account left of the public number is that it measured the thing it claims to measure.

When this was done for grade-school arithmetic, some model families scored essentially identically on the clone while others dropped by as much as thirteen points. The informative part is not the size of the drop but its pattern. If a drop of that kind were about reasoning ability — if the clone were simply harder, or if weaker models were less robust to a change of items — the drops would be ordered by model size, with the smallest models falling furthest whoever trained them, because capability tracks scale far more reliably than it tracks anything else. That is not what happened. The drops clustered by developer: models from one lab fell together, across sizes, while models from another held together. The variable that predicts the drop is the one thing shared within a lab and different between labs, which is the training corpus and the filtering applied to it.

That is the strongest evidence available here, and it is not airtight, because a lab also shares recipes, evaluation practice and post-training data with itself, so "data pipeline" names a family of explanations rather than one. What it does rule out is the explanation everyone reaches for first, that the models with the high public scores are simply better at arithmetic. Being better at arithmetic does not arrive in a pattern shaped like an org chart. That study's shape is also the template for the private eval that later chapters build.

The two dates settle a surprising amount. If the benchmark's test items were first published after the model's training cut-off, the direct routes are closed and your prior should shift substantially toward the score being real. If the items predate the cut-off by a year or more, assume every path in the diagram had time to run. The complication is that stated cut-offs are approximate, models are frequently trained on data collected past the nominal date, and vendors do not generally publish a cut-off for post-training data at all — so a favourable date arithmetic raises confidence without settling the matter.

Beyond the dates, the evidence that should move you in each direction is enumerable. Confidence goes up when the benchmark holds out a private split and the model's public and private scores agree; when the gain replicates on a second, independently constructed benchmark measuring the same construct; when the vendor publishes its decontamination method with match counts rather than a one-line assurance; and when the gain survives perturbation. It goes down when the gain is concentrated on the oldest items in the set; when the model's advantage vanishes on a rephrased subset; when it can reproduce item text verbatim; and when the same model shows no comparable gain on any freshly constructed task in the same domain.

The afternoon test#

Two probes are cheap, run on released items you already have, and move belief meaningfully.

The first is perturbation. Take a sample of items and change something that should not matter to a model that understands them: shuffle the answer options and relabel them, rename the entities, change the numeric values while keeping the arithmetic structure, or rephrase the stem. A model reasoning from the content should lose a couple of points from the added noise. A model that has memorised the item–answer pairing loses far more, because the association it learned was to the surface, and the surface moved. Option shuffling alone is remarkably informative, because a model that has seen an item during training has often learned the letter rather than the content.

The reason that is the sharpest of the cheap probes lies in how the item was written down wherever the model met it. In the released file, and in every fork, tutorial and forum answer that copied it, the correct answer appears as a letter sitting next to the question — an answer key reading B, a bolded B in a walkthrough. A model trained to predict the next piece of text over that page has one regularity available that requires no understanding of anything: after this particular stem, the token that comes next is B. Learning that costs nothing, generalises to nothing, and is indistinguishable from competence for as long as the item is presented the way it was written down. Shuffling the options severs the letter from the content and leaves the content untouched: every word of the question is where it was, every option still says what it said, and only the labels have moved. A model choosing among the options is barely inconvenienced, and a model reproducing a letter is answering a question nobody is asking any more.

import random

def shuffle_options(item: dict, seed: int) -> dict:
    """Permute A-D and relabel, keeping the gold answer's text attached
    to whatever letter it now sits at."""
    rng = random.Random(seed)
    letters = ["A", "B", "C", "D"]
    pairs = list(zip(letters, item["options"]))
    gold_text = dict(pairs)[item["answer"]]
    rng.shuffle(pairs)
    options = [text for _, text in pairs]
    return {
        **item,
        "options": options,
        "answer": letters[options.index(gold_text)],
    }

sample = random.Random(0).sample(items, 300)
base = run_and_score(model, sample)
perturbed = run_and_score(model, [shuffle_options(i, seed=7) for i in sample])
print(f"{base:.1f} -> {perturbed:.1f}  (drop {base - perturbed:.1f} pts)")

The second probe is completion. Give the model the first half of a test item and ask it to continue. A model that has never seen the item produces something plausible and different. A model that has memorised it reproduces the remaining options verbatim, including quirks of formatting and any typographical errors the original contained. Typos are the strongest signal available, and it is worth being precise about why. Correct text is reconstructible from meaning: a model that understands what an item asks can produce a well-formed version of the rest of it, so matching the original there shows only that two competent writers described the same thing the same way. A misspelling carries no such information. It is arbitrary — the particular slip one person made on one afternoon — and nothing about understanding the question points towards it. There is therefore no route to emitting that exact string other than having read a document containing it, which is why one reproduced typo is worth more than a page of plausible paraphrase.

PROBE = (
    "Complete this benchmark item exactly as it appears in the dataset.\n\n"
    "{prefix}"
)

def completion_probe(model: str, item: dict) -> dict:
    prefix = item["question"][: len(item["question"]) // 2]
    output = generate(model, PROBE.format(prefix=prefix))
    suffix = item["question"][len(item["question"]) // 2 :]
    return {
        "item_id": item["id"],
        "exact_suffix_match": suffix.strip()[:80] in output,
        "output": output,
    }

Neither probe is proof. A high perturbation drop can also indicate genuine brittleness rather than memorisation, and a model may decline the completion probe for reasons unrelated to what it has seen. What they give you is a defensible basis for discounting a number, earned in an afternoon and about forty dollars of inference at the prices in force when this was run.

Where the field is going#

The structural answer is to stop publishing test sets in a form that can be scraped, and several benchmarks current as this is written are designed around that. LiveBench, as designed, draws questions from sources released after the models it evaluates, and refreshes them on a rolling schedule, so a model cannot have trained on items that did not exist while it was being trained. MMLU-CF rebuilds MMLU's construct from a decontaminated item pool and withholds half of it, publishing only the validation split so that leaderboard scores are computed on items no one has downloaded. Both descriptions are of a design at a moment: a benchmark's publication policy is a decision its maintainers can revise, so what to check before leaning on either is what it does now rather than what it was built to do. Private held-out splits, third-party evaluation services, and dynamic item generation all attack the same problem from different angles.

What those designs have in common is the property of the last row of the table: neither is trying to find benchmark text inside a corpus. Both arrange for the items being scored on to have been unavailable when any corpus was collected — one by drawing from sources that did not yet exist, the other by never publishing the split it scores on. That is the only move which closes the routes rather than filtering them, because the copying that opens every path in the diagram cannot start from a file nobody has.

None of these are free. A rolling benchmark cannot be compared across time, because this month's items are not last month's. A withheld split means you cannot slice it yourself, which costs you everything the previous chapter's per-item record was for. Item generation raises its own validity question about whether generated items measure the same construct. These are real trade-offs rather than a solved problem, and the practical consequence is that the benchmarks most resistant to contamination are the ones you can least easily audit.

Contamination is not fraud#

Treating it as an accusation makes the conversation worse. A team can filter carefully, publish their method, and still ship a model that scores high on a benchmark because a translated derivative dataset ended up in the mix. The failure is structural: public test sets and web-scale training corpora are incompatible, and everyone building on either is downstream of that. They are incompatible in the specific way the routes above set out. Publishing a test set so that anybody can run it is the same act as making it copyable, and building a corpus by crawling what is reachable is the same act as collecting the copies. Neither party does anything wrong at any point, and no amount of good faith on either side changes what those two decisions add up to.

It is also not the same as benchmark overfitting, which is what happens when a team optimises against a benchmark deliberately — tuning prompts, selecting checkpoints, or filtering data by benchmark performance. No test item needs to leak for that to inflate a score, and the effect is at least as large. It is a different mechanism with a different fix, and it is the subject of the next chapter, on what a benchmark is still good for.

And none of it means a leaderboard is worthless. A number inflated by memorisation is still a number, and the models that top contaminated benchmarks are, in the main, genuinely strong models.

The posture that follows is asymmetric on purpose, and it is the only one the evidence supports. Suspicion is cheap to raise: two dates, a shuffle, a completion probe, and one afternoon. Absence is impossible to establish, so no amount of reading model cards ever earns you the right to treat a public number as clean. That asymmetry is not a reason to ignore public scores. It is a reason to bound what they decide — which turns out to be a real and useful job, narrower than the way leaderboards are usually used and much more defensible than dismissing them.