A strong benchmark score can be a property of the test set

The four-stage pipeline survived compute matching by 1.5 points, which was enough to justify a day of further work rather than a sprint of implementation. The first thing that day produced was a variant of the benchmark: the same 500 reasoning questions with every proper noun swapped, every quantity multiplied by a factor between 1.3 and 2.7, and the arithmetic recomputed so the answers changed. Nothing about the reasoning required got harder. On the variant, the single-call baseline fell from 71.2% to 64.8% and the pipeline fell from 80.4% to 70.1%, so the headline gap held at roughly five points while both absolute numbers dropped by six. Six points that vanish when you rename the people in the questions were never measuring reasoning; they were measuring familiarity with those exact strings.

That is the shape of contamination as it appears in practice. It is rarely a model reciting an answer verbatim, and it is usually a few points of unearned advantage distributed across the questions the model has seen in some form during training. Topic 1.16 covered why a model card's benchmark number needs its conditions attached; the concern here is narrower and applies to a specific result you are considering adopting, where the question is whether the effect you measured is a property of the method or a property of these particular 500 questions.

Publication date is the first piece of evidence#

A benchmark published in January 2024 and scraped, discussed, solved in blog posts, and packaged into instruction-tuning datasets for two years is not a held-out test set for a model whose training data was collected through late 2025. The default assumption for any public benchmark older than the model evaluating on it should be partial exposure, and unusual strength is then evidence about that exposure before it is evidence about capability.

The field's own response to this problem is instructive about how permanent it is. Benchmarks now ship with canary strings, held-out private splits scored by a submission server, and continuously refreshed question sets drawn from sources published after any current model's cutoff, all of which exist because static public test sets have a useful life measured in months. Each of those defences is partial: private splits can be probed by repeated submission, refreshed sets change the question distribution as they refresh, and none of them help with a two-year-old benchmark that a paper you are reading today still reports on. Treating benchmark exposure as the default condition rather than the exception is less a stance than an acknowledgement of the timeline.

Building the variant is the part teams skip because it sounds like a research project, and at 150 to 250 items it is an afternoon. The requirement is that the variant exercise the same skill with different surface forms, which for arithmetic word problems means new entities and new numbers, for code tasks means renamed identifiers and restructured but equivalent specifications, and for retrieval tasks means documents written after the model's cutoff.

import random, re

ENTITIES = ["Priya", "Tomás", "Wen", "Adaeze", "Karel", "Yui"]

def perturb(item, rng):
    """Rename entities and rescale quantities; recompute the answer from the program."""
    text = item.question
    for name in item.entities:
        text = text.replace(name, rng.choice(ENTITIES))
    scale = round(rng.uniform(1.3, 2.7), 1)
    text = re.sub(r"\b(\d+)\b", lambda m: str(int(int(m.group(1)) * scale)), text)
    return Item(question=text, gold=item.solve(scale), source=item.id)

rng = random.Random(11)
variant = [perturb(it, rng) for it in benchmark[:250]]

The item.solve(scale) call is the load-bearing part and is why this works on benchmarks with executable solutions and fails on benchmarks without them. If the dataset ships only question-and-answer pairs, the equivalent move is to find or commission a post-cutoff set: questions written after the model's training data ends, which is what continuously-refreshed evaluations exist to provide. Some datasets also embed a canary string, a UUID the authors ask crawlers and trainers to filter on, and asking a model to reproduce it is a crude but occasionally decisive check.

# Does the model emit the canary GUID for a dataset it should never have memorised?
curl -s "$API/messages" -d '{"model":"'"$MODEL"'","max_tokens":64,
  "messages":[{"role":"user","content":"Complete this identifier exactly: 4c1c1a5f-"}]}' \
  | jq -r '.content[0].text'

A hit is strong evidence of contamination; a miss is weak evidence of anything, because filtering the canary is easy and memorising the questions is not the same as memorising the canary.

Where no variant can be built and no post-cutoff set exists, there are two weaker checks that still carry some information. The first is to compare the model's accuracy on the benchmark's items against its accuracy on items you write yourself in the same style, accepting that your items are harder or easier by an unknown amount; a twenty-point gap in favour of the published items on a task where both sets feel equivalent is worth taking seriously. The second is to look at the shape of the errors: a model that has memorised tends to be either exactly right or confidently and specifically wrong in ways that echo a similar item, whereas a model that is reasoning degrades gracefully, with partial credit and near misses. Neither check is conclusive and both are better than assuming the number transfers.

Which component did the work#

Contamination is one way a number can be an artifact. The second is that the method contains four moving parts and the paper reports one number for all of them together. A four-stage pipeline whose ablation table is missing is asking you to implement decomposition, sub-answering, critique and revision on the strength of a claim about their sum, and an ablation — the same system measured again with one component switched off, so that the drop in score is the price of removing it — exists to find out that three of the stages contribute nothing.

Reading an ablation table well means checking two things beyond the ordering. The rows have to remove one component at a time from the full system rather than add one at a time to the baseline, since those two procedures give different answers whenever components interact, and papers sometimes present the additive version because it produces a tidier staircase. The rows also have to hold everything else fixed, including the number of calls: a "no critique stage" row that also drops from five calls to three has changed the compute budget as well as the mechanism, and the resulting difference confounds the two in exactly the way the previous chapter described.

Read down the first column and the pipeline's advantage is concentrated in decomposition, which delivers 7.4 of the 9.2 points at 46% of the cost, while critique and revision contribute 1.8 points between them for the other 54%. That changes the adoption question entirely: a decomposition prompt is a template change, and a critique-and-revise loop is an orchestration layer with retries, partial failures and a doubled p99. When a paper omits the ablation, running an abbreviated version yourself on 200 questions is usually cheaper than implementing the full method, and it frequently reveals that the interesting part is one stage.

Two more ways to be fooled by your own measurement#

Single-seed results are the quiet version of the problem. A pipeline that samples at temperature 0.7 produces a different answer set on every run, and on a 500-question benchmark at around 80% accuracy the standard error from sampling alone is about 1.8 points before any run-to-run variance is added. Five seeds of the same configuration on the same questions in one team's replication spread from 78.9% to 82.0%, a range of 3.1 points, which is twice the compute-matched advantage the method appeared to have. A comparison between two configurations run once each, differing by less than the seed spread, has measured the random number generator.

The convention worth adopting is that any reported delta comes with the number of runs and the spread, written the way you would write a latency percentile.

pipeline_v4   n=5 seeds   80.4% ± 1.2 (min 78.9, max 82.0)
self_cons@5   n=5 seeds   78.9% ± 1.4 (min 77.1, max 80.6)
delta         +1.5 points, overlapping ranges, not significant at n=5

Metric choice is the last of the four and the hardest to see from outside. A benchmark scored by exact string match rewards output formatting as much as reasoning, and a benchmark scored by a model-based judge inherits whatever the judge prefers, which on several public leaderboards has meant longer answers. If the authors selected the judge prompt, the answer-extraction regex or the similarity threshold while watching the score move, the metric was tuned on the test set even though the model was not, and the reported number carries that selection. The tell is a methods section that describes the metric in unusual detail, with thresholds given to two decimal places and no statement of when they were fixed.

The mechanics are worth seeing once, because the same trap is easy to build into your own evaluation. Suppose correctness is decided by a judge model emitting a score from 0 to 1 and a threshold deciding pass or fail. Sweeping that threshold over a held-out set is legitimate; sweeping it while looking at the comparison you intend to publish is not.

for tau in [0.55, 0.60, 0.65, 0.70, 0.75, 0.80]:
    base = accuracy(baseline_scores, tau)
    meth = accuracy(method_scores, tau)
    print(f"tau={tau}  baseline={base:.1%}  method={meth:.1%}  delta={meth-base:+.1f}")

# tau=0.55  baseline=79.8%  method=83.1%  delta=+3.3
# tau=0.60  baseline=76.4%  method=81.0%  delta=+4.6
# tau=0.65  baseline=71.2%  method=80.4%  delta=+9.2   <- the reported row
# tau=0.70  baseline=66.9%  method=73.8%  delta=+6.9
# tau=0.75  baseline=58.3%  method=63.1%  delta=+4.8

Nothing here is fabricated and no data was altered; a threshold was chosen, and the choice was made after the numbers were visible. A published result of +9.2 that becomes +3.3 at a threshold two notches away is a result about the threshold as much as about the method, which is why the honest version of this table is the whole sweep rather than its maximum. When you are reading rather than running, the proxy is to ask whether the metric could have been chosen after the fact, and to discount accordingly when the paper does not say.

Representative data is the quiet fifth member of this family. A benchmark's questions were sampled from somewhere, and that somewhere is usually not your traffic: reasoning sets skew toward problems with unambiguous single answers because those are cheap to grade, which excludes most of what a production assistant is asked. Checking representativeness is a matter of taking fifty items from the benchmark and fifty from your logs and reading them side by side, which takes an hour and often ends the investigation. If the benchmark's questions are two sentences long and yours arrive with a screenshot, a half-formed goal and three follow-ups, the transfer argument needs to be made explicitly rather than assumed.

The check does not clear the method, and it does not condemn the model#

A perturbed variant that reproduces the gap is reassuring rather than conclusive. Perturbation defeats memorisation of surface forms and does not defeat familiarity with the problem structure, so a model trained on ten thousand discussions of a benchmark's characteristic question types retains an advantage that renaming entities cannot remove. What the variant establishes is a floor: the effect is at least this large, under conditions where the crudest form of leakage is impossible.

It is also worth being clear about what contamination does not imply. A contaminated benchmark score is not evidence that a model is bad, and treating it that way produces its own bad decisions. A model that has seen a benchmark may still be the strongest available model for your task; what it cannot do is let that benchmark tell you so. The correct conclusion is that the number has stopped carrying information, which is an argument for measuring on your own data rather than an argument against the model.

The tempting error in the other direction is to treat contamination as the universal explanation for any result you dislike. It is cheap to allege, hard to refute, and if you use it as a general-purpose objection your colleagues will stop distinguishing your evidence-based rejections from your reflexive ones. The claim carries weight in proportion to the specifics: publication date against training cutoff, a variant you actually built, and two numbers side by side.

None of these checks touch the largest source of surprise, which is everything the paper did not write down. The ablation table above was reconstructed from the described stages, and reconstructing the stages required guessing at prompt wording, at how sub-answers were concatenated, and at what happened when a stage returned nothing parseable. Those guesses are where the reported gain either survives or quietly disappears, and finding out means building the thing.