The evaluation harness

Between a benchmark item and a reported percentage there are six or seven decisions, each defensible on its own, each worth points. Collectively they are the evaluation harness: the prompt template and how many solved examples it carries, the token limits and reasoning configuration on the generation call, the rule that decides which span of the output is "the answer", the normalisation applied before comparison, the comparison itself, and the way per-item verdicts are averaged into a total. None of these is the model, and all of them are inside the number.

The reason to care is practical rather than pedantic. Two teams can run the same model on the same released items, both believe they are doing exact match, and land more than ten points apart; a leaderboard and a vendor's release post can quote the same benchmark for the same model and be measuring different things. Until the harness is written down, a benchmark comparison is a comparison between two unnamed procedures, and the ranking it produces is not transferable to anyone who did not run it.

Where a score can leak#

Two of the labels in the diagram below are worth spelling out first. A few-shot prompt places some number of solved examples in front of the real question, so that five-shot means five of them and zero-shot means none, and chain-of-thought prompting, abbreviated CoT, asks the model to write out its intermediate reasoning before committing to a final answer. The items themselves usually arrive as JSONL — JSON Lines, the one-object-per-line format most benchmarks ship in.

Why any of those boxes can outweigh the model is worth being precise about, because the claim sounds like rhetoric and is not. A model's contribution to a score is the difference between how well it answers the items and how well some other model answers them, and on two comparable frontier models that difference is a few points over a few thousand questions: the two agree on most of the set and part company on a thin band of hard items. A harness knob does not change how well anything was answered. The generations are fixed strings by the time the knob is turned, and what the knob changes is how many of the correct ones are counted as correct.

Those two quantities are therefore not on the same scale, and one of them has no ceiling imposed by capability. Suppose a scoring rule throws away one correct answer in five because the model wrote it in a shape the rule does not accept: it subtracts a fifth of the score outright, and a fifth of a score in the sixties or seventies is a double-digit number of points. Nothing in that subtraction is a fact about what the model knows, and nothing about it has any reason to be small — it is a fact about how often the model's phrasing and the rule's expectations disagree, which is a quantity nobody measured and nobody set out to control. Stated plainly: the model decides what is in the response, the harness decides what is read out of it, and reading can lose more than answering ever gains.

Extraction is where most of a large discrepancy usually hides. A model asked a four-way multiple-choice question rarely replies with the single character C. It replies with "The correct answer is C, because the ligature described in the stem inserts on the greater tubercle." A harness that demands exact string equality scores that wrong. A harness that searches for the gold token anywhere in the response scores it right. Both rules are implementable in three lines and they disagree on a large fraction of items — on the zero-shot run below, on 9.4% of them.

That extraction rather than, say, normalisation is where a large gap lives follows from what each of the two rules is asking. Normalisation adjudicates near misses, deciding whether c counts as C, and in every case the string it is looking at is already the answer in some form. Exact-match extraction asks something else: whether the model emitted the bare answer and nothing else. That is a question about formatting rather than about knowledge, and a model trained to be helpful answers a question in a sentence, so on a set where it explains itself the rule is largely measuring how often it happened to stay quiet. Normalisation can only move the items sitting on a boundary; extraction can move every item on which the model wrote a sentence, and that is most of them.

Few-shot count is the next largest source after extraction. Providing five worked examples before the question does two things: it teaches the task, and it teaches the output format, and the second effect is often bigger than the first. Many published MMLU numbers are five-shot, and the benchmark's own paper reports few-shot results as the headline. A zero-shot run of the same model on the same items scores lower because it answers in prose the extractor cannot parse. The model knows exactly as much either way.

The last sentence is the one to earn, because "it teaches the format" sounds like a small thing. Solved examples show two things at once: what the task is, and what a correct response looks like on the page — a bare letter, or a letter followed by a line of justification, in whatever shape the template happens to use. A model that already knows the subject gains nothing from the first of those, since it could answer the question before the examples arrived. It gains the second regardless, because a prompt ending in several answers laid out the same way makes the next answer overwhelmingly likely to arrive laid out that way too. The extractor's job then gets easier — the span it is hunting for is where it expects the span to be — so the total rises while the model's grasp of the subject stays exactly where it was.

Reasoning elicitation is the third. A prompt that says "think step by step before answering" — or a model configured to reason before responding by default — produces longer output, better arithmetic, and an answer buried further from the start of the response. This interacts with extraction: chain-of-thought raises accuracy and simultaneously breaks a first-token extractor, so the two changes together can move a score in either direction depending on which dominates.

Normalisation is smaller but not negligible, and it is the knob most likely to differ between two teams who both believe they are doing exact match. Does c match C? Does C. match C? For a short-answer benchmark, does "the Treaty of Versailles" match "Treaty of Versailles", and does "1919" match "1919 AD"? Every open-source harness ships a normalisation function — lowercase, strip punctuation, drop leading articles, collapse whitespace — and no two of them are identical. On free-text short-answer sets, swapping one project's normaliser for another's typically moves a score by two to four points, and it does so silently, because both runs report themselves as exact match.

Then there are the quieter ones. A max_tokens set too low truncates the response before the answer appears, which an extractor reads as a wrong answer rather than as a truncation. A stop sequence chosen for tidiness can cut the answer off mid-token. Aggregation choices matter on unbalanced sets: MMLU has 57 subjects of very different sizes, so a flat mean over items and a macro-average over subjects — which averages the 57 per-subject percentages, letting a subject with 100 items count for as much as one with 1,500 — give different numbers for the same run.

The two ways of averaging agree more often than the fuss around them suggests, and knowing when they part is what makes the choice between them a real one. A flat mean weights each subject by the number of items it contributes; a macro-average gives every subject the same weight whatever its size. If the model scored the same everywhere, re-weighting would change nothing and the two totals would land in the same place. They separate exactly when subject size and subject difficulty are correlated: when the large subjects are the easy ones the flat mean comes out higher, and when the large subjects are the hard ones it comes out lower. So letting a hundred-item subject count for as much as a fifteen-hundred-item one is not a technicality about averaging — it is a decision about what the score is a score of, breadth across fields or performance on the item pool as it happens to be stocked.

Code benchmarks add a dimension the multiple-choice ones do not have: how many attempts count. HumanEval's headline metric is pass@1, the probability that a single sample passes the unit tests, but pass@10 — the probability that at least one of ten samples passes — is reported nearly as often and runs fifteen to thirty points higher on the same model. Both are honest metrics answering different product questions: pass@1 tells you about a system that ships the first generation, pass@10 tells you about one that generates candidates and filters them against tests. A ranking that mixes a pass@10 number for one model with a pass@1 number for another is not a ranking, and the metric name is often omitted from the summary table that gets shared.

The gap between the two metrics is large for a reason that has nothing to do with the model being cleverer on its tenth try. pass@10 asks whether any of ten samples passes, so the only way for a problem to be scored wrong is for all ten to fail, and independent chances of failing multiply. If a model fails a given problem about half the time, failing it twice running is about half of a half, failing it three times running is half of that again, and by the tenth sample the chance of having failed every one has been halved ten times over. Sampling is not perfectly independent in practice — the same problem tends to defeat the same model in the same way, so the real any-sample rate sits below what the multiplication promises — but the direction holds, and it is why a middling per-sample rate turns into a high any-sample one. Which of the two belongs in a comparison is then settled by the two product questions above rather than by the size of the number.

Rescoring the same outputs four ways#

Running stored outputs through different scoring rules costs nothing, because the generation already happened. Changing prompt assembly or generation settings requires re-running, but there are only a handful of configurations worth trying.

import re

GOLD_LETTERS = {"A", "B", "C", "D"}

def score_exact(output: str, gold: str) -> bool:
    return output.strip() == gold

def score_contains(output: str, gold: str) -> bool:
    return gold in output

def score_first_letter(output: str, gold: str) -> bool:
    """Common harness rule: the first standalone A-D token in the response."""
    match = re.search(r"\b([A-D])\b", output)
    return match is not None and match.group(1) == gold

def score_last_letter(output: str, gold: str) -> bool:
    """Better for chain-of-thought: the answer usually comes last."""
    matches = re.findall(r"\b([A-D])\b", output)
    return bool(matches) and matches[-1] == gold

SCORERS = {
    "exact": score_exact,
    "contains": score_contains,
    "first": score_first_letter,
    "last": score_last_letter,
}

def rescore(records: list[dict]) -> dict[str, float]:
    return {
        name: 100 * sum(fn(r["raw_output"], r["gold"]) for r in records) / len(records)
        for name, fn in SCORERS.items()
    }

Applied to one model's stored outputs from a zero-shot run over one benchmark's released items, and then to a re-run with five in-context examples, the whole range becomes visible at once.

Read it one axis at a time before reading it whole. Along a row, the model, the prompt and the generations are all fixed, and the only thing moving is which rule was applied to strings already sitting on disk, so every difference along a row was produced without asking the model anything. Down a column the rule is fixed and the prompt is what moved, so those differences did require new inference. The first row makes the point on its own: 61.2 under exact match and 70.6 under containment differ by 9.4, which is the same 9.4% quoted above, because containment accepts everything exact match accepts and then accepts the items where the answer arrived inside a sentence. Each item the two rules disagree about is worth one item's share of the total, so the disagreement rate and the point gap are the same number seen twice.

A published 74 is reproducible in three different ways there, which is the point. Five-shot with a lenient extractor gets to it; zero-shot with reasoning and a last-token extractor gets to it; five-shot with reasoning and a first-token extractor lands far below it while being, by any reasonable account, the strongest configuration in the table. An earlier draft of this chapter put that last distance at seventeen points, which the table does not support: the published 74 against that row's 47.1 is nearer twenty-seven, and the arithmetic is the thing to trust because it is the part a reader can check. The spread across this one model and one item set is 33.5 points, against the 3.3-point headline gap that a model-selection exercise is typically trying to adjudicate.

That 33.5 is a distance between two cells rather than a property of the run, and it is worth putting a finger on which two. The largest number in the table is the five-shot reasoning row read by the containment rule, at 78.2. The smallest is the zero-shot reasoning row read by the first-letter rule, at 44.7. Take the second away from the first and 33.5 is what is left, so the whole span is the distance between the most permissive reader on the strongest prompt and the least permissive reader on a prompt with no examples in it. What separates the two ends is how many examples preceded the question and which span of the response was read out of it, and neither of those is a statement about how well the model answered: it answered as well as it answers everywhere else in the table. The 3.3 points that model selection was meant to settle fits inside that span ten times over.

Note the diagonal. Reasoning elicitation helps under contains and last, and destroys the score under first, because a reasoning model's first standalone capital letter is usually part of the working rather than the conclusion. A harness change and a prompt change interacting is the normal case, not the exotic one, which is why you cannot audit these knobs one at a time and add up the deltas.

The mechanism under that diagonal is one property of reasoning output meeting one property of the rule reading it. A model asked to work through the question first writes prose about the options: it names A in order to rule it out, weighs B against C, and commits only at the end. Those letters are standalone capitals in exactly the sense \b([A-D])\b means, and they sit in the working, ahead of the conclusion. So the first-letter rule reads out whichever option the model considered first, which is right only when the model happened to consider its own answer before the others. The last-letter rule reads the final one, and on a response that ends with its conclusion the final one is the answer — same generations, same model, opposite verdicts on a large share of the set, which is the collapse from the seventies into the forties in the two reasoning rows.

Reproducing a published figure is the check most teams reach for first, and it is much weaker than it looks. The belief underneath it — that landing on the number validates the setup — is tempting for a respectable reason: matching the published figure is what reproduction means everywhere else in engineering, and a run that lands there has, in the ordinary sense of the word, reproduced it. The table is the counterexample. Three configurations reach the published 74, and they are not variants of one setup: one is a five-shot prompt, a strong configuration by any reading, and another has no examples in front of the question at all and arrives at the same total only because the extractor was reading the end of a chain of reasoning. A match therefore tells you that some combination of prompt, elicitation and extraction summed to that total, and the table shows several combinations that do.

What it does not tell you is which of the six or seven decisions you got right, because the total is the only thing you compared and every one of those decisions is inside it. Two runs are the same measurement when the settings agree — the template actually sent, the number of examples in it, whether reasoning was elicited or on by default, which span of the response was read, what was normalised before the comparison, and how the per-item verdicts were combined — and agreement between the totals is then a consequence rather than the evidence. That is the difference between reproducibility as a property of a procedure and a coincidence between two percentages, and it is the whole justification for the provenance block below. It also makes a mismatch the more informative outcome, since a run that lands short has one thing to go and find and a run that lands on the number has six or seven things it has not checked and no reason to look.

Writing the generation settings down#

The generation call itself is part of the harness, and the fields that matter are not always the ones people record. Model string and prompt get logged; token limits, stop sequences, and reasoning configuration usually do not.

from anthropic import Anthropic

client = Anthropic()

def generate(model: str, prompt: str) -> str:
    response = client.messages.create(
        model=model,                       # e.g. "claude-opus-5"
        max_tokens=2048,                   # low values truncate CoT into a wrong answer
        system="Answer the multiple-choice question.",
        messages=[{"role": "user", "content": prompt}],
        thinking={"type": "adaptive"},     # reasoning is a harness knob, not a detail
        output_config={"effort": "medium"},
    )
    return "".join(b.text for b in response.content if b.type == "text")

Two things about that call are worth pausing on. Reasoning is configuration, not a property of the model, so "which model scores higher" is underspecified until you say whether reasoning was on and at what depth — and on several current model families, at the time of writing, it is on by default unless the request switches it off, which means a harness written against an older model can silently change behaviour when the model string is updated. And the sampling controls that older harnesses relied on for determinism, temperature and top_p, are on several current models not accepted at all rather than quietly ignored; sending them returns a 400. The traditional advice to "set temperature to zero for reproducible evals" is not a lever you can always pull, which makes run-to-run variance a thing you measure rather than a thing you suppress. That measurement problem gets its own treatment in chapter 6, on auditing your own eval.

Both of those are claims about a moving target, and an unconditional sentence about provider behaviour is the likeliest thing in a harness write-up to have gone stale by the time somebody reads it. Which parameters a request may carry, and whether a model reasons unless told not to, are properties of one model string on one provider on one date — not of models in general, and not even of the family a string belongs to, since a family gains members that behave unlike the ones already in it. So the step to make somebody demonstrate is not an argument about which defaults are usual. It is the provider's current reference for the exact model string the run used, read on the day the comparison is being made, set beside the request body as it was actually sent. A harness that logs the request body settles the question in seconds; a harness that logs only the model name cannot settle it at all.

What an audit records#

The output of this exercise is not a number. It is a short block of provenance attached to every score you intend to cite, listing: benchmark name and release version, item count actually run, prompt template and few-shot count, generation settings including reasoning configuration and token limits, extraction and normalisation rules, aggregation method, and the date. Every field in that list is one that has been observed to move a published score by more than a model swap does, which is the whole justification for recording it.

Alongside the provenance, write the falsifier — the sentence stating what evidence would make the comparison invalid. This is the discipline that separates an audit from a note-to-self, because a claim with no stated falsifier cannot be checked by the person who inherits it. For a same-harness comparison of two models the falsifier is usually "if the two models were run with different few-shot counts, different reasoning configuration, or different extraction rules, this ranking is void." For a number lifted from a vendor's post it is "if their harness used a lenient extractor and ours used exact match, the two numbers are not on the same scale." Writing it down has a useful side effect: about a third of the time, drafting the falsifier is the moment you notice that the condition you are describing is in fact true of the comparison you just made.

Legibility, not trust#

Matching the harness does not make the leaderboard trustworthy; it makes it legible. The published 74 and a reproduced 74 now mean the same thing, and what they mean is narrower than it looked. That is progress, and it is all the progress this chapter offers.

Nor does an official harness solve the comparability problem across sources. Two leaderboards can both use the benchmark's official evaluation code and still be incomparable, because the official code takes parameters and each site picks its own. A number scraped from a vendor's release post, a number from an academic paper, and a number from a community leaderboard are three measurements under three protocols; ranking models by picking whichever number is highest for each is not a comparison at all. The only defensible cross-model claim is one where you controlled the harness yourself, which is the honest limitation of everything here: it tells you how to make a comparison valid, and the price is that you can no longer borrow other people's.

What makes the borrowing feel safe is that a benchmark name reads like a unit. Units are comparable across laboratories because the definition fixes the whole measurement, and a benchmark name fixes only the questions and the answer key. Everything between a question and a percentage — the six or seven decisions above — is left to whoever ran it, and none of it is recoverable from the name afterwards. A benchmark name is closer to a syllabus than to a unit: it says what was examined and not how the scripts were marked. That comparison stops in one place, and it is the place that matters here — a lenient examiner leaves evidence on the script, and a lenient extractor leaves none in the percentage, which is why the settings have to be recorded at the time rather than inferred from the result later.

Finally, this is not the contamination question. Everything above assumes the model is genuinely solving the items and the only issue is how the solving got scored.

The harness is therefore the first thing to establish and the last thing to assume. A score with a provenance block attached is a measurement someone else can repeat and disagree with; a score without one is a percentage of unknown origin, and the fact that it is published on a well-maintained leaderboard does not supply the missing information. Two numbers are comparable when they came out of the same procedure, and nothing about sharing a benchmark name guarantees that.

Suppose you now do all of it correctly — same harness, same prompts, same extraction, same aggregation, both models run by you on the same afternoon — and one model comes out eight points ahead. There is still a reason that gap might have nothing to do with capability, and it has to do with where the test items were before they were a test.