Frozen context and ground truth

Everything under test in an eval run should be the thing you changed, and everything else should be a constant. A set that stores only the customer's message and the expected outcome fails that condition badly, because a generative assistant reaches out to a retrieval index, an order service, a shipping API and a policy store on its way to an answer, and every one of those is a moving part owned by somebody else. Re-running such a set measures the system and whatever those dependencies happened to be doing that afternoon, and there is no way to attribute a movement in the score to any one of them.

That impossibility is worth making concrete rather than leaving as a principle, because it is the reason behind every design decision in this chapter. Suppose a run comes back two points lower than last week's, and in the intervening week the prompt was edited and the search index was rebuilt. Those two points are one number produced by cases whose behaviour depends on both changes at once. The reading is consistent with a two-point prompt regression against a neutral index; with a five-point prompt regression partly rescued by a better index; with a prompt improvement swamped by a worse index. Nothing in the number separates the three, and no amount of staring at it will, because the information that would separate them was never collected. The only way to collect it is to hold one of the two still while the other moves, and holding something still is what freezing means.

This chapter defines what a row has to contain for that attribution to be possible, which is roughly what a recorded HTTP interaction contains plus a fuzzier assertion at the end. It covers which fields nobody logs by default and therefore have to be added to production tracing before any capture is possible, what stays frozen and what is deliberately recorded instead, and the residue of non-determinism that freezing does not remove. The second half is about the other half of the row. Ground truth can be a bit, a written ideal answer, or a list of criteria, and the three differ by a factor of three in minutes per case, in what they can tell you about a failure, and in how they are scored — which is why a set whose labelling was estimated before that choice was made overruns.

A row is a replayable execution, not a question#

The unit that solves the attribution problem is familiar to anyone who has recorded HTTP interactions for a test suite — a cassette in the sense the VCR library made standard, a golden file, a fixtures directory. The test does not call the payment gateway; it calls a recorder that returns the exact bytes the gateway returned the day the fixture was taken, so the test fails when your code changes and not when someone else's service does.

Concretely, a row carries the conversation as the customer sent it, every retrieved document with its id and the text as it stood, every tool call with its arguments and the response that came back, the identifiers of the prompt and model version that produced the original trace, the ground truth, and the labelling metadata. Written out for one multi-account case:

{
  "case_id": "ma-0007",
  "slice": "multi_account",
  "source_trace_id": "trc_01HQ8Z3M4K2",
  "captured_at": "2026-03-11T09:42:17Z",
  "messages": [
    {"role": "user", "content": "hi — did my delivery for the boots go out yet"},
    {"role": "assistant", "content": "Happy to check. Could you confirm the email on the account?"},
    {"role": "user", "content": "j.moreau@example.net"}
  ],
  "fixtures": {
    "retrieval": [
      {"doc_id": "acct_88421", "score": 0.81, "text": "Account: Julien Moreau, 14 Rue des Lilas, 69003 ..."},
      {"doc_id": "acct_88437", "score": 0.79, "text": "Account: Juliette Moreau, 14 Rue des Lilas, 69003 ..."},
      {"doc_id": "pol_returns_v7", "score": 0.44, "text": "Returns are accepted within 14 days for sale items ..."}
    ],
    "tools": [
      {"name": "get_orders", "args": {"account_id": "acct_88421"},
       "response": {"orders": [{"id": "ORD-77120", "item": "Chelsea boots", "status": "processing"}]}}
    ]
  },
  "environment": {"prompt_version": "sys-2026-03-02", "model": "claude-opus-5"},
  "ground_truth": {
    "kind": "rubric",
    "must": ["resolves to acct_88421, not acct_88437",
             "does not disclose any order belonging to acct_88437"],
    "should": ["states the current status of ORD-77120",
               "asks a disambiguating question if confidence in the match is low"]
  },
  "labelling": {
    "labelled_by": ["dmoreno", "ashaikh"],
    "minutes": 11,
    "ambiguity": "Both labellers initially split on whether naming the item without the status counts as an answer.",
    "rule_applied": "LG-14: a status question is answered only if a status word appears."
  }
}

The fixtures block is what removes a search-index rebuild from the measurement. At run time the harness does not hit the search cluster or the order service; it substitutes an adapter that serves the recorded results for that case and fails loudly on any call it has no recording for, because a silent empty response would let the system under test pass by taking a path that never existed.

You can only capture what production wrote down, which makes tracing a prerequisite rather than a detail. A service that logs the customer's message and the final answer cannot produce a row like the one above at any price, and retrofitting it means waiting weeks for enough new traffic to accumulate under the richer logging. The fields that turn out to be needed are the ones nobody logs by default: retrieval scores as well as document ids, the document text as retrieved rather than a pointer to a mutable store, tool arguments as well as tool names, and the prompt template version. Storing document text rather than a pointer is the one that gets debated on storage grounds, and the debate is usually settled by the arithmetic — 179 cases averaging 40 KB of fixtures is about 7 MB, which sits in the repository next to the code and diffs cleanly in review. Sets that outgrow that keep the fixtures in object storage with a content hash in the row, which preserves immutability and gives up the ability to read a case in a pull request.

The model version is deliberately not frozen, and the environment block records rather than pins it. Pinning would make the set a permanent record of one model's behaviour and defeat the main use of the gate, which is to find out what changes when you upgrade. Recording lets you attribute: when a run on the same set version scores differently and the only field that moved is the model, the attribution is done. Everything the case controls is frozen and everything under test is recorded, and getting a field on the wrong side of that line is the most common structural mistake in a case format.

Classifying a field nobody has thought about yet takes two questions in order. Can this change without anyone on your team deciding to change it? If it can, and it is not the quantity you are trying to measure, freeze it — otherwise it will move on its own and its movement will be attributed to your work. Is this the quantity you are trying to measure? If it is, record it, because pinning it removes the measurement along with the noise. Engineers are routinely surprised by which fields answer yes to both questions at once — a retrieval index you own and are also actively improving is the usual one — and for those the resolution is not a better answer but two runs, which is what the scenario below ends up building.

Three kinds of ground truth, three different budgets#

The ground_truth field in that record says rubric, and that was a decision with consequences reaching all the way to the labelling schedule. There are three shapes it could have taken, and teams routinely start labelling before choosing between them, which is how a two-day estimate turns into a two-week project.

Pass/fail is the cheapest and the coarsest. For 147 cases at five minutes it is roughly twelve hours of one person's attention, which fits inside two engineer-days with room for a second pass on part of the set. It is the right choice for anything with a bright line — did the assistant disclose another account's data, did it follow the authorisation rule, did it refuse when it should have refused — and a poor choice where quality is a matter of degree, because everything gets compressed into one bit and a case that got 80% of the way there is recorded identically to one that hallucinated.

Reference answers are the most expensive to produce and the most treacherous to score. Writing the ideal answer to a support conversation takes twelve to eighteen minutes once the labeller has read the trace, checked the policy and written prose — 147 cases is thirty-plus hours, more than triple the budget. Scoring then has to decide whether the produced answer means the same thing as the reference, and every mechanism for that has a known bias: exact match rejects valid paraphrases outright, embedding similarity rates a fluent wrong answer close to a correct one, and an entailment check — asking a second model whether the reference answer follows from the produced one — inherits that judge's own errors.

Those three failures have three different causes, and telling them apart matters because each one invites the belief that a better version of it would work. Exact match compares strings while the quantity you care about is a meaning, so its errors are about wording alone: a valid paraphrase is not approximately rejected, it is rejected outright. Embedding similarity compares directions in a vector space, and direction in a general-purpose embedding space is dominated by topic and register rather than by truth, so a fluent and confident wrong answer about a returns policy points almost exactly the way the correct one does. The entailment check asks the closest thing to the right question and has the most dangerous errors, because a judge model's mistakes are correlated with the system's: both are working from the same idea of what a plausible answer looks like. That is the shared-prior problem from the first chapter of this topic, arriving one layer further down, where it is harder to see because a second model really does look like a second opinion.

Reference answers earn their cost where the correct output is genuinely near-unique, which covers extraction into a schema, translation, SQL generation and classification.

Rubrics sit in between and are usually the right default for assistant conversations. The labeller reads the trace and records what must be true and what should be true rather than composing prose, which is faster than writing an answer and far more informative than a bit. The must-criteria give a hard gate; the should-criteria give a graded score that moves when quality moves. The setup cost is real — four to six hours to design a rubric that different people apply the same way — and it is amortised across every case and every refresh.

The labelling block is not bookkeeping either. Recording minutes per case is what converts the next refresh from a guess into an estimate: after 147 cases you know that multi-account cases take eleven minutes and order-status cases take four, so adding a forty-case slice for a new intent is a number you can put in a plan. One team's real average came out at 7.4 minutes against the 5 they had assumed, which is the difference between 12 hours and 18. Recording ambiguity and rule_applied matters for a different reason: cases whose verdict came down to a judgement call are the ones most likely to be labelled inconsistently by the same person on a different day, and writing down which rule decided a borderline case makes the inconsistency findable later — when the rule changes, a query over rule_applied returns exactly the cases that need re-labelling.

What makes that a query rather than a re-read is that the field turns the labelling guide into an index. Each row names the rule that decided it, so rules and cases point at each other, and editing a rule hands you the list of affected cases without anyone opening a trace. A set without the field holds the same information for exactly one day — it is in the labellers' heads while they work and nowhere afterwards — and recovering it later costs a full second pass over every case.

The signal that freezing removes#

Fixing the context fixes the input to generation; it does not make generation deterministic. The same case can produce different text on two runs and occasionally a different verdict on a borderline case, for reasons an earlier topic covered. The practical accommodation is to run each case a small number of times — three is the usual compromise between cost and stability — and to record both the majority verdict and the disagreement rate, because a case that produces 3/3 passes and one that produces 2/3 are different facts about the system, and collapsing them loses the more interesting one.

Three is odd on purpose, so that a majority always exists, and it is three rather than five because the multiplier lands on the whole set: five runs of 147 cases is five times the tokens and five times the wait, on every pull request, for a stability gain that shrinks with each extra run. What three runs do not buy is determinism. A case that flips half the time still flips; the majority verdict simply agrees with the more common outcome more often than a single run would. The disagreement rate is the part worth keeping, because it is the only field that separates a case the system handles from a case the system happened to handle.

Freezing also removes real signal along with the noise, and the trade has to be made explicitly rather than discovered. A replayed case can never tell you that retrieval got worse, because retrieval is a constant in it — the only thing under test is what the model does with a fixed context. That is precisely what you want when the question is whether a prompt change broke anything, and it is useless when the question is whether the search index is still good. Both questions need answering, which is why the arrangement is two runs over the same cases rather than one.

Comparing those two runs against each other, rather than each of them against a threshold, is what makes the live one informative. The replay run is a control: its inputs came off disk and cannot have moved since capture. So a difference between the live run and the replay run on the same cases has one candidate cause left, which is the dependencies the live run resolved and the replay run did not. That is the same hold-one-thing-still argument the fixtures were built for, applied a second time at a different level, and it is why a widening gap between the two arrives already attributed instead of arriving as a mystery in a gate.

The fixtures go stale in a way that is worth watching, because they encode last quarter's document corpus. When pol_returns_v7 is superseded by v8 with a different return window, every case carrying v7 is now testing behaviour against a policy the company no longer has, and the case will keep passing while production gets it wrong. Storing the document version in the fixture is what makes that findable: a weekly job listing every fixture whose source document has changed since capture turns an invisible decay into a work queue, and the last chapter of this topic turns that queue into a policy.

A row that is a replayable execution costs more to capture, more to store and more to keep current than a row that is a question, and what it buys is attribution. When the score moves, the only things that could have moved it are the model, the prompt and the scorer, because everything else in the run came off disk. The other half of the row is the outcome it is scored against, and the choice between a bit, a written answer and a list of criteria is a budget decision as much as a design one: pass/fail at five minutes a case, rubrics at eight to twelve after four to six hours of design, reference answers at twelve to eighteen and a scoring problem of their own.

Freezing the context has one consequence that the record above makes plain if you read it as a security reviewer would. Case ma-0007 contains a customer's name, their email address, their street address, their order id, and a second household member's account document, all captured verbatim and now sitting in a repository that runs in CI, gets cloned onto laptops, and is readable by everyone who touches the assistant. Production access to that data is behind an approval workflow; the eval set is behind a git clone. The obvious remedy is to strip the identifiers — to overwrite each one with a placeholder — and the obvious remedy destroys the case, because the whole reason ma-0007 is valuable is that "Julien Moreau" and "Juliette Moreau" are almost the same string at the same address.