An eval suite outlives prompts, APIs and model names, or it is not one

The commit was titled rewrite summariser prompt, +8% on user feedback, and it deleted tests/test_summariser.py, all 1,140 lines of it, with a message in the body explaining that the tests were failing because the prompt had changed and the expected strings no longer matched. That was accurate. The file contained sixty cases, each of which looked roughly like this:

def test_summary_mentions_refund_window():
    resp = openai.ChatCompletion.create(
        model="gpt-4-0613",
        messages=[
            {"role": "system", "content": SUMMARISER_PROMPT_V3},
            {"role": "user", "content": TICKET_4471},
        ],
        temperature=0,
    )
    out = resp["choices"][0]["message"]["content"]
    assert out.startswith("Summary:")
    assert "30 days" in out
    assert len(out.split("\n")) == 3

Sixty cases, eleven months of accumulated knowledge about what this feature is supposed to do, and every one of them was unrunnable the moment the prompt stopped emitting a Summary: prefix. The knowledge was real — ticket 4471 genuinely is a case where the refund window has to survive into the summary, and someone learned that from an incident — and it was stored in a form that could not survive the system changing. Six weeks later the team moved to a different provider whose response object is shaped differently, which would have killed the same file a second time.

The failure is that each case welds together three things that change on completely different schedules: the situation being tested, the machinery that turns a situation into an output, and the judgement about whether the output is acceptable. Situations are durable, because a ticket that needed its refund window preserved in 2025 still needs it in 2027. Machinery changes every few weeks. Judgement changes rarely and deliberately. A suite that survives is one where those three live in separate places and only the middle one is allowed to churn.

Software testing worked this out decades ago for a related problem and the vocabulary transfers. Golden-file testing separates the fixture from the assertion so that a compiler's output format can change without invalidating the corpus of programs it is tested on; characterisation testing records what a legacy system currently does so that a rewrite can be checked against it. Both rest on the same structural claim, which is that the valuable, slowly-accumulated part of a test is the situation and the expectation, and the plumbing that connects them to a running system is disposable. The model-evaluation version arrived late partly because the first generation of these suites was written inside application repositories by people adding a few assertions next to their feature code, where the plumbing is right there and separating it feels like ceremony.

The case is data, and it is the only durable part#

A stored case holds the input, the expected outcome, and whatever the outcome needs in order to be checkable. It holds no prompt, no model name, no SDK call and no response parsing, because all four of those belong to the machinery. Written that way, a case is a small document that a person can read six months later and still understand.

# cases/summarise/0047.yaml
id: summarise-0047
added: 2025-04-02
provenance: incident INC-219 — refund window dropped from summary, agent quoted 14 days
input:
  ticket_id: TICKET_4471
  body_file: fixtures/tickets/4471.txt
  locale: en-GB
expect:
  must_contain_fact:
    - key: refund_window
      value: "30 days"
  must_not_contain_pii: true
  max_words: 60
  structure: three_sections   # situation, action taken, open question
scoring: deterministic
weight: 3          # incident-derived cases count triple
version: 2         # v2 added max_words after the 400-word summary regression

Nothing in that file mentions how the summary gets produced, which is what lets it outlive the production of summaries. The provenance line is the part people leave out and then regret, because in eighteen months somebody will propose deleting this case on the grounds that it seems arbitrary, and the answer to that proposal is the incident number.

The machinery goes behind an adapter with one method, so that "the system under test" becomes a thing you can swap rather than a thing woven through sixty files. An adapter for the current production path assembles the live prompt and calls the live provider; an adapter for a candidate model does the same with a different client; an adapter for a hypothetical future provider is written on the day that provider exists and touches nothing else.

from dataclasses import dataclass
from typing import Protocol

@dataclass(frozen=True)
class Case:
    id: str
    input: dict
    expect: dict
    weight: int
    version: int

class System(Protocol):
    name: str          # "prod-2026-07 / claude-sonnet-4-5"
    def run(self, case: Case) -> "Output": ...

@dataclass(frozen=True)
class Output:
    text: str
    latency_ms: int
    input_tokens: int
    output_tokens: int
    cost_usd: float
    raw: dict          # kept for forensics, never asserted on

Output is deliberately provider-neutral and deliberately carries more than the text. Latency, token counts and cost are recorded on every run of every case because they are free at the point of capture and expensive to reconstruct later — the next chapter's Thursday question turns on having eleven months of them already in the results table rather than measuring them under time pressure.

The scorer belongs to you, not to the model#

Judgement is the third piece, and the constraint on it is independence: the thing that decides whether an output is acceptable must not be part of the thing being evaluated. The version of this that catches people uses the same model, with a prompt like is this a good summary?, to grade its own output. That arrangement moves in lockstep with whatever it is measuring, so a model that has become more confident and more wrong scores itself higher, and a model release changes both the system under test and the ruler at the same time, leaving you with a number that cannot be compared to last month's.

Deterministic scoring covers more ground than teams expect once cases are written with checkability in mind. Fact presence, forbidden strings, schema validity, word counts, structural constraints, tool-call sequences and citation resolution are all decidable in code, run in microseconds, and cost nothing. The summarise-0047 case above is fully deterministic, and it was made fully deterministic on purpose by phrasing the expectation as the string "30 days" appears rather than the summary conveys the refund window, which is a small loss of fidelity in exchange for a check that will still work in 2030.

Where the judgement genuinely needs a model — tone, helpfulness, whether an explanation is coherent — the arrangement is an LLM judge, meaning a second model that is given the output, a written rubric describing what an acceptable answer looks like, and a request for a verdict against it. The judge is then pinned as part of the instrument. That means a specific model version, a specific judge prompt, a specific temperature of zero, and a rule that changing any of the three is a ruler change requiring the previous system to be re-scored so the two numbers can be related. Treating the judge as infrastructure rather than as a convenience is what keeps a model-graded score comparable across releases.

def score(case: Case, out: Output) -> "Result":
    checks = []
    for fact in case.expect.get("must_contain_fact", []):
        checks.append(Check(fact["key"], fact["value"] in out.text))
    if case.expect.get("max_words"):
        checks.append(Check("max_words",
                            len(out.text.split()) <= case.expect["max_words"]))
    if case.scoring == "judged":
        checks.append(judge(              # pinned, versioned, temperature 0
            rubric=case.expect["rubric"],
            output=out.text,
            judge_model="claude-sonnet-4-5-20250929",
            judge_prompt_version=4,
        ))
    return Result(case_id=case.id, case_version=case.version,
                  passed=all(c.ok for c in checks), checks=checks,
                  latency_ms=out.latency_ms, cost_usd=out.cost_usd)

A result row records the case version and the system name alongside the verdict, because a run is only interpretable as a triple of case set, system and date. A results table keyed that way answers the questions that come up later — did this case ever pass on the old model, when did it start failing, what did it cost then — without anyone needing to rerun anything. Ninety cases against four systems is 360 rows, which is a rounding error in any database and the difference between a claim and a recollection.

Recovering a suite that has already died#

The deleted file was recoverable, which is the usual situation, and the recovery is instructive about what proportion of a welded suite was ever worth keeping. Reading the sixty tests in the git history and asking of each one what did somebody learn that made them write this produced three piles. Thirty-eight were genuine situations with genuine expectations — this ticket must retain its refund window, this input must not produce a name in the output, this malformed order reference must be echoed rather than guessed at — and those became cases in about a day and a half of work, mostly spent tracking down the fixtures the tests had inlined as string constants.

Fourteen were assertions about the prompt's output format rather than about the task: startswith("Summary:"), the three-newline structure, a specific bullet character. Those encode a formatting contract that changed on purpose, and the correct home for them is a schema check on the production path rather than a case in a comparison suite, because a model should not be penalised in a switching decision for failing to reproduce a prefix that the current prompt happens to request.

Eight were tests of the SDK, verifying that a response object had the shape the SDK documents. They were deleted without replacement.

Of 1,140 lines, roughly 60% of the knowledge survived and none of the code did, which is the ratio to expect and the reason the migration is worth doing rather than starting from scratch. Rebuilding thirty-eight cases from scratch means rediscovering thirty-eight incidents, and the incidents are the expensive part.

Two neighbours it gets mistaken for#

This is not a unit test suite, even though it borrows the vocabulary and the file layout. Unit tests are deterministic and binary: a failure is a defect, and a suite with one red case is a broken build. A model eval is a sampled measurement of a stochastic system, so an individual case flipping between runs is normal, the meaningful signal lives in aggregates and in per-case histories, and the correct response to a two-point drop is to look rather than to revert. Wiring an eval suite into CI as a pass/fail gate on every commit produces so much noise that the gate is disabled within a month, which is a worse outcome than not having built it. The suite runs on model changes, prompt changes and a schedule, and it produces a report that a person reads.

It is also not the golden dataset from building a golden dataset, though the two are close relatives and can share fixtures. A golden dataset is sized and stratified to estimate product quality with a defensible error bar, which is why it runs into the hundreds of cases and needs sampling discipline. The suite described here is an instrument for comparing systems, so it optimises for stability and cheapness of rerun rather than representativeness. Sixty to a hundred cases, heavily weighted toward past failures, executable in under ten minutes for a few dollars, is a suite that actually gets run on the Wednesday after a Tuesday release. A 1,400-case golden set that takes six hours and $80 is a suite that gets run twice a year, and an instrument you do not run is not an instrument.

Nor is it a public benchmark. The numbers it produces are meaningless to anyone outside your codebase, which is a feature; nobody can train on it, nobody argues about its methodology, and it measures the tasks your product actually performs. The corresponding limitation is that it tells you nothing about capabilities you have not thought to encode, so a model that is dramatically better at something you do not currently ask for will score exactly the same as one that is not.

What the suite gives you, once cases are data and the system is swappable, is the ability to point it at a model that did not exist when the cases were written and get a quality number back the same afternoon. Quality alone does not settle a switching decision, because the model that scores three points higher may be four times the price or twice the latency at your prompt lengths, and the version of that comparison assembled after somebody asks for it takes a fortnight and produces an anecdote.