Golden files and the acceptable set

Testing a feature built on a language model normally starts by borrowing the tools that already work on the rest of the codebase: a fixture for the input, an expected value committed beside it, an equality assertion between them. This topic is about which of those borrowings survive contact with a sampled generator, and which of them keep running long after they have stopped asserting anything. The golden file is the natural place to start, because it is the first tool a team reaches for when the output is too large to type by hand.

A golden file is a response the program produced once, which somebody read, judged acceptable, and committed to the repository as the expected answer; a golden set is the directory of them, one file per case, that every later run is diffed against. The technique exists because writing expected values by hand does not scale to large outputs — nobody wants to type a forty-field JSON document into an assertion, and the person who tries ends up checking the six fields they thought about while the other thirty-four go unwatched. Recording the real output covers every field at no cost and reduces the question "has anything changed?" to a diff. It is more familiar under other names: snapshot testing in Jest, where toMatchSnapshot() writes a __snapshots__ file on the first run and compares against it forever after; approval testing in the .NET and Java worlds; pytest-regressions in Python; recorded HTTP cassettes in VCR-style libraries. The workflow is identical in all of them — run the code once, eyeball the artifact, commit it, let the diff be the assertion. What this chapter works out is what that diff is actually pinning when the thing in the middle is a model call.

What a golden file pins#

Golden-file testing arrived from compilers and renderers, where it does excellent work. A regression in a parser shows up as a changed AST dump; a regression in a layout engine shows up as changed pixels. The technique rests on one assumption that nobody bothers to state, because in that world it is obviously true: the program is a function, so the same input yields the same output, and any difference means something changed in the code.

Sampling breaks the assumption. A language model emits a probability distribution over the vocabulary at each position and the decoder draws from it — the mechanism covered in topic 1.6, on how the next token is chosen. What gets stored in a golden file is therefore not the output of a summarisation function; it is the output of one draw, taken on one afternoon. The golden set pins one member of the set of sequences the model produces for that input, and says nothing about how large that set is or how many of its other members would have been equally acceptable.

That set is larger than intuition suggests, and it grows multiplicatively. A one-sentence summary of a support ticket is around 12 to 16 tokens. Suppose that at only four of those positions the distribution has two candidates each carrying more than 5% of the mass — "Duplicate" or "Customer", "charge" or "charged", "on" or "for", "4471" or "#4471". Four independent binary choices give sixteen distinct strings, every one of which the model will produce at a rate you would notice within a few dozen runs, and all sixteen of which satisfy every requirement the feature has. Real distributions are less tidy than that, and the number of outputs with non-negligible probability is normally in the hundreds. The stored blob is one of them. It was never special; it was first.

Sixteen is worth arriving at rather than accepting, because the reason it is a multiplication and not an addition is the whole of why acceptable sets are big. Start with two of those positions. The first can go two ways and the second can go two ways, and the second choice does not depend on the first, so every one of the first position's two options can be followed by either of the second's: Duplicate-charge, Duplicate-charged, Customer-charge, Customer-charged. Four strings from two binary choices. Add the third position and each of those four splits in two, giving eight. Add the fourth and each of the eight splits again, giving sixteen. Each new place the sampler can go two ways doubles the count of whole sentences rather than adding one to it, which is why the total is two multiplied by itself once per free position — sixteen for four positions, several thousand for a dozen.

That is also why the intuition "the model basically says the same thing every time" is both correct and useless. It is correct at the level of meaning: every one of the sixteen carries the same facts. It is useless at the level of bytes, which is the level an equality assertion works at, and the gap between those two levels grows by a factor of two for every position where the model has a genuine choice.

What that means for a build is worth saying plainly, because the resulting failure looks so much like a regression. The signature of a real regression — green for weeks, then red with nobody's fingerprints on the commit that broke it — is exactly the signature of a resample. A build gives you no way to tell the two apart, and that is the more expensive half of the problem.

Five runs, five summaries#

The cheapest way to see the size of the acceptable set is to stop reasoning about it and measure it. Run the same prompt against the same model five times and print what comes back.

import hashlib
import json

from anthropic import Anthropic

client = Anthropic()

TICKET = (
    "I was charged twice for order 4471 on 12 March - two lines of "
    "GBP 38.50 each on the same card. Please refund one of them."
)

PROMPT = (
    "Return JSON with keys `summary` (one sentence) and `next_action` "
    "(one of: refund, escalate, reply, close).\n\nTicket:\n" + TICKET
)

for i in range(5):
    message = client.messages.create(
        model="claude-opus-5",
        max_tokens=200,
        messages=[{"role": "user", "content": PROMPT}],
    )
    body = message.content[0].text
    digest = hashlib.sha256(body.encode()).hexdigest()[:8]
    print(i, digest, json.loads(body)["summary"])

Five runs, five different summaries, and the interesting part is not that they differ but what they have in common:

Every one of the five names the order id, describes a double charge, fits in one sentence, and routes to refund. Every one of the five would pass a human reviewer without comment. Four distinct strings appeared across five draws, which tells you the set has at least four members and almost certainly many more — a sixth run would very likely add a fifth string.

Take that inference apart, because it is the first of many in this topic where a small sample is asked to say something about a population. Four distinct strings in five draws is a lower bound and only a lower bound: you have seen four members, so there are at least four, and nothing in the experiment caps the number above. Runs 0 and 3 returned the same digest, and the temptation is to read that collision as evidence the set is small. It is weak evidence at best. Repeats show up quickly even when the number of possible outputs is large, because the probability mass is concentrated on a few common phrasings and the sampler returns those far more often than the rare ones. What a collision after five draws tells you is that some outputs are much more likely than others, which you already knew. To bound the size of the set from above you would need to keep drawing until new strings stopped appearing, and on a summary they do not stop.

The digest column is what makes that comparison mechanical rather than a reading exercise. A hash maps any string to a short fixed-length value, and equal strings always hash equal, so scanning the digest column for repeats answers "how many distinct outputs did I see" without anybody comparing sentences by eye. It is the same operation the golden-file assertion performs, run for information rather than for a verdict.

If the golden file happened to be recorded on a run that produced row 1, then under exact-string equality the suite reports a pass rate of one in five on this case. That number is real, but it measures how often the sampler reproduces one particular string, not whether the service works. Nothing in it moves when the feature actually breaks, and nothing in it stays still when the feature is fine.

Say precisely what that pass rate is a rate of, because the statement is sharper than it first looks. A golden-file assertion passes exactly when the sampler emits the one string in the file. Its long-run pass rate is therefore the probability the model assigns to that string — no more, no less. Two consequences follow, and both are uncomfortable. The measured number depends on which draw somebody happened to record: had the golden been taken on the run that produced the most common phrasing, the same unchanged service would report a much higher pass rate, and had it been taken on a rare one, a much lower. Nothing about the feature differs between those two worlds. And the number is a property of the sampler's distribution, so anything that reshapes that distribution — a new model version, a changed decoding setting, a reworded prompt that is equally good — moves it, while a change that makes every answer subtly wrong in the same way need not move it at all.

The failure that carries no information#

Suppose the summary really had broken. A prompt edit drops the order id, and the field comes back as "Customer was charged twice" — no number, so an agent reading it cannot find the order. That is a genuine bug and you want the build to catch it.

It does. It catches it in precisely the same way, with precisely the same message, as the synonym:

E       AssertionError: assert {'next_action':...} == {'next_action':...}
E         Common items:
E         {'next_action': 'refund'}
E         Differing items:
E         {'summary': 'Customer was charged twice'} != {'summary': 'Duplicate charge on order 4471'}

The two failures are indistinguishable at a glance, and the comparison is doing nothing to help you distinguish them:

Every row produces one bit: same or different. The severity ranges from nothing at all to a customer not getting their money back, and the assertion flattens all of it into the same red. Because the whole object is compared as one string, the failure cannot tell you which field moved without a human reading the diff, and it cannot tell you which requirement the moved field broke, because the requirements were never written down anywhere the test can see them. They live in the golden blob only in the sense that the blob happens to satisfy them.

Where exact comparison still works#

None of this is an argument against fixture comparison in general, and applying it that way costs you good tests. The code around a model call is full of ordinary deterministic work: the function that renders the prompt template, the retrieval query builder, the JSON parser that turns the model's text into a typed object, the routing logic that maps a next_action value onto a queue. Every one of those is a function in the compiler sense, and golden-file testing on them is the right tool for exactly the reason it always was. If a prompt template renderer starts emitting a different string for the same ticket, that is a code change, and a byte-level diff shouting about it is what you want.

The rule is about which side of the model call the assertion sits on. Inputs to the call are deterministic and can be pinned exactly. Outputs of the call are draws and cannot. A suite that pins the rendered prompt to a golden file and asserts properties on the response is testing both halves correctly; a suite that does it the other way round has them backwards.

It is also not an argument that model output is unbounded chaos. The five summaries above are strikingly similar. They agree on every fact, they agree on the routing decision, they agree on length and register. The variation is confined to a region, and the shape of that region is worth noticing: the facts are stable and the framing moves. Every run named order 4471, every run identified a double charge, every run routed to refund. What differed was whether the sentence led with the customer or the charge, whether the amount was included, whether "twice" or "duplicate" carried the meaning. Stable content with mobile phrasing is the whole reason property assertions work, and building them is the subject of the third chapter of this topic.

The size of the acceptable set is also not a constant. It scales with how much freedom the output format leaves, which means some model calls really can be exact-matched:

The size of an acceptable set is decided by one thing, and reading the table through it makes the rows predictable rather than memorable: how many distinct strings satisfy the requirement. An enum has one, because the requirement names the string — there is no way to write refund that is also refund and also different. An extracted field has a small handful, because the content is fixed by the input and only presentation is free, so normalising case and whitespace collapses those few into one. A summary has hundreds, because every position where the model could reasonably choose between two words multiplies the count, by the argument earlier in this chapter. Nothing about the model changes across those three rows. What changes is how much room the output format leaves.

A classifier that routes tickets into refund, escalate, reply and close has exactly one correct answer per case, and assert result["next_action"] == "refund" is a perfectly good assertion that will never go red for a synonym. This is why one team can have some model tests that behave like ordinary unit tests and some that do not, and why the distinction confuses people: the mechanism is identical in both, and only the output space differs. Notice too that next_action was stable across every draw in the five-run table — the top row of that JSON object was already exact-matchable, and the summary beside it dragged the whole comparison down.

That suggests a real design lever. Constraining the output — moving a decision out of prose and into an enum, returning a document id instead of a sentence naming the document — shrinks the acceptable set and buys back testability. It has limits, because the reason the feature exists is usually the prose, and a summary constrained until it can be exact-matched is no longer a summary.

A golden file, then, is a perfectly good instrument pointed at the wrong half of the system. It answers "is this byte-identical to the artifact somebody approved once", and on a template renderer or a parser that question is worth asking. On a sampled output it produces a number that drifts with the sampler, cannot rank its own failures by severity, and teaches the people reading it that red means nothing.

What happens first, in most teams, is that somebody tries to make the red build go away without changing what it asserts. The obvious moves are to mark the case flaky and quarantine it, or to wrap it in a retry so a passing draw somewhere in three attempts counts as a pass. Both are things engineering organisations already do, with policies written down, for tests that fail intermittently for entirely different reasons. Telling those two situations apart — and seeing what a retry does to the claim a test is making — is where the next chapter starts.