The replay harness

Attributing a quality regression to a model version is a measurement problem before it is a model problem. The version is one of several things that move on their own schedules — a retrieval index rebuilds, a chunker upgrade ships, a new customer arrives with unfamiliar documents — and each produces the same graph.

The instrument for telling them apart is a replay harness: a stored sample of requests you already served, re-run offline against two model versions with everything except the version pinned to what was originally sent. What it depends on is captured requests, which is the part teams discover they do not have on the morning they need it.

Three variables moved over one weekend#

Monday morning, an extraction endpoint's schema-validation failure rate — the share of responses whose JSON does not match the shape the parser demands — is 6.1%. On Friday afternoon it was 0.4%. Three things happened in between. The endpoint calls the model through a family alias, the moving pointer from chapter 1 that the provider re-targets on a schedule of their own. A retrieval index — the searchable store of document passages the endpoint pulls from to build the context it sends with each request — rebuilds every night at 02:00, and picked up an upgrade to the chunker, the component that decides where a document gets split into those passages, which shipped Thursday. And a new tenant onboarded on Friday afternoon with 41,000 scanned purchase orders in a layout unlike anything else the endpoint had seen, which by Sunday was 22% of its volume. Each of those three can move a schema-validation rate on its own, and the graph looks the same whichever one did it.

A single-variable experiment in a three-variable system#

The fastest experiment available is to pin to last month's snapshot, deploy, and watch whether the failure rate drops. It takes about twenty minutes and everybody will read the result as an answer about the model. It is not one, and the reasons are worth being precise about, because the same shape of mistake shows up in every regression investigation where more than one thing moved.

The precision starts with what an experiment can do. Attributing a change in an output to one input requires that the input be the only thing that moved between the two observations being compared. That is the whole of what a controlled experiment is, and the rest of experimental design is machinery for arranging it. Production on that Monday arranges the opposite: the failure rate is one number produced by a system with three inputs, all three moved over the same weekend, and what came out is a single total to which each contributed an unknown share. Pinning back moves one of the three and leaves the other two moving on their own schedules, so the second observation differs from the first in three respects rather than one, and no arithmetic on two totals recovers three contributions.

Production also cannot be made to supply the missing condition, because the other two variables are not yours to hold still. The index rebuild is a nightly job whose purpose is to keep running. The traffic mix is decided by customers who did not agree to stop sending documents while you investigate. You can stop the rebuild and you cannot stop the tenant, and even a stopped rebuild leaves you comparing this week's traffic against last week's.

If you pin back and the rate falls to 2%, four explanations fit that observation equally well: the model changed and pinning fixed it; the new tenant's Monday volume happened to be lower than its weekend backfill; the index reindexed twice more and the third rebuild was better than the first; or some combination in which the model contributed a third of the effect. Nothing in the observation distinguishes them, and the meeting will pick one anyway.

If the rate does not fall, the experiment is even less informative, because the tenant's documents may break under both versions. A model change that is real can be masked by an input distribution shift that is also real, and "we pinned back and it didn't help, so it isn't the model" is a conclusion people reach confidently from exactly this evidence.

There are two further costs. Pinning back spends your rollback before you know whether you need it: the previous snapshot is now serving production, so if the real cause turns out to be the chunker, you have made an unrelated change to the production model under incident conditions and you now own re-qualifying the pin you rolled back to. And if last month's snapshot is itself two months from retirement, what you have bought is days rather than a fix, with a migration deadline now sitting inside an active investigation.

The alternative is not more careful watching of production. It is to take the question out of production entirely.

Taking it out of production is legitimate because of a property of the interface, and the property is worth stating before anything is built on it. A request to a hosted model carries everything the model will use: the prompt, the retrieved passages pasted into it, the parameters, and the whole exchange so far if there is one. The provider keeps nothing between calls — there is no session on their side that your second request continues, and no state of yours ageing while you are not calling. That is why the same bytes sent two months later are not a reconstruction of the original request but the same request, and it is the property the entire harness stands on. An interface that held state between calls could not be replayed at all, because the state would have to be replayed too and you never had a copy of it.

What the property does not cover is anything your own code fetches at the moment of sending. The bytes are stable once assembled; the assembly is not. A request whose context comes from querying an index builds part of itself out of a store that has reindexed several times since, so re-running the code path produces different bytes and the replay is no longer of the request you served. That is why the capture record stores retrieved document ids and the index snapshot they came from rather than the query that found them. Capture moves the boundary of what is fixed outwards, from your API client to the last live thing the request touched.

The capture record a replay is built from#

A capture record has to contain enough to reconstruct the request byte for byte, plus enough to slice the results afterwards. The list is short and every field on it earns its place:

{
  "request_id": "req_7f2c91ab",
  "captured_at": "2026-08-01T14:22:07.418Z",
  "tenant_id": "t_9931",
  "feature": "extraction",
  "prompt_template_version": "extract-v14",
  "model_requested": "claude-sonnet-4-5",
  "model_served": "claude-sonnet-4-5-20250929",
  "params": { "max_tokens": 2048, "temperature": 0 },
  "retrieval": {
    "index_snapshot": "idx_2026-07-31T02:04Z",
    "doc_ids": ["d_5512#p3", "d_5512#p4", "d_8890#p1"],
    "context_sha256": "9c1e…a7"
  },
  "input_sha256": "3ab0…f1",
  "input_uri": "s3://captures/2026/08/01/req_7f2c91ab.json",
  "output_sha256": "b74d…22",
  "validation": { "schema_ok": false, "error": "invoice_total: not a number" },
  "latency_ms": 2140
}

The three fields that make the difference between a harness that works and a folder full of unusable rows are retrieval.doc_ids, prompt_template_version, and model_served. Storing the retrieved document ids and the index snapshot they came from means the replay can rebuild the exact context the model saw, rather than re-querying an index that has reindexed four times since. Storing the template version means you can pin the prompt to what was actually sent rather than what is on main today. And model_served, recorded off the response as chapter 1 established, is what tells you which snapshot to use as the baseline arm — without it you would be guessing at what the alias pointed to last Tuesday.

Storing the ids is a step short of storing the context, and the gap between the two is where the harness's most confusing outcome comes from — the run that finds no difference on either arm while production is still failing. An id names a passage; it does not carry the passage's text. If a rebuild re-chunked the documents, the passage that id refers to can be longer, shorter, or cut in a different place than the one the model actually saw, so a replay resolving ids against a later index rebuilds a context that is plausible rather than identical. Nothing in the run announces this. Both arms agree with each other, because both were fed the same wrong context, and the harness reports no model change while production keeps failing.

The capture record already carries the answer in context_sha256, a fingerprint of the context bytes as sent — a short value computed from the text such that different text almost never produces the same value. Recompute it over the rebuilt context and compare, and a mismatch says the rebuild is unfaithful before a single score has been read. Cases that mismatch either get their context restored from the stored snapshot or get excluded and counted, because a run that quietly drops the cases where retrieval changed is measuring the model on the traffic where nothing happened.

Capture does not have to be complete to be useful. Sampling 2% of an 1.1M-request day yields around 22,000 records, which at roughly 25 KB each is 16 GB a month, and gives every tenant above a few hundred daily requests a workable sample. Store the raw input under its own key with the retention policy that applies to customer content, and keep the metadata row in your warehouse where analysts can join it.

A uniform sample is a sample of volume rather than a sample of your customers, and the distinction decides what you are able to slice afterwards. Every request has the same chance of being kept, so the number of records a tenant contributes is proportional to how much traffic that tenant sends. An account sending a handful of requests a day lands in the store a handful of times a month, which is not enough to compute anything per-slice about it. That is a bad property in exactly the situation the harness exists for, because the account that prompts an investigation is often a small one whose documents are unusual rather than a large one whose traffic is ordinary. The correction is a floor rather than a higher rate everywhere: sample a larger proportion of tenants below some volume, or keep everything for a named list, which is affordable precisely because those tenants are small.

Four things held fixed, one varied#

Four things are held fixed across the two arms: the input bytes, the prompt template, the retrieval context rebuilt from the stored doc_ids rather than a live query, and the sampling parameters. One thing varies, which is the model version. Cohort membership is a second factor deliberately crossed with the first, so that a difference confined to the new tenant separates itself from one that applies to everybody.

ARMS = ["claude-sonnet-4-5-20250514", "claude-sonnet-4-5-20250929"]
N_SAMPLES = 5   # per case per arm, to separate a real gap from run-to-run variance

def replay(case, model_id):
    ctx = rebuild_context(case["retrieval"]["doc_ids"])      # stored ids, not a live query
    tmpl = template_at(case["prompt_template_version"])       # the template as sent
    resp = client.messages.create(
        model=model_id,
        max_tokens=case["params"]["max_tokens"],
        system=tmpl.system,
        messages=tmpl.render(load_input(case["input_uri"]), ctx),
    )
    text = resp.content[0].text
    return {
        "case_id": case["request_id"],
        "cohort": "new_tenant" if case["tenant_id"] == "t_9931" else "existing",
        "arm": model_id,
        "model_served": resp.model,      # confirms the arm actually served
        "schema_ok": validate(text).ok,
        "output": text,
    }

rows = [replay(c, m) for c in sample(1200) for m in ARMS for _ in range(N_SAMPLES)]

The tempting reading of that arrangement is that everything is now held still, so any difference between the two arms must be the model. It is tempting because the list of held-fixed things is genuinely complete on your side of the wire. It is wrong for a reason chapter 1 already established: a dated snapshot pins one set of trained weights and leaves free the picking step that turns the model's scores into text, the composition of the batch your request was scheduled into, and the hardware it landed on. None of those three is yours, and none of them is constant between two calls. Two runs of the same arm on the same case therefore disagree, and they disagree with nothing at all having changed.

The N_SAMPLES loop is not decoration. Extraction under a fixed prompt is not deterministic, and a two-point gap between arms on 600 cases can be sampling noise. Running each case five times per arm gives you a spread to compare against the between-arm difference, and it is the cheapest defence against declaring a regression that is not there.

The spread it produces is a yardstick made out of the system's own noise, and that is what makes it the right scale to read the arms against. A between-arm difference smaller than the within-arm spread is a difference the harness cannot tell from the system idling, since the same gap turns up between two runs of one arm where the version is identical by construction. A difference comfortably larger than the spread is one the version has to account for, because the version is the only thing the two arms do not share. Without that scale there is no threshold at all, and a run reports whichever gap it happened to produce.

The bill for that run is small enough not to need approval. Twelve thousand calls at roughly 1,200 input and 400 output tokens each is 14.4M input and 4.8M output tokens; at $3 and $15 per million that is about $115, and at eight concurrent requests it finishes in well under an hour. Compare that against the cost of an incorrect conclusion that sends two engineers at the chunker for three days.

Diffing outputs is not string comparison#

The harness produces 12,000 outputs and the useful question is how to reduce them to a number. Exact-match rate is the first thing everyone reaches for and the least informative: two extractions that differ only in whether invoice_total is "1284.50" or "1,284.50" are byte-different and semantically identical, so an exact-match rate of 31% tells you nothing about whether anything is wrong.

The reason it fails is in how the output is produced. The model emits a sequence of fragments one at a time, and a very large number of distinct sequences denote the same extracted values — a thousands separator, a trailing zero, a different field order, a space after a colon. Byte equality asks whether the two arms produced the same sequence. What you need to know is whether they produced the same values, which is a weaker property that many more sequences satisfy. Testing the stronger property when you care about the weaker one is why the number comes back low and means nothing: it counts cosmetic disagreements at full weight alongside real ones.

Normalisation is the step that closes that gap, and it is a decision rather than a formality. To normalise a field is to map every spelling of a value onto one canonical form before comparing — parsing both sides as numbers, or as dates, and comparing the parsed things instead of the text. Each normalisation you write is a claim that a particular difference does not matter to whoever consumes the output, and the claim can be wrong. If a downstream system parses the raw string rather than your parsed value, a formatting change is a real change, and normalising it away hides the regression you were looking for.

Three comparisons do most of the work on a structured task like extraction. Schema validity is binary and cheap and is the metric the failure rate is about, so it goes first. Field-level agreement — the proportion of fields whose parsed values match after normalisation, computed per field rather than per document — localises the change: a version that agrees on fifteen fields and disagrees on due_date in 40% of cases has one problem, not a general degradation. And for the cases that disagree, the position of the first divergence in the output is a surprisingly good triage signal, because a divergence in the opening tokens usually means a different overall interpretation of the document while a divergence at token 300 usually means one field went wrong.

def compare(a, b, fields):
    pa, pb = parse(a), parse(b)
    return {
        "schema_ok": (pa is not None, pb is not None),
        "field_agreement": {
            f: normalise(pa.get(f)) == normalise(pb.get(f)) for f in fields
        } if pa and pb else None,
        "first_divergence": next(
            (i for i, (x, y) in enumerate(zip(a, b)) if x != y), None
        ),
    }

For free-text tasks — a summariser rather than an extraction endpoint — none of those apply and the comparison has to be a judged one, which is slower and noisier and needs its own rubric. That asymmetry is worth knowing in advance: structured outputs are cheap to diff, and prose outputs are not, so the surface where you most want a replay harness may be the one where building it costs the most.

Reading the four cells#

The 2×2 gives four numbers, and each pattern points somewhere different.

That last row is the one people forget to plan for, and it is the most instructive. If replay with stored contexts shows no difference at all while production is still failing at 6%, the harness has eliminated both the model and the input distribution, and the remaining suspect is the thing you deliberately froze. Re-running one arm with contexts fetched from the live index instead of the stored doc_ids turns the frozen variable back on, and a gap between those two runs is the chunker. The harness is not only a model-change detector; it is a way of taking variables out of the system one at a time and putting them back deliberately.

Reading that row correctly depends on the fingerprint check above. A harness whose stored contexts failed to rebuild faithfully produces the same fourth cell — no difference on either arm — for an entirely different reason, because both arms ran on a context nobody ever served. The two cases are distinguishable only by whether the rebuilt context still matched context_sha256, and they point in opposite directions: one says go and look at retrieval, the other says the run was invalid and has to be done again. Verifying the fingerprints before reading the cells costs nothing and is the difference between a finding and a coincidence.

The industry name for the general practice is regression testing — re-running cases whose behaviour you already know in order to check that a change has not broken something that used to work — and Watch the language: A tutorial on regression testing for LLMs walks the same loop with a different worked example. The framing that matters here is narrower than a test suite: this is a differential between two versions of one component, run on your traffic, to attribute a change rather than to approve one.

Divergence is a pointer, not a verdict#

A replay harness answers "did the model's behaviour change on these inputs". It does not answer whether the change is bad, which is a question about your users rather than your diff. A snapshot that reformats dates from 03/04/2026 to 2026-04-03 will show enormous output divergence and may be strictly better; a snapshot that quietly drops one optional field in 2% of cases will show almost no divergence and may cost you a customer. Divergence rate is a pointer at things to look at, and somebody still has to look.

A harness cannot replay traffic nobody captured#

It also cannot recover what you did not store. If the first capture record was written the morning of the regression, there is no baseline traffic from before the change, and the harness has nothing to replay. Capture is infrastructure you build on an ordinary week for use on a bad one, which is why the low sampling rate matters: 2% is cheap enough that nobody argues about it in advance, and dense enough to answer the question afterwards.

Cached summaries and stored embeddings sit outside the differential#

The harness does not extend to anything stateful downstream. Cached summaries, stored embeddings — the numeric vectors a model produces to represent a piece of text so that similar texts can be found by comparing them — and materialised classifications were produced by whichever model ran at the time; replaying the request tells you what the model does now, not what is sitting in your database from six months ago. Those artefacts need their own reasoning about when they get regenerated, and pretending a replay covers them is how a migration ships with half a document store embedded by one model and half by another.

A regression investigation with three moving variables has one honest answer and several confident ones, and what separates them is whether the experiment held anything still. Replay holds things still for roughly a hundred dollars and under an hour, and the cost of not being able to run it is a capture pipeline you needed months earlier.

Suppose it comes back the way it did here — the alias moved, both cohorts regress, and you have pinned to the snapshot that works. You have bought time. How much time is a question with an actual answer, printed on a page you have probably never opened, and the answer is usually three different dates that people treat as one.