A floating alias makes last month's answer unreproducible

The escalation arrives as a forwarded email with a screenshot. On the 3rd of last month, an enterprise customer's trade compliance team asked your assistant about export restrictions on a category of equipment, and the answer they were given was wrong in a way that mattered: it stated that a licence was not required for a destination where one is. They acted on it. Their legal counsel is on the thread, and the question they have asked is narrow and reasonable — what happened, and is it fixed.

You go to the logs. The request identifier from the screenshot's footer resolves to a row, and the row contains the response text, a latency of 3.4 seconds, an HTTP 200, a tenant identifier, a user identifier, and a timestamp of 14:12:07 UTC on the 3rd. It also contains the string claude-sonnet-4-5, which is what your code sent, and not what served it. There is no request body, because the client wrapper logs responses and metrics on the grounds that request bodies are large and contain customer data. The prompt lives in an internal tool where three people have edit rights and the tool keeps no history. The retrieval index was rebuilt on the 19th.

What the row supports, and where speculation begins#

Being precise here is not pedantry; the difference between the two columns below is the difference between what you can put in writing to a customer's counsel and what you cannot.

Everything below the first row is speculation, and speculation dressed as a finding is how these threads become considerably worse. The temptation is to run the question through production now, watch it answer correctly, and reply that the issue is resolved. That reply is unsupported on three independent grounds. You do not have the question, only a quoted answer, and reconstructing a plausible question is not the same as reproducing the one that was asked. The stack differs from the 3rd's in at least the model snapshot behind the alias, the prompt, and the retrieval index, so a correct answer today is a statement about a different system. And a single correct sample from a stochastic system is weak evidence about a defect that may have a 15% incidence — the run-to-run spread you learned to measure when you built the CI gate applies here too, at n = 1.

What you can honestly say has a shape. You can confirm the interaction occurred and quote the answer. You can characterise the defect from the answer's content — an assertion made without a source, on a jurisdiction question, in a flow where the prompt at the time may not have required citation. You can list what has changed since, with dates. You can say that you cannot reproduce it, and say why, which is more credible than a confident claim they can see through. And you can describe what you are changing so that the next escalation gets a demonstration rather than an apology. Customers with compliance teams are considerably more comfortable with "we cannot reproduce this and here is why, and here is what we are fixing about that" than with a reassurance that dissolves under one follow-up question.

Four changes, and what each costs#

{
  "request_id": "req_9f2c41",
  "ts": "2026-07-03T14:12:04.221Z",
  "tenant_id": "t_8812",
  "surface": "compliance_assistant",
  "model_requested": "claude-sonnet-4-5",
  "model_resolved": "claude-sonnet-4-5-20250929",
  "params": { "temperature": 0, "max_tokens": 1024, "top_p": 1 },
  "prompt_id": "compliance_answer",
  "prompt_version": "v41",
  "prompt_sha256": "3b1f8ac2d0e5...",
  "messages_ref": "s3://llm-audit/2026/07/03/req_9f2c41.json.zst",
  "retrieval": { "index_version": "kb-2026-06-28", "doc_ids": ["kb_4471", "kb_1180"] },
  "tools_called": [{ "name": "lookup_eccn", "args_sha256": "9c0a...", "result_ref": "s3://..."}],
  "usage": { "input_tokens": 4188, "output_tokens": 262 },
  "latency_ms": 3412,
  "output_ref": "s3://llm-audit/2026/07/03/req_9f2c41.out.txt"
}

Pinning to a dated snapshot removes the first unknown by making the model identity a property of your deployment rather than of the vendor's calendar. Its cost is a migration obligation on a schedule you do not control, which is a large enough argument to be the whole of the next chapter.

Logging the resolved identifier is the cheapest of the four by a wide margin. The value is already in every response body; capturing it adds one field of about 30 bytes per request and a line in the client wrapper. It is worth doing even if you pin, because it verifies that the pin is in force — an SDK default, a proxy rewrite, or a fallback path can quietly send something else, and the response tells you.

Immutable prompt versions mean the prompt is content-addressed and published rather than edited: an identifier and a version, a hash of the exact rendered template, and a store where v41 will still return the same bytes in a year. The cost is friction on the people who currently edit the prompt in place, which is usually the point of the internal tool. A registry with a draft-and-publish flow keeps most of that convenience while making published versions immutable, and the request log then carries prompt_version and prompt_sha256 rather than a promise.

Two properties of the prompt store matter beyond immutability. The stored artefact should be the rendered template rather than the source, including whatever your code interpolates around it — the tool descriptions, the output schema, the system preamble your framework adds by default — because a template that references a tool list which has since gained an entry is not the prompt that ran. And the hash should cover the whole rendered string, so that a replay can assert equality rather than trusting a version label somebody may have overwritten.

Retaining inputs is the one people expect to be expensive on storage and is not. At 1.2 million requests a day with an average serialised record of 6 KB compressed, you are writing about 7 GB a day and 220 GB over a 30-day retention, which is a few pounds a month on object storage. The genuine cost is legal and it is substantial: those payloads contain customer data, so retention has to be covered by your data processing agreement, encrypted, pinned to the right region, deletable on request, and time-limited. It also enlarges the blast radius of any breach. Sampling is the usual compromise — full retention on the surfaces where an escalation is plausible and expensive, a 2% sample elsewhere, with a shorter TTL on both — the time to live, the age at which the store deletes a record without being asked.

Choosing the TTL is an empirical question that most teams answer by guessing 30 days. The distribution to look at is how long your escalations take to arrive: pull the last two years of them, measure the gap between the interaction date and the day it reached engineering, and set the retention above the p95 of that gap, the age that all but one escalation in twenty comes in under. For a product sold to enterprises with quarterly compliance reviews, that gap has a long tail — a median of 9 days and a p95 of 34 is typical, because an incident has to be noticed internally, raised with an account manager, and forwarded — and a 30-day TTL would have destroyed the evidence for one escalation in twenty. Ninety days on the high-stakes surfaces, fourteen on the rest, is a defensible shape, and the numbers should come from your own history rather than from this paragraph.

Storage is also where the privacy work is concentrated, and doing it at write time is much easier than retrofitting. Redact the categories you have no business keeping before serialising — payment details, credentials, anything your classifier flags as special-category data — encrypt the payloads with a key scoped per tenant so that a deletion request is a key destruction rather than a hunt through object storage, and write the records append-only with the bucket's object lock enabled. That last property is what makes the log usable in a dispute: a record that engineering can edit is a record the other side's counsel is entitled to be sceptical about, and immutability at the storage layer costs nothing.

Replay is the point of all of it#

The four changes are only worth their cost because together they make a stored request executable again.

That diff is what turns an escalation into an investigation. Running the stored request against the snapshot it originally used tells you whether the behaviour is intrinsic to that configuration; running it against today's model and today's prompt tells you whether the change since then fixed it or merely moved it. Doing it thirty times rather than once turns "it answers correctly now" into an incidence rate, which is the form a compliance team can act on.

Be careful about what an incidence rate from thirty replays supports. Four failures in thirty is a point estimate of 13% with a 95% interval running from about 4% to 31%, which is wide enough that "roughly one time in eight, and certainly not rare" is the strongest claim available. Zero failures in thirty under today's configuration bounds the rate below roughly 10%, which is a genuine improvement over a single passing run and is not the same as saying it cannot happen. If the surface is important enough that the difference between 2% and 10% matters, run three hundred replays; at a few pence each that is an affordable answer to a question the customer is entitled to ask precisely.

Replay is not a promise of identical output. Provider inference is not bitwise reproducible even at temperature zero, so what you reproduce is a distribution rather than a transcript, and a fair conclusion is "the defect appears in 4 of 30 replays under the 3rd's configuration and 0 of 30 under today's". Anyone expecting the exact original string back will be disappointed, and a process that depends on getting it is built on a guarantee nobody offers.

It is also worth separating this from the tracing you already have. Distributed traces record that a call happened, to which service, and how long it took, which is what they are designed for and why the escalation could be located at all. Provenance records what the call contained and which artefacts it bound to, and the two are usually stored in different systems with different retention. A team with excellent observability and no request retention has complete information about the shape of an incident and none about its content.

Of the four changes, three are engineering work that a team can simply decide to do. The fourth is a policy argument that has a second side, and the platform team proposing it and the product engineers resisting it are both correct about something.