A revision id alone will not reproduce yesterday's answer

Provenance, for a model call, is the set of identifiers that together name every input the model was given, recorded at the moment the call was made rather than reconstructed afterwards. The previous three chapters produced one of those identifiers — an immutable prompt revision — and a request log carrying it looks like the loop closing. It is not, because the prompt is one of at least four inputs that move on their own schedules, and the other three move without anybody on your team doing anything.

The test of such a record is a replay: take a logged request, reconstruct its inputs, send it again, and see whether the behaviour comes back. When it does not, a good record names which input moved and a poor one leaves you guessing.

Everything else that was an input#

Three candidates account for almost all of these cases, and it is worth walking each one because the way they drift is different.

The model is the first. A request that specifies model: "orders-agent-v4" is naming an internal alias that maps to a provider model string in configuration, and a platform team moving that mapping from a dated snapshot to the next one to pick up a price reduction changes what serves without touching the service. The alias in the log is stable by design, which is what makes it useless as provenance. Providers publish both floating names and dated snapshots for exactly this reason, and a floating name is a pointer with none of the properties the pointer in this topic required: no approval, no immutable id in your logs, and a change you may learn about from a changelog rather than a deploy.

The decoding parameters are the second — the settings that govern how the model picks each next token, chiefly temperature, top_p and max_tokens, where a higher temperature makes the less likely continuations more available and a lower one drives the model toward its single most probable wording. An endpoint running at temperature: 0.7 because product wanted varied phrasing in summaries will typically have that value in a config file that "never changes", unlogged, until somebody tuning a summariser edits the shared client defaults. A replay today at whatever the current defaults are is not the same experiment, and the difference between 0.7 and 0.2 on a hedging-sensitive question is very plausibly the difference between one answer and its opposite.

Why that is more than a change of phrasing is worth one step down, and one step only. At each position the model produces a score for every token it could emit next, and the decoding parameters decide how a single token gets chosen from those scores. Temperature rescales them before the choice is made: a higher value flattens the differences and gives lower-ranked continuations a real chance of being picked, a lower value sharpens them toward the single most probable one. On a question where the honest answer hedges, covered and not covered are both live continuations with scores close together, and the whole of the rest of the answer follows from whichever one was emitted, because every token after it is conditioned on it. That is how a decoding parameter nobody logged becomes a support ticket about a warranty.

The retrieved context is the third and the most slippery. An agent that retrieves three chunks from a policy index before answering — chunks being the short passages the policy documents were cut into and indexed as, so that a search step can pull the few most relevant ones into the prompt at request time — usually logs that retrieval happened and how many chunks came back, and not which ones. When the underlying document is edited and reindexed, the chunk boundaries move, and the same query later returns text that is both different and better. Replaying the question retrieves the corrected policy and produces the corrected answer, which tells you nothing about what the model saw at the time.

What makes that one hard to catch is what a chunk id actually names. Chunking splits a document into passages by a rule applied to the document's text — so many characters, or the boundaries between headings — and the ids are usually assigned by position, so one of them means the seventh passage of that policy. Edit a paragraph in the middle of the document and every split point after it moves while the numbering does not. pol_warranty_ext#7 is therefore a stable name for whatever happens to be seventh today, and a log holding only ids has recorded a name that outlives the thing it named. Storing a hash of the passage beside its id is what turns a silent substitution into a detectable one, which is why the table below asks for both rather than either.

Two further inputs belong on the list even though they are less often the culprit. The conversation history is part of the prompt in every sense that matters — a request on turn six is governed as much by the five turns before it as by the system prompt, and a log line that records only the latest user message cannot reconstruct what the model was reading. And the client library contributes defaults you did not write: a tool-choice setting, a default max_tokens, an automatically injected instruction about tool formatting, or a change in how it serialises multi-part content. Those move when you upgrade the SDK, which is a dependency bump that nobody thinks of as a behaviour change, so the library version belongs in the record alongside everything else.

The useful comparison is a build. A commit sha tells you which source produced an artefact and does not let you rebuild it, because the build also consumed transitive dependency versions, a compiler and a base image; the lockfile and the toolchain pin exist to close that gap. A prompt revision is the commit sha of an LLM call. The model snapshot, the decoding parameters and the retrieved chunk ids are the lockfile, and a request record without them is a repository with no lockfile checked in.

What the record has to carry#

Pinning the model to a dated snapshot in configuration, rather than only recording it, is the stronger version of that row, and it has a cost worth naming so the trade is made knowingly. Snapshots are retired on a published schedule, typically with several months of notice, so a service pinned to one has a recurring migration to do — evaluate the new snapshot, compare on the frozen set, move the pin — perhaps two or three times a year per model. Floating aliases remove that chore by making the migration happen without you, at whatever moment the provider chooses, with no rollback available and no entry in your change log. Teams that ship anything regulated pin snapshots for exactly this reason, and teams that pin nothing tend to discover the trade during an unexplained quality drop.

Recording the model from the response rather than the request is a small detail that decides whether the whole table works. Most providers echo the resolved model in the response body, and that value is the one that was actually served; the string in your request is your intention, and during a provider-side alias move those two disagree without any change on your side.

{
  "request_id": "req_8812f30a",
  "ts": "2024-11-14T10:22:41.117Z",
  "prompt": {"name": "warranty_agent.system", "revision": "pr_7c1f9a2b", "source": "control_plane"},
  "model": {"requested": "orders-agent-v4", "served": "claude-sonnet-4-5-20250929"},
  "decoding": {"temperature": 0.7, "top_p": 1.0, "max_tokens": 800, "stop": ["</answer>"]},
  "tools": {"count": 6, "schema_sha256": "b31f0a92"},
  "retrieval": {
    "index_snapshot": "policies@2024-11-11T03:00Z",
    "embedding_model": "text-embedding-3-large",
    "chunks": [
      {"id": "pol_warranty_ext#7", "sha256": "4c9e11d0", "score": 0.81},
      {"id": "pol_warranty_ext#8", "sha256": "0af7c2b5", "score": 0.77},
      {"id": "pol_returns#2",      "sha256": "e28d9134", "score": 0.62}
    ]
  },
  "template": {"engine": "jinja2@3.1.4", "vars_sha256": {"customer_tier": "9a1c", "question": "77be"}},
  "usage": {"input_tokens": 2148, "output_tokens": 312}
}

With that record, the replay stops being a fresh experiment and becomes a reconstruction:

def replay(record: dict) -> str:
    prompt = revision_store.get(record["prompt"]["name"], record["prompt"]["revision"])
    chunks = [chunk_store.get(c["id"], at=record["retrieval"]["index_snapshot"])
              for c in record["retrieval"]["chunks"]]
    for c, logged in zip(chunks, record["retrieval"]["chunks"]):
        if sha256(c.text) != logged["sha256"]:
            raise ContextDrift(f"{logged['id']} changed since {record['ts']}")
    return client.messages.create(
        model=record["model"]["served"],          # the snapshot, not the alias
        system=render(prompt, chunks, record["template"]),
        **record["decoding"],
    )

The ContextDrift check is the part that repays the effort fastest. When the chunk text has changed under a stable id, the replay cannot be faithful and the harness says so instead of quietly running a different experiment. Storing the hash costs 32 bytes per chunk and converts a silent wrong answer into an explicit one.

Bitwise replay is not what this buys#

Pinning every field in that table does not make the call deterministic, and promising that it does will cost you credibility the first time someone checks. Providers serve requests on batched GPU inference where the composition of the batch affects floating-point reduction order, mixture-of-experts routing — where each token is handled by a subset of the model's parameters chosen at run time — can depend on what else is in the batch, and kernel selection varies with hardware. On one internal test, the same fully pinned request replayed twenty times at temperature: 0 produced seventeen byte-identical outputs and three that differed; two of the three differed only in phrasing and one selected lookup_shipment where the majority selected lookup_order. A seed parameter, where a provider offers one, reduces the variance and is documented as best-effort rather than a guarantee, because the backend it is seeding is not stable across deployments.

The belief that comes apart there is that temperature zero means deterministic, and it is worth being exact about which half of it is true. Temperature zero does make the selection deterministic: given the scores, take the highest. What it does not make deterministic is the scores. Each one is the end of a long chain of floating-point additions, and floating-point addition is not associative, so summing the same quantities in a different order can give a slightly different result. The order depends on how your request was grouped with other people's into a batch, on which kernel the hardware chose, and on a mixture-of-experts model on which subset of parameters each token was routed to. None of those is yours to fix. Almost always the differences change nothing, because the top score sits well clear of the second. When two candidates are close — which is exactly the hedging case above — a difference in the last bits is enough to swap them, and every token after that one follows the other branch. Seventeen identical replays out of twenty is what that looks like from outside.

The purpose of the record is therefore attribution rather than proof. Before it, "why did this answer differ" has an unbounded answer set. After it, the replay either reproduces the behaviour — in which case you have a reproducible case to attach to an eval suite — or it fails to, and the record tells you that the model snapshot moved on the eighteenth and the policy chunk changed on the nineteenth, which is two concrete things to check instead of a week of forensics. That is the same standard a lockfile meets: it does not guarantee a bit-identical binary on every machine, and it does turn "the build broke" into "this dependency moved".

All of it depends on being able to find the record at all, which is a plumbing problem that sinks more investigations than the missing fields do. A complaint arrives as a support ticket quoting a sentence of prose, and matching that back to req_8812f30a requires the request id to have travelled outward: rendered in a debug footer, attached to the conversation object the support tool displays, or returned in a response header the front end stores. Without it the search is by user and timestamp across a day of traffic, which works until the user has had four conversations and the timestamps are in the wrong timezone. Surfacing the id costs a line of code and is the difference between a five-minute lookup and an hour of grep.

It is also not observability in general. Traces and spans tell you where time went and which service called which; this record tells you what the model was given. The two live in the same log pipeline and answer different questions, and a team with excellent distributed tracing and no prompt provenance will still spend a day on a warranty ticket.

The part you cannot afford to store naively#

The obvious next step — store the fully rendered prompt for every request — runs into arithmetic. At 60 requests per second a service handles about 5.2 million calls a day, and a rendered prompt with three retrieved chunks runs around 8 KB, which is 41 GB a day before the response. That is a real bill and, more awkwardly, a copy of every customer question and every retrieved document sitting in a log store with a different retention policy and a different access-control model from the systems those came from.

Both multiplications are worth carrying, because every decision left in this section is a fraction of one of them. Sixty requests a second across the eighty-six thousand seconds in a day is a bit over five million calls, and five million prompts at about eight kilobytes each is the forty-one gigabytes. The sampling arrangement then reads straight off the same figures: keeping the full text for one request in a hundred keeps a hundredth of forty-one gigabytes, which is the four hundred megabytes, while the seven-hundred-byte record kept on every request is about three and a half gigabytes a day, and a year of that is the terabyte and a bit quoted further down. The shape worth noticing is that the cheap record scales with coverage and the expensive one scales with retention. That is why they get two policies rather than one compromise.

The arrangement that survives both constraints is to store the hashes always and the full text selectively: every request gets the record above, which is roughly 700 bytes, and full renders are retained for a 1% sample plus every request that errored, was escalated, or was thumbed down. That keeps 100% coverage of the identifiers needed to reconstruct and about 0.4 GB a day of raw text, and it means a complaint that arrives about an unsampled request is reconstructed from the pins rather than read off a stored blob. Field-level redaction on the stored renders is a separate requirement and belongs to whatever policy governs the underlying data, not to this pipeline.

Retention is the other constraint that decides whether any of this pays off, and it should be set against how long complaints take to arrive rather than against a default. A warranty ticket landing seven days after the conversation is fast for a customer complaint; disputes routed through a partner or a regulator routinely surface at sixty or ninety days. A log pipeline retaining fourteen days answers the first case and none of the others, so the 700-byte record is worth keeping far longer than the sampled renders — a year of it at 5.2 million requests a day is around 1.3 TB in cold storage, which is affordable in a way that a year of full renders is not.

What all of this sets up is the ability to say, with evidence, that behaviour changed at revision pr_7c1f9a2b and not before. That sentence is only actionable if the revision means something — one revision, one decision — and a pull request that reorders sections, rewraps lines and changes a rule in a single commit produces a revision id that cannot be interpreted at all. Keeping revisions interpretable is the reviewer's job, and it is the subject of the next chapter.