Freezing a trace

Debugging one bad output is a different activity from measuring quality across a sample, and the difference is in the question being asked. A quality measurement asks how often something happens, and answers with a rate over many cases. A single bad answer asks which part of one request produced one piece of text, which is a causal question, and causal questions are settled by comparing runs that differ in one thing. This topic works through that comparison from the first screenshot to the decision about what, if anything, to change. This chapter is about the step that gets skipped, which is preserving the thing to be compared.

The unit of work has a name. A trace is the complete record of one exchange with the model: everything that went in — the system prompt, any retrieved documents, the conversation so far, the user's turn, the model identifier, the sampling settings — together with the text that came back and the metadata describing how generation ended. Traces exist because the input to a language model is not a stable artifact sitting in a repository. It is assembled at request time by your own code out of half a dozen moving parts, none of which appear in the answer, so a sentence like "yes, on the Business plan" is simultaneously the only thing everybody has and the only thing that carries no information about which part put the idea there. In practice you meet traces as rows in a tracing tool such as Langfuse, LangSmith or Braintrust, as OpenTelemetry spans if your team instrumented model calls the way it instruments HTTP calls, or as a JSON document your own middleware writes to object storage. Freezing a trace means copying that record somewhere immutable, with the full rendered request rather than a pointer to it, so that it stays fixed while the system around it keeps moving.

What a request log throws away#

The reason to slow down is not caution for its own sake. You cannot compare runs if the run you care about no longer exists, and after a deploy, an index rebuild or a log expiry, it does not. The first job is not to explain the failure. It is to make sure the failure is still there tomorrow, in a form somebody else can pick up and re-run.

Almost every team discovers, at this exact moment, that their observability was built for a different question. A support tool stores the conversation: user turns, assistant turns, timestamps, a request id. That is the output side. Everything that determined the output was assembled at request time and discarded when the response was returned.

Consider what actually goes into a support request of this kind. A system prompt rendered from a Jinja template whose version changed on the last deploy. Six retrieved passages, selected by an embedding model against an index that is rebuilt every night at 02:00 UTC — retrieval here meaning that a knowledge base is cut into chunks of a few hundred tokens, each stored as a vector of numbers positioning it by meaning, and the chunks whose vectors sit nearest the question's are pasted into the prompt — so the passages retrieved on Saturday may not exist as chunks any more. A conversation history that was compacted at turn nine by a summarisation call whose output was never stored. A user profile block containing the customer's plan tier, which the customer has since upgraded. Sampling parameters that come from a config service and were adjusted for the same deploy. A model identifier that resolves to a snapshot, and snapshots get retired.

Re-running "the same request" against today's system reproduces none of that. If the answer comes out clean, you have learned nothing, because at least six things changed. If it comes out bad, you still do not know which of the six is responsible. This is why the instinct to immediately re-ask the question in a staging chat window is close to useless as evidence, however tempting it is as a first move.

The mechanism under that is worth walking one step at a time, because it is the whole reason freezing is an activity rather than a habit. What a request log typically keeps is a list of names: support-assistant/v14 names a template, claude-opus-5 names a family of weights, kb/plans/business names a document, kb-2026-08-05T02:00Z names an index build. A name is not a value. It is an instruction to go and look something up, and the lookup is deferred until whoever reads the log asks for the text. That deferral is what makes the record small, and it is the same property that makes the record decay.

Follow one name through. The log stores the identifier of a document rather than the document's bytes. The store that maps identifiers to bytes is rebuilt nightly, from the corpus as it stands at rebuild time. Therefore the bytes that identifier resolves to now were assembled after the failure happened, and nothing in the record can tell you whether they match the bytes it resolved to on Saturday. Therefore a replay that resolves the name is assembling a prompt that was never sent to anything. Therefore whatever that replay produces is a fact about a system other than the one that failed. Each name in the record opens that gap independently, and any single one of them is enough on its own — the model alias resolving to newer weights would do it with every document byte identical.

Freezing is the substitution of values for pointers, performed once, while both still exist. Every name in the record is resolved at capture time, and the result is written down beside the name rather than instead of it: the document text next to the document id, the dated snapshot next to the model alias, the rendered system prompt next to the template version. The name stays because it is how you find the live thing again; the value is what keeps the record readable after the live thing has moved. A frozen trace, in one line, is the same record with every lookup already performed.

A second belief is doing quiet damage alongside the belief that a name still resolves to what it resolved to, and it is the more comfortable of the two: that a request log is a record of the request. It is a record of the arguments your application assembled — a list of message objects, a model name, a dictionary of sampling settings — and those arguments sit upstream of the bytes the provider's model actually ran on. Between the two there is a run of transformations, and your logging line is in a position to see none of them.

Naming them is worth the space, because each is a place where the log and the execution part company. The message list is rendered by a chat template into one sequence, with role markers and special tokens wrapped around each turn, and which template is applied is a property of the model rather than of your code. The client library serialises that structure to JSON, escaping some characters and normalising others. The library, or a gateway sitting in front of it, adds material of its own: tool definitions, cache breakpoints, default stop sequences, and on some endpoints a preamble the provider prepends to everything. The text is then tokenised into integer ids, which is the only form the model reads. If the assembled sequence overruns the context window, something truncates it, and what gets dropped is chosen by a policy you did not write.

Each of those is invisible for the same structural reason. A log line executes at the call site, so the only thing it can see is the value of the local variables at the instant of the call. Everything after that instant happens inside a library, inside a gateway or inside somebody else's process, and none of them hand back a copy of what they built. The response returns the output text and a count of input tokens, and a count is not a document. So a request log is accurate about what your code intended and silent about what was executed — which is the subject of the next chapter, where the same messages are pushed through the model's own chat template and the resulting token ids are compared against what the endpoint was actually sent.

The record that makes a conclusion reproducible#

The thing to capture is the request as it left your process, plus enough surrounding state to re-materialise it. Written down as a document, it looks like this.

{
  "trace_id": "e2b1c9f4-support-4471",
  "captured_at": "2026-08-08T09:14:02Z",
  "occurred_at": "2026-08-05T21:37:11Z",
  "service_sha": "a91f0c3",
  "prompt_template_version": "support-assistant/v14",
  "model": {
    "id": "claude-opus-5",
    "resolved_snapshot": "claude-opus-5-20260214",
    "endpoint": "messages"
  },
  "sampling": {
    "temperature": 0.8,
    "top_p": 0.95,
    "max_tokens": 1024,
    "seed": null,
    "stop_sequences": []
  },
  "rendered_request_sha256": "3f7c...9a1e",
  "rendered_request_path": "s3://llm-traces/2026/08/05/e2b1c9f4.json",
  "block_tokens": {
    "system": 912,
    "retrieved[0]": 604, "retrieved[1]": 588, "retrieved[2]": 771,
    "retrieved[3]": 512, "retrieved[4]": 690, "retrieved[5]": 445,
    "history_summary": 302,
    "user_turn": 41,
    "total_input": 4865,
    "output": 118
  },
  "finish_reason": "end_turn",
  "retrieval": {
    "embedding_model": "text-embedding-3-large",
    "index_snapshot": "kb-2026-08-05T02:00Z",
    "doc_ids": ["kb/plans/business", "kb/exports/api", "blog/2025-q4-roadmap",
                "kb/plans/comparison", "kb/settings/overview", "kb/exports/manual"],
    "scores": [0.913, 0.887, 0.874, 0.871, 0.869, 0.862]
  },
  "output_text": "Yes — scheduled exports are available on the Business plan...",
  "tools": []
}

Several of these fields carry more weight than they look like they do, and it is worth being explicit about which decision each one unblocks.

A couple of those rows are better read as chains than as entries, because the middle column compresses several steps into a question. Take block_tokens first. The six retrieved passages in this trace come to 3,610 tokens, which is roughly three quarters of the whole input; the system prompt is under a fifth of it and the user's own question is a rounding error. Suppose the record held the retrieval result and not the per-block counts. A budget that dropped the last passage on the way into the prompt would then leave no mark anywhere: the retrieval section still lists six documents, the request carried five, and the two records agree with each other. The step that breaks is the coverage check — reading the retrieved passages to see whether any of them states the fact the answer needed, which is the third chapter of this topic — because that check would be run against a list of what retrieval returned rather than against what the prompt contained.

index_snapshot and doc_ids break later and at a different step. Ablation, in the fifth chapter of this topic, replaces one retrieved block with a neutral passage of the same length and re-runs, so that content varies while position and token budget hold still. Constructing that substitution requires the original block's text. Without a snapshot identifier there is exactly one place to get it, which is today's index, and today's index is the thing most likely to have changed — so the ablation would be varying two things at once and reporting one. The pattern generalises across the table: a missing field rarely announces itself at capture time, and shows up instead as an experiment three steps later that cannot be built, or worse, can be built and quietly means something else.

Two entries are empty in this example and would not be in an agent system. The tools array should hold the tool definitions verbatim, including the JSON schemas — a tool definition being the JSON description of a function the model may ask your service to run on its behalf, carrying a name, a sentence about when to use it, and a schema for its parameters — because a schema description is prompt text that the model reads and that somebody edits on a different release cadence from the system prompt. Where the turn involved tool calls, the tool results belong in the record too, in the exact serialisation that went back into the conversation, since a 4,000-token API response pasted into the context is usually the largest single block in the request and the one nobody has ever read. The seed being null is also a finding rather than a detail: this service never set one, which means nobody can pin a replay until the calling code is changed — and a pinned replay is the first thing the fourth chapter of this topic does, to establish that your own stack produces a sane answer from the recorded bytes before any rate gets estimated. That is a small piece of work you want to discover now rather than four experiments from now.

The finish_reason line deserves a note only because of how often it short-circuits the whole investigation. Topic 1.8 covered why a length-capped response reads as fluent and complete; the operational consequence here is that checking one field takes two seconds and occasionally ends the debugging session before it starts. In this trace it says end_turn and the output is 118 tokens against a 1024 cap, a little over a tenth of the budget, so truncation is not the story and the investigation continues.

Why the comparison trace has to be the same shape#

A frozen bad trace on its own supports very little. What it supports is a diff, and a diff needs a second document. The second document is a known-good trace: a request from the same route, the same template version, and roughly the same structure, whose answer was correct.

A diff is worth this much, and it is worth being precise about the amount. If the cause lies in the request at all, it lies somewhere in the list of differences, because the two requests agree everywhere else and identical inputs cannot account for different outputs. That guarantee holds unconditionally, and it is also weak: it names a set rather than a member of it. Promoting the set to a member costs one run per candidate — hold the bad trace fixed, reverse one difference, send it, and see whether the failure survives. The length of the list is therefore a budget, and choosing a comparison trace well is the craft of keeping that budget small enough to spend.

"Roughly the same structure" is doing real work in the description of that second document. If the bad trace carries six retrieved blocks totalling 3,610 tokens and the good trace is a one-line FAQ answer with two blocks and 700 tokens, the diff between them will be enormous and almost entirely irrelevant. Every difference is a candidate explanation, so a diff with four hundred differences has told you nothing. The good trace should be chosen so that the list of differences is short enough to read: same template version, same tool set, similar block count, similar total input tokens, ideally the same customer plan tier and the same broad question type.

What makes a mismatched shape damaging is confounding rather than untidiness, and the mechanism is worth stating exactly. Block count is not one property. Changing it also changes the position of every block below it, the total input length, how much of the window is left, which blocks survive a truncation policy, and how far the user's question sits from the instructions. Those move as a bundle. A run that reverses one of them reverses the rest along with it, so no single run isolates any single member, and a difference that cannot be isolated cannot be attributed to anything. The bad trace's retrieved section runs to 3,610 tokens against the good one's 700, more than five times as much, and every one of the hundreds of differences that follows is entangled with that one fact. The list comes out long and each entry on it comes out untestable, which are two defects arriving together and for the same reason.

In practice this means the good trace is best pulled from the same day and the same route, filtered on total input tokens within perhaps 20% of the bad one. When the diff comes back with eleven differences instead of four hundred, each of those eleven is a hypothesis you can test in one run, which is the entire point of the exercise.

# both traces normalised to sorted-key JSON first, so the diff is semantic
jq -S . bad-e2b1c9f4.json  > /tmp/bad.json
jq -S . good-77a30f21.json > /tmp/good.json
diff -u /tmp/good.json /tmp/bad.json | grep -E '^[+-]' | head -40

What that short diff usually surfaces is unglamorous and worth expecting. One trace carries a retrieved block from a blog post and the other does not. One has a compacted history summary and the other has the raw turns because it was shorter. One was served by snapshot 20260214 and the other by 20260119 because the deploy rolled through mid-week. None of these is yet an explanation, and the discipline is to write them down as a numbered list of candidates rather than to pick the one that looks guilty. The list is what the rest of this topic works through, one candidate at a time: a snapshot difference is settled by a pinned replay at low temperature, a retrieved block by swapping in a neutral passage of the same length, an instruction by halving the system prompt and re-running each half.

Storage cost and rotting pointers#

Capturing everything is the obvious response to all of this — log the full rendered request on every call — and for a while that is the right answer. Get the arithmetic in front of you before you commit to it. A 5,000-token prompt is roughly 20 KB of text; at 40,000 requests per week that is about 800 MB per week, or 40 GB per year, which is unremarkable. A 30,000-token agent prompt with tool schemas and long tool results is closer to 120 KB, and at the same volume that is 4.8 GB per week and 250 GB per year, per environment, before you count the multi-step traces where a single user turn produces nine model calls. At that size the storage bill stops being a rounding error and the retention policy starts getting shortened by someone who is not thinking about debugging.

Two things in those figures repay a second look. Both lines carry the same cost per token — 20 KB for a 5,000-token prompt and 120 KB for a 30,000-token one both come to about four bytes of stored text per token — so the second figure follows from the first by multiplication rather than from a separate measurement. And the two yearly totals differ by more than six times on identical request volume, because prompt size is the only term that moved between them. The systems whose traces are hardest to rebuild from ingredients are agent systems with long tool results and many calls per turn, and those are precisely the systems whose full bytes cost the most to keep, which is why this trade-off gets decided under pressure rather than at leisure.

The cheaper design is to store only the ingredients — template version, retrieved document ids, parameters, seed — and re-render on demand. It works exactly as well as your rendering is pure and versioned, and rendering usually is not. Pure here means that the same ingredients produce the same bytes on every occasion, with nothing consulted that is not itself in the ingredient list: no clock, no mutable row, no second model call, no index lookup.

Real renderers break that condition in thoroughly ordinary ways, and each of them breaks it somewhere different. Prompts contain the current date. User profile blocks read a mutable row. Conversation summaries were produced by a model call that was itself sampled. The index that maps a document id to text is rebuilt nightly, so a document id resolves to different bytes on Wednesday than it did on Saturday — when the underlying document or the chunking changed, and not otherwise, which is the uncomfortable part, because the record gives you no way to tell those two cases apart. Each of these makes a stored pointer decay into an approximation, and you find out which ones decayed only when the re-render disagrees with the recorded output hash.

The hash is what makes the decay detectable at all. A re-render that silently disagrees with the original looks exactly like one that agrees, unless something computed from the original bytes was kept to check it against, and a hash is exactly that: a short witness of a long document, cheap enough to store on every request when the document itself is not. What it establishes is bounded, and the bound is worth holding onto. Matching hashes mean the bytes you re-rendered are the bytes your process sent; they say nothing about what a client library, a gateway or a provider did with those bytes afterwards, which is a separate question and the one the next chapter opens on.

The compromise most teams land on is a hybrid: ingredients plus a rendered_request_sha256 on every request, full rendered bytes on a sampled fraction, and full bytes unconditionally on any request that a human escalated, that hit a guardrail, or that a downstream check flagged. The stored hash is what makes the hybrid honest, because it lets a re-render prove it reproduced the original rather than assert it. When the hashes differ, the pointer rotted, and you have learned something about your own pipeline that was worth knowing anyway.

Freezing is not reproduction#

Preserving a trace is not the same activity as error analysis over a labelled sample — the subject of topic 3.6, where many outputs are read, sorted into named failure modes, and counted, so that the modes worth anyone's attention are the frequent ones. Here there is one output, the population is unknown, and the question is causal rather than distributional. The two feed each other — a taxonomy of failure modes tells you which single traces are worth freezing — but the methods do not transfer, and running frequency statistics on n=1 produces a number with no meaning.

Freezing is also not reproduction. You now have a record of what happened; you do not yet have any evidence that your own stack, today, will produce the same output from the same bytes. That gap is larger than it sounds and it is the subject of the fourth chapter of this topic, where the recorded bytes are replayed at pinned low temperature and then re-run under production sampling to estimate how often the bad output appears. Until a replay produces the failure at least sometimes, every experiment you run is measuring your harness rather than the model.

The boundary is worth stating in both directions, since a frozen trace is easy to over-trust once the work of making it has been done. It licenses three things. Any claim about this failure can now be phrased as a run against fixed bytes, which means it can come back false. The input side of the argument is settled, so two engineers can stop disagreeing about what the model was given. And a diff against the known-good sibling converts the space of possible causes into a list somebody else can pick up and work through. It licenses nothing about frequency, because there is one case and no denominator — a rate needs repeated sampling under production settings, and a decision to change the system needs the class rate of the last chapter of this topic, where siblings of the failing input are constructed and the share of them that fails is counted. It does not license the belief that your stack reproduces the output today, and it does not license the belief that the bytes you stored are the bytes the model consumed.

And a frozen record is not itself an explanation. It is the substrate on which explanations become falsifiable: any claim about this failure now has to survive a re-run against the stored bytes, and any claim that cannot be stated as a re-run is an opinion.

That constraint is the whole value of the exercise, and it is why an afternoon spent on capture buys more than the instruction it displaces: a frozen trace and a known-good sibling cost nothing to keep and turn every subsequent claim into something with a pass or a fail attached to it.

The first claim to put through that filter is the most basic one available, and the one people skip because it seems beneath checking. Everything above assumes that the bytes you recorded are the bytes the model consumed. Between your list of messages and the model's input there is a rendering step you may not own — the chat template that folds a messages array into one sequence of token ids — and when that step is wrong, every other conclusion drawn from the trace is drawn from the wrong document.