The rendered prompt is the only proof of what the model saw

A chunk is a passage of a few hundred words cut out of a document and stored under its own identifier. A retrieval system answers a question by searching for the chunks most likely to contain the answer, pasting them into a prompt beside the question, and asking a model to answer from them. When such a system gives a wrong answer, the complaint almost never arrives while the query is still running. It arrives days later, from somebody who was not there, about a query nobody kept anything from.

The counters from the previous chapter — how many chunks each document produced — make ingest observable in aggregate and say nothing about one query. A wrong answer is always about one query, and the evidence that would explain it — which chunks came back, in what order, and what string was assembled from them — lives for a few hundred milliseconds inside a process and is then gone unless something writes it down at the moment it exists.

Re-running the question is the obvious move, and it settles nothing. Retrieval is a function of a set of documents and an index build, and both change underneath you: documents are added overnight, an index is rebuilt on a schedule, and a question asked today can reach a chunk that did not exist when the answer being objected to was produced. A re-run tells you what happens now, and what was asked was what happened then. Of everything that has to be written down at the time, one field cannot be recovered afterwards under any circumstances, and that field is the exact string handed to the model.

The string is the only thing the model receives#

A generation call is a function from one string to another. The model has no handle on your retriever, no view of your index, no access to your template file and no way to read your logs. Everything it is able to condition on arrived inside that string, and everything it produced is a function of it. That is not a simplification for teaching purposes; it is the interface.

Two consequences follow immediately. Any sentence of the form the model was told X is a claim about the contents of one string, so it is settled by reading that string and by nothing else. And any artifact that is not that string is a hypothesis about it — a good hypothesis, often a nearly correct one, but a hypothesis, and the distance between it and the string is where this chapter's failures live.

A retrieval log is the clearest case. It records the identifiers a search returned, in rank order, with scores. Between that list and the string sits a render step, and the render step is not the identity function. It truncates, dropping chunks once an assembled context passes a token budget. It deduplicates, collapsing two near-identical passages into one. It reorders, because where a chunk sits in the block changes how reliably it is used. And it substitutes into a template, which can produce an empty field without complaint when a variable it expects is absent.

Every one of those four operations changes the string while leaving the retrieval log byte-identical. That is the property that matters. The log was written before the render step ran, from data the render step had not yet touched, so it reads exactly the same whether a chunk reached the model or was discarded on the way. A test that returns the same value in both of the cases it is meant to tell apart is not a weak test. It is not a test.

That is the belief worth naming, because it is the one a competent engineer arrives at on their own: the correct chunk was retrieved at rank 1, so the model had it and ignored it. Rank 1 is the strongest possible retrieval result, and it is tempting to treat the strongest available evidence as sufficient evidence. It is not sufficient here, because the strength of the retrieval evidence and the question of whether the chunk was rendered are unrelated quantities — the first can be at its maximum while the second is zero.

It helps to write the requirement as a product. A correct answer needs four things to hold at once: the fact is in the index, retrieval returns the chunk holding it, rendering puts that chunk into the string, and the model uses it. Treat each as a factor that is one when it holds and zero when it does not, and the answer is right only when the product is one. Any single zero decides the outcome by itself, and the other three factors become irrelevant to the result — a chunk indexed perfectly, retrieved at rank 1 and then truncated away contributes exactly as much as a chunk that was never indexed at all.

The awkward part is which factors get measured. Retrieval logs measure the second. Evaluation sets measure the product, by looking only at the final answer. In a default pipeline nothing at all measures the third, and a factor nobody measures is a factor that can sit at zero for weeks without anybody noticing, because the two instruments either side of it both keep reporting normal numbers.

As an analogy — and it understates the problem rather than overstating it — the retrieval log is the order and the rendered prompt is the delivery. It breaks in one specific place, and the break is the useful part: an order and a delivery note are written by two parties, so a discrepancy between them is visible to anybody holding both. Here both are written by the same process, and that process writes the second one only if somebody has asked it to. A missing delivery note in commerce is an anomaly somebody chases; a missing prompt string is the default state of every pipeline nobody has changed on purpose.

One record, written at the boundaries you already have#

The pipeline already produces every value worth keeping; it just discards them. Assigning a query id at the top and appending to a single dictionary as the stages complete costs almost nothing and produces one JSON object per query. That object is the trace record: every input and output of one question's journey through the pipeline, held in one place and keyed by an identifier, so that reading it is a lookup rather than a reconstruction.

The arrows into Trace are the whole of the new code, and every value on them already exists at the moment it is drawn: query_vector_dim is the length of the list the embeddings API just returned, ms_search is a subtraction between two clock reads the search already needed, chunks_dropped is a list the render function can build while it decides what to drop anyway. None of these arrows adds a network call, a lock or a query. They add an assignment.

The record for a query about log retention, had it been written when that query ran, would look like this. It is worth reading field by field, because each one is there to eliminate a specific hypothesis rather than to be thorough.

{
  "qid": "01HQ7X3M4K",
  "ts": "2026-03-10T14:22:07Z",
  "question": "how long do we retain logs for enterprise accounts?",
  "index_version": "build-2026-03-09T02:11Z-9f3ac1e",
  "embed_model": "text-embedding-3-small",
  "gen_model": "claude-sonnet-4-5-20250929",
  "temperature": 0,
  "top_k": 5,
  "retrieved": [
    {"rank": 1, "chunk_id": "policies/retention-2022.md#0",   "score": 0.641},
    {"rank": 2, "chunk_id": "handbook/logging.md#800",         "score": 0.618},
    {"rank": 3, "chunk_id": "policies/retention-current.md#400","score": 0.607},
    {"rank": 4, "chunk_id": "faq/support.md#1200",             "score": 0.588},
    {"rank": 5, "chunk_id": "handbook/logging.md#1600",        "score": 0.581}
  ],
  "prompt": "Answer the question using only the context blocks below.\n\n[policies/retention-2022.md#0]\nLog retention for Enterprise plans is 90 days from...",
  "prompt_tokens": 1502,
  "chunks_in_prompt": 4,
  "chunks_dropped": ["handbook/logging.md#1600"],
  "output": "Enterprise accounts retain logs for 90 days.",
  "finish_reason": "end_turn",
  "usage": {"input_tokens": 1502, "output_tokens": 11},
  "ms": {"embed": 96, "search": 4, "render": 1, "generate": 1380}
}

Two things in that record answer the question raised by an output field somebody has since objected to, which is whether the assistant invented 90 days or read it somewhere. The model did not invent it; it read it from policies/retention-2022.md, which was retrieved at rank 1 with the highest score in the set, and which is a superseded policy that nobody had deleted. And prompt_tokens is 1,502 where a top-5 render of 400-token chunks should be near 2,100, which is the signature of a chunk that never made it into the string. The chunks_dropped field names it: the budget clamp in the render stage stopped adding chunks once the assembled context passed a limit, and the fifth chunk was silently discarded.

That second reading rests on a comparison against a prediction rather than against a measurement. Five chunks at four hundred tokens each, plus the fixed cost of the template and the question, is where 2,100 comes from — nothing measured it, it was multiplied out. What was measured is 1,502, and neither number means anything alone. A pipeline that records prompt_tokens and never states what it expected has kept the number and thrown away the comparison.

Three fields close that gap exactly, and the closure is worth stating as an identity. top_k is 5 and retrieved holds five entries; chunks_in_prompt is 4; chunks_dropped names one chunk. Four rendered plus one dropped is five retrieved, with no residual and no unexplained remainder. What the identity licenses is a strong claim: every chunk the retriever returned is accounted for by name as either rendered or dropped. It closes in one direction only, because rendering can remove a chunk and can never add one, so chunks_in_prompt can never exceed the number retrieved and any shortfall must appear in chunks_dropped. A record where those two do not add up would be reporting that a chunk went missing for a reason the render stage does not know about, which is a different bug and a worse one.

A second identity sits in the same record, between two numbers produced by two different parties. prompt_tokens and usage.input_tokens both read 1,502. The first was counted locally, by the render stage running a tokeniser over the string it had just built; the second was reported by the API, counted on the string the API received. Two independent instruments landing on the same number licenses one specific claim: nothing sat between the render call and the model that changed the string. Where the two disagree, the difference is whatever the client library added on the way past — a system message, a chat wrapper, a tool schema — and the size of the gap narrows down which.

One deduction falls out of chunks_dropped that changes the diagnosis, and it comes from the field pair rather than from any single field. Only handbook/logging.md#1600 was dropped, and that chunk was at rank 5, so ranks one through four were all rendered. Rank 3 is policies/retention-current.md#400. The model was handed the superseded policy in the first block and the current policy three blocks below it, and answered from the first. Deleting the superseded page fixes this query; it says nothing about why the model preferred one of two conflicting blocks, and it would not have fixed the next query where the same shape recurs with a page nobody has spotted yet. The record is what makes those two separable, and a complaint on its own cannot separate them.

finish_reason is doing a job the answer text cannot do for itself. A sentence that ends in a full stop looks the same whether the model finished or was cut off at a token limit, because the string carries no evidence either way — the truncated version of a short factual answer is often still a grammatical sentence. end_turn is the only field in the record that tells the two apart, and it is the reason a short answer here can be read as complete rather than as a symptom.

The clamp that dropped the fifth chunk is worth showing, because the version that drops a chunk without saying so is the version most people write first.

MAX_CONTEXT_TOKENS = 1400

def render(hits, chunks_by_id, question, template):
    blocks, used, dropped = [], 0, []
    for h in hits:
        text = chunks_by_id[h["chunk_id"]]
        n = len(ENC.encode(text))
        if used + n > MAX_CONTEXT_TOKENS:
            dropped.append(h["chunk_id"])      # was: `break`, silently
            continue
        blocks.append(f'[{h["chunk_id"]}]\n{text}')
        used += n
    prompt = template.format(context="\n\n".join(blocks), question=question)
    return prompt, len(ENC.encode(prompt)), dropped

Reading that loop settles something the record appears to contradict. prompt_tokens is 1,502 and MAX_CONTEXT_TOKENS is 1,400, and a token count above the clamp looks at first like a clamp that is not working. It is working, and the two figures measure different quantities. used accumulates len(ENC.encode(text)) over chunk texts only, and it is used that the clamp bounds. The number returned as prompt_tokens is len(ENC.encode(prompt)) over the whole rendered string, which is the chunk texts plus the template's instruction line plus the delimiters around each block plus the question. The excess over the clamp is therefore the template, the delimiters and the question, and that residual is worth naming rather than rounding away: it is the part of your context window that grows whenever somebody edits the template, and it appears in no per-chunk accounting anywhere in the pipeline.

The comment on the dropped.append line records that the statement there used to be break, and swapping break for continue did two separable things at once. It started recording the drop, which is the visible fix. It also changed which chunks survive: break stops at the first chunk that would overflow and keeps a prefix of the ranking, while continue skips that chunk and carries on, so a lower-ranked short chunk can now be rendered while a higher-ranked long one was not. After the change the rendered set is no longer guaranteed to be a prefix of the retrieved list. Shipping both changes in one commit means any movement in answer quality afterwards cannot be attributed to either of them, which is a small instance of the problem the whole record exists to prevent.

The original had break and no dropped list, so the pipeline had been running for three weeks quietly cutting top-5 to top-4 on every query whose chunks ran long. Nothing about the answers looked wrong, because a top-4 answer is usually a top-5 answer, and the counter that would have exposed it — the token count against its expected value — was not being compared to anything.

What each artifact rules out#

The useful test for whether a field belongs in the record is not whether it is interesting but which explanation it eliminates. Below is the minimum set for a pipeline going in front of internal users, with the hypothesis each one kills.

Read the cost column as gaps rather than as levels. Six of the seven rows are priced in tens or hundreds of bytes; one is priced in kilobytes, and it is larger than the other six put together by more than an order of magnitude. Add the column up and it comes to about the ten-kilobyte record that the retention arithmetic below assumes, which is two independent routes to the same figure — one from summing what each field costs, one from measuring what a day of records weighs. Almost all of that ten kilobytes is a single field. Every argument anybody will have about the cost of tracing is therefore an argument about the prompt string, and the six cheap fields should never be allowed into the negotiation at all.

The prompt string is also the only one of these that cannot be reconstructed later under any circumstances, and that is what decides build order. Retrieval can be re-run if you kept index_version and the build it names, since a matrix multiply at temperature nothing is deterministic. Scores can be recomputed. The output can be regenerated at temperature 0 and will usually match. The prompt, though, is a function of the retrieved chunks and the template and the budget clamp and the chunk text as it stood at that build.

That conjunction is why reconstruction fails, and the word to hold on to is and: reconstructing it requires that all four be pinned, not three, because a single unpinned input produces a different string and nothing in the output says which input moved. Of the four, templates are edited more casually than any of the others and are the only one that usually carries no version marker at all — a prompt tweak deployed on Wednesday makes every Tuesday prompt unreproducible with no trace that anything changed. Persisting the string is nine lines of code and removes the entire dependency. Everything else in the table can wait a day; that one cannot wait at all.

Why scattered debug lines and 1% sampling both fail#

The instinct is to turn on debug logging for the whole pipeline — question, chunks, prompt, answer, timings — and grep when somebody complains. It fails on three distinct counts, and each one is fatal on its own.

Unstructured lines cannot be joined. The one operation you need is show me every stage of the query with id 01HQ7X3M4K, and with log lines you get five records emitted at five moments, interleaved with lines from concurrent requests, correlated only by timestamp and whatever the logger happened to prefix. Under any concurrency the interleaving is real rather than theoretical, and the prompt string in particular spans multiple lines, so a multi-line payload in a line-oriented stream is already corrupted by the time it reaches storage.

Debug is the level that gets turned off. It is turned off precisely when volume is high enough for the logging bill to be noticed, which is the same condition under which incidents happen, so the reliable outcome is that the query you need is the one that ran while debug was off. That is not a hypothetical failure mode; it is the default trajectory of every debug-level integration that survives contact with a finance review.

Grep answers the wrong question. What you will be asked is whether the correct chunk was retrieved for a query last Tuesday, and answering it means filtering by date, then by question similarity, then joining a chunk id to a document — three operations on a structured field, none of which a text search over a log stream performs. A JSONL file with one object per query answers it with a jq filter, and the same file loads into DuckDB — an analytical database that runs inside your own process and reads JSON and Parquet files where they already sit — or into a warehouse table, without a parser.

Sampling at 1% breaks on arithmetic rather than principle. Complaints are not a random sample of traffic — they arrive attached to specific queries, and the probability that a named query is in a 1% sample is 1%, so ninety-nine times in a hundred the answer to "what happened on Tuesday" is that nothing was kept. Sampling is the right tool when you want an unbiased estimate of a distribution, such as latency percentiles, and the wrong tool when you need a particular record on demand.

The general form is worth holding, because it applies to every sampled instrument. A sample rate answers what does the distribution look like, since any random subset of a large population answers that within a known error. When the question names a member of the population, the evidence either exists or it does not, and the sample rate is exactly the probability that it does. Diagnosis is always the second kind of question.

The costs sampling is defending against are worth pricing, because they are smaller than they feel. At 3,000 queries a day, a 10 KB record is 30 MB a day and 900 MB a month, which compresses to roughly 220 MB of gzipped JSONL and costs about five cents a month in object storage at $0.023 per GB-month. Ninety days of full retention is under 3 GB. On latency, serialising and appending the record takes about 0.4 ms against a generate call of 1,380 ms, and if that still bothers you, hand the dictionary to a background writer and it becomes 0.02 ms on the request path. There is no version of these numbers in which sampling is a cost decision.

The storage figure and the latency figure are safer read as ratios than as amounts. Whether the storage line is taken as one month's compressed increment or as the whole ninety-day store at steady state, it lands in cents either way, against a query whose model call costs more than that on its own; and the write is under a thousandth of the request before you move it off the request path at all. Neither is close enough to any plausible threshold for a more careful calculation to change the decision.

One detail decides whether any of this is usable, and it is not in the record at all: the qid has to travel to wherever the complaint originates. Returning it with the answer and attaching it to the feedback control in the UI means a thumbs-down arrives carrying the identifier of the exact trace, which turns a later conversation from a reconstruction into a lookup. Without that, finding the record means searching by question text, which fails on paraphrase, on a customer relaying the question secondhand, and on any query asked more than once. Rendering the id in small grey text at the bottom of the answer is the crude version and it works, because the person reporting the problem will paste it.

The storage shape follows from what you will ask of it. One JSON object per line, one file per day, is enough for months of this, and it is directly queryable without an ingest step — DuckDB reads queries-*.jsonl in place, so which queries retrieved a chunk from a superseded policy last week is a single statement over a nested column rather than a script. The alternative of a relational table with the retrieved chunks in a child table is better once you are joining traces against labelled outcomes at volume, and it is a migration you can do later precisely because the JSONL is already structured. What does not work is the middle option of pushing records into a log aggregator as strings, which reintroduces the parsing problem the format exists to avoid.

What sampling is sometimes standing in for is a data-retention concern, and that one is legitimate. The record contains the user's question verbatim and chunk text that may include customer names, so it inherits whatever classification your support data has, and the honest answer is a retention window measured in weeks with deletion by qid, plus redaction of the question field for regulated corpora — not throwing away 99% of the evidence and calling it a privacy control.

Diagnosis is not evaluation, and a trace is not a manifest#

A directory of traces tells you exactly what happened on each query and nothing about whether the answers were right, because no field in the record holds a judgement. Reading traces is how you diagnose a failure somebody else found; it is not how you find failures. Evaluation runs a fixed set of questions and scores the outputs; diagnosis takes one output somebody has already objected to and walks backwards, and a record ideal for the second is inert for the first.

It is also not a substitute for the ingest manifest, meaning the record of which documents were processed into which chunks at each build. The trace explains what happened to a query given an index; it cannot tell you that a document was never in that index, because a chunk that does not exist cannot appear in a retrieval block and its absence looks identical to it being outranked. The two artifacts answer adjacent questions and neither covers the other, which is why the diagnosis in the next chapter starts by checking the manifest and only then reads the trace.

That identity of appearance is the retrieval log's defect arriving one stage upstream, and the two are cured differently. Rendering is fixed by writing down what the render stage did, because the render stage knows. Absence from an index cannot be, because nothing in the query path ever holds the fact that a document exists elsewhere and was never processed — that has to be written on the other side, at build time.

It is also not distributed tracing, though it looks adjacent. An OpenTelemetry span for the retrieve stage will give you duration, status and a handful of attributes, and span attributes are capped at sizes that make an 8 KB prompt string an abuse of the format. Spans are the right home for the ms block and the wrong home for the payload, so the usual arrangement is spans for timing and a separate durable record for content, joined by qid.

One limit is worth stating even where the record is complete. It says what was sent and what came back, and it has no field for why the model used one block rather than another; there is no such field available at any price. What the record buys is the narrowing — it establishes which factors in the product were one, so the remaining question is confined to a single stage instead of spread across four. The word usually in "the output can be regenerated at temperature 0 and will usually match" is carrying weight of the same kind: a re-run that differs is not proof that the prompt changed, which is precisely why the stored string rather than the stored output is the artifact that settles the argument.

With one record per query and a corpus that reports its own gaps, a wrong answer stops being a story and becomes a lookup. The retrieval block, the token count and the dropped list are three views of one product, and reading them together turns the model got it wrong into a statement about which factor was zero.

The next chapter takes two of those lookups — one where the correct chunk never entered the top 5, one where it sat at rank 1 and appears verbatim in the prompt — and shows that the records make them different failures with different owners, which matters because the fix that helps one of them measurably harms the other.