Reading a trace end to end

Error analysis is the practice of taking a sample of real failures, reading each one from beginning to end, and writing down what produced it, so that failures can be grouped by cause and counted rather than merely described. It exists because every other quality signal an AI system emits is an aggregate — a pass rate, a mean score, a line on a dashboard — and an aggregate is a compression that discards precisely the part you need in order to act. A run reporting 87 failures out of 300 cases tells you the system is wrong about three times in ten; it cannot tell you that a fifth of those failures come from one date being resolved to the wrong year inside a database query. The procedure is borrowed rather than invented, since it is open coding from qualitative social research applied to machine transcripts instead of to interviews, and this topic works through it in order: assembling a pool of failures, choosing which to read, naming the modes they fall into, turning counts into rates over real traffic, and deciding which mode is worth an engineer's week.

You meet it at particular moments rather than continuously. A release drops the pass rate and somebody has to say what broke. A quarter is being planned and somebody has to say which quality work is worth doing. A customer-visible incident lands and somebody has to say whether it was one conversation or the visible edge of a recurring mode. Or the dashboard has been flat for two quarters and nobody can say whether that is good news. In each of those the useful input is the same, and it is the conversations sitting underneath the score.

This chapter is about the reading itself, which is the step everything after it depends on and the one that a results grid quietly prevents. The unit you work in is the trace: the complete record of everything the application did for a single conversation, kept in the order it happened. A trace holds every user and assistant turn; every tool call, meaning a request the model emits for the application to run a named function such as an order lookup and hand the result back; the raw response that came back, including errors and latencies; whatever retrieved text was pasted into the prompt; and the versions of the prompt and model that produced the run. A results grid shows three fields out of all of that, and the three it usually shows are the input, the last thing the assistant said, and whether a scorer approved of it.

One case, in two views#

Here is a single failing case as a grid renders it. This is the entire record it offers:

input:  "did my refund for the boots go through? ordered them in march"
output: "I don't see any boot orders on your account from March. Refunds are
         normally processed within 5 business days of the item being scanned…"
score:  0

Read that and you will write down "wrong answer" or "claims order doesn't exist", and both are accurate descriptions of what the user saw. Here is the same case as the trace, which is what the application actually did:

{
  "conversation_id": "c_8842",
  "prompt_version": "sys-2026-07-02",
  "turns": [
    { "role": "user",
      "content": "did my refund for the boots go through? ordered them in march" },
    { "role": "assistant",
      "tool_calls": [{ "name": "search_orders", "arguments": {
        "customer_id": "u_31877", "query": "boots",
        "date_from": "2026-03-01", "date_to": "2026-03-31" }}] },
    { "role": "tool", "name": "search_orders",
      "content": { "results": [], "latency_ms": 412 } },
    { "role": "assistant",
      "content": "I don't see any boot orders on your account from March…" },
    { "role": "user",
      "content": "i definitely ordered them, order number 44-119-8823" },
    { "role": "assistant",
      "tool_calls": [{ "name": "get_order",
        "arguments": { "order_id": "44-119-8823" } }] },
    { "role": "tool", "name": "get_order",
      "content": { "error": "not_found",
        "detail": "order_id must be normalised (no hyphens)" } },
    { "role": "assistant",
      "content": "That order number doesn't appear in our system. Refunds are…" }
  ]
}

Reading a record like that is not scanning it for something that looks wrong. It is applying one two-part test to each step in the order the steps happened: whether everything the step was handed was correct, and whether what it emitted was correct given what it was handed. The first step answering yes and then no is where the system diverged, because it is the first place something went in sound and came out unsound. A step answering no to the first question is downstream of a divergence that happened earlier, whatever it emitted, and its own correctness says nothing about its own quality.

Applying that test turn by turn finds two distinct defects here, and neither of them is knowledge of the return policy. The first is that search_orders was called with a date window of March 2026, because the current year was 2026 and the user said "march"; the order was placed in March 2025, so the tool correctly returned an empty result for a query that was wrong. The second is that get_order rejected the hyphenated order number with a validation error, and the model read {"error": "not_found"} as evidence that no such order exists rather than as a malformed request. Both failures are in the seam between the model and the tools, and no prompt rewrite reaches either of them.

Walking the turns is how those two get located. Turn one is the user's message, where the first question is trivially yes, since what the user said is what the user said. Turn two is the model's call to search_orders, and the arguments are the whole of what that step emitted: a customer identifier, the query boots, and a window running from the first of March 2026 to the last. The user's message contains the word "march" and no year at all, so the step emitted a determination present in nothing it had been handed. Yes, then no: the divergence is here.

Turns three and four both fail the first question and pass the second, which is what downstream looks like. The tool at turn three was handed a window for the wrong year, and an empty result list is the correct reply to a search for boot orders in a month that contains none. Turn four was handed that empty result, and "I don't see any boot orders" is an accurate report of it. Neither turn is a defect, and neither would move under any amount of work on the model, the prompt or the decoding settings.

The second divergence is in the second half of the conversation, and not where the surface suggests. Turn six sends get_order the identifier the user typed, hyphens included, and turn seven is the tool rejecting it under the name not_found while explaining in its detail field that the identifier needed normalising. Two conditions calling for different handling — a record that does not exist, and a request that could not be parsed — were collapsed into one token, inside the tool rather than inside the model. Turn eight reads that token as an absence and says so, which is correct given what it was handed: the model is the last step in the chain and has nothing to consult about how the token was produced. Of the model's own turns, every one after turn two is right given its input.

The trace also tells you something the grid actively concealed: the failure happened at turn two, and turns three through eight were the model behaving reasonably given a false premise it had been handed. The tool's naming defect at turn seven sits upstream of the scored message in the same way, in a component the grid has no column for. Count this case as "hallucinated an answer" and you will look for the fix in decoding parameters or in an instruction about hedging, which is a part of the system that was working.

What made that reading possible is a property of the record rather than any skill in the reader. Deciding whether a step caused a failure needs three things present together: what the step emitted, what the step was handed, and a guarantee that nothing between the two is missing from the record. A trace has all three by construction, because every row states both what a component received and what it produced, the rows sit in the order they happened, and there is no gap in which an unrecorded step could have intervened. That is what turns "what went wrong" into a list of yes-or-no questions whose answers are inside the record, rather than into a matter of opinion.

A grid holds the endpoints of that same process and nothing in between. Its three fields are where the conversation started, where it ended, and a verdict on the ending, and every hypothesis about the middle is consistent with a record of endpoints alone. The boots case would look identical in the grid if the model had never called a tool, if the tool had timed out, if retrieval had returned nothing, or if the prompt had genuinely lacked the policy. A reader holding a record that cannot discriminate between those does not stop reading and declare the question unanswerable; they fill the middle in from whichever component they already suspect, and the sentence that comes out is a statement about their priors in the costume of a finding.

The belief underneath that is worth stating plainly, because careful people hold it: that the first 200 characters of a failing output tell you what failed. It gets one thing right, and what it gets right is why it survives contact with evidence. Every failure does appear in the output, since a conversation whose final message was fine is not a case anybody flagged, so the output is a complete census of symptoms. In a system with one moving part — a prompt, a model, a reply — it was also a complete census of causes, and reading it really did discriminate between them, because there was only one component in which a cause could live. The habit formed there, and it was carried into systems with a retrieval step, a tool layer and a context assembler, each of which can fail while the model behaves impeccably.

The structural reason the cause is not in the opening span is that the output is written by the last step in the chain, conditioned on the state that reached it. Whatever an earlier step contributed arrives at the final step only as an effect on that state, never as a description of itself. An empty result set reaches the model as an empty result set, not as an empty result set caused by a window built for the wrong year; the provenance is absent from the model's input, so it cannot be present in the model's output. The symptom is rendered at one end of the record while its cause sits at another, and the field a grid keeps is the end holding the symptom.

Two consequences follow. Different causes converge on similar text, because the sentence reporting an absence is written by the same component doing the same job whether the absence is real or manufactured three turns upstream. And the text is fluent either way, since fluency is conditioned on the words so far rather than on whether the premise behind them is true. Truncation at 200 characters adds a smaller problem on top of a large one: a usable hint, such as a hedge, appears wherever the model happened to put it, and the prefix is the length that fits a column.

Symptom names and cause names#

The distinction that matters is between naming a failure by what the user experienced and naming it by what produced it. Symptom names cluster things that have nothing in common. Three failures that all look like "wrong answer" in a truncated output can come from a retrieval step that returned zero chunks — retrieval being the lookup that pulls passages, or chunks, out of a document store so they can be pasted into the prompt — from a tool contract mismatch, or from a context window that dropped the first user turn, and those three take different fixes owned by different people.

Every row on the right is invisible in a grid that shows the final message, and every row on the left is what you will write in your notes if the grid is what you read. The reading that recovers the right column is not clever; it is just slow. Open the whole trace, start at turn one, and go forward until you reach the point where the conversation first departed from what it should have done. That point is often several turns before the output that got scored.

Reading forward to the first divergence#

The mechanic worth internalising is that you are looking for a single location — the earliest point at which the system had correct information and produced something incorrect, or had incorrect information handed to it from outside. Everything after that point is downstream and mostly uninformative, because a model given a false premise will elaborate on it competently. The earliest point matters practically as well as logically: a fix applied to a downstream step leaves the upstream defect standing, and that defect goes on manufacturing false premises for whatever replaces the step you fixed.

Earliest is not the same as only, and the boots case is the counterexample. The tool's collapse of a parse failure into not_found was not caused by the date window, and it would have happened on a clean first turn, because it is a property of that tool's response shape rather than of anything the model did. So the rule holds within a causal chain and not across a conversation: read to the first divergence, then keep reading, because a later step handed correct input again can diverge on its own account.

Take a second case. A user asks what the return window is on a ceiling light, and the output says 60 days, which is wrong; the window on that product line has been 30 days since April. Read as a symptom, this is "the model doesn't know the return policy". Read forward, the trace shows a retrieval step returning four chunks, all of them from the shipping FAQ, none of them containing a return window at all, followed by a model that answered from parametric knowledge — what it absorbed during training, as opposed to what the application handed it in the prompt — without indicating that it had nothing to go on. The divergence is at the retrieval call, and specifically in the query it was given, which was the user's raw message including the product name: a query that matched shipping content because the product page text mentions delivery times four times and returns once.

Working the test is what puts the divergence at the query rather than at the index. The retrieval step was handed a query built from the raw user message, and the four chunks it returned are the closest matches to that query in a store where the product page leans on delivery language: handed something wrong, it emitted something correct given it. The step that built the query was handed a message about returns and emitted a search for a product name, which is the yes-then-no pair. The answering step then fails on its own account, since a correct output from four chunks containing no return window would have reported having nothing to answer from, and a number came out instead.

The note that comes out of that reading is one sentence — retrieval returned four chunks, none containing a return window, and the model answered anyway. It names a location, a condition and a behaviour, and it can be counted later against other cases with the same shape, which is what open coding and a codebook (chapter four, where notes like this are grouped into named modes and the names are written down tightly enough that a second reader applies them the same way) are for. "Wrong return policy" names none of those things and will merge with three unrelated failures the moment somebody tallies it.

Four things are worth having in front of you for each case, and if your logging does not capture them the reading is not possible: the full turn sequence including every tool call and its arguments, the raw tool responses including errors and latencies, the retrieved context with document identifiers and scores, and the version of the prompt and model that produced the run. Each of those exists to make one half of the test answerable. Arguments are what a model turn emitted, while responses and retrieved context are what the next turn was handed, and without the arguments you can see that a step returned nothing useful and never whether it was asked for the right thing. Every tool failure then reduces to "the tool didn't help". Traces missing tool arguments are the most common gap and the most expensive one, because arguments are where a large share of agentic failures actually occur. Retrieval traces that record chunk text but not chunk identifiers are the second most common, and they make it impossible to tell whether the same bad chunk is being pulled repeatedly across cases, which is usually the whole finding.

Expect four to six minutes per trace once you have a rhythm, longer for conversations past about ten turns. Cases that resist — where you have read the whole trace twice and still cannot say where it went wrong — should be marked as such and left, rather than escalated into a debugging session. A small number of unreadable cases is normal and worth recording as a count, because a rising count means the logging has degraded.

A pass rate carries no composition#

The same problem appears one level up, at the number everybody watches. Suppose a release drops the pass rate on a 200-case set from 0.86 to 0.79. That tells you 14 more cases fail than did before, that the difference is larger than the run-to-run noise you would expect from sampling alone — with p = 0.86 on n = 200 the standard error is about 2.5 points, so a 7-point move sits near three standard errors — and that whatever changed is broad enough to appear on a set of this size. That is genuinely useful, and it is where the usefulness stops.

What the number does not carry is composition. It does not say whether the 14 additional failures are one mode appearing fourteen times or six modes appearing once or twice each; whether the new build broke something or merely exposed cases that were fragile before; whether some previously failing cases now pass, which would mean more than 14 cases changed state in each direction; or whether any of the movement is the generator sampling differently on a rerun of the identical build. Aggregates are designed to compress exactly the information you need in order to act, which is why an engineer can stare at one for an hour and come away with a sentence that could have been written before the run started.

That compression runs one way, and being precise about why is what makes instrumentation a product decision rather than housekeeping. On this set the two rates mean twenty-eight failures before and forty-two after, and every arrangement of forty-two failures across the possible modes produces the same rate: all of one mode, a dozen modes with three or four cases each, or any split between. The rate is the total of a column of per-case verdicts, and a total does not record which entries were set. Each of those entries was itself produced by mapping a whole conversation onto one of two values, so the discarded material is not sitting compressed inside the score waiting to be decoded. It was dropped when the verdict was written. No operation performed on 0.79 recovers it, and the only route back is the traces themselves, which exist only if something wrote them down at the time. The same property is why a rate that improves cannot say whether a named mode was fixed, since it moves the same way when the mode is untouched and other modes improve, and the question is settled instead by counting that mode's own cases before and after (chapter seven, where a fix is measured on the cases carrying the mode rather than on the whole set).

There is a tempting shortcut worth naming now because it recurs. Faced with 42 failures where there used to be 28, the natural move is to read only the cases that flipped from pass to fail, on the theory that those are precisely the regression. The theory depends on pass and fail being stable labels for a case, and with a sampled generator they are not: rerunning the same build on the same set typically flips a few per cent of cases in each direction. The chain behind that is short. The step that picks each piece of text is allowed to be random, which makes the same build produce different wording on a second run over the same input. A scorer reading different wording can return a different verdict. Pass and fail are therefore labels on a run of a case rather than properties of the case. Some of the flipped cases are noise, and some genuinely new failures sit inside cases that were already failing for an older reason. The flipped set is a reasonable place to start reading and a poor definition of the regression, which is why the choice of what to read is made deliberately (chapter three, where the sample is drawn to cover the traffic rather than taken from whichever rows changed state).

Reading is not debugging#

Conflating the two is the fastest way to ruin the exercise. Debugging means picking one case and driving it to root cause with a debugger, replays and instrumentation, which can take an hour per case. Reading means spending four to six minutes on a trace, writing one or two sentences about what went wrong and where, and moving to the next one. The output is not a fixed bug; it is a note. The temptation to stop and fix the first thing you find is strong, and giving in to it is how you end up with a confident opinion built on the first eight cases you happened to open.

The two-part test also has a precondition that will not always hold. Asking whether a step's output was correct given its input presumes there is something to check the output against. A tool call has one, since its contract says what a well-formed request looks like and what each error name means. A retrieval step often has none, because no document says which four chunks were the right four for a question about ceiling lights. Where the specification is missing the first question stays answerable and the second becomes a judgement, which two readers will make differently, and what keeps such a case usable is writing the judgement into the note rather than leaving it implied.

Reading is also not a replacement for aggregate metrics. Pass rate on a fixed set is a fine regression alarm and a fine communication device, and nothing here suggests dropping it. The claim is narrower: the alarm tells you to look, and only the traces tell you what to look at. Nor does it replace a golden dataset of the kind topic 3.4 assembles — that set is what produces the failing cases in the first place, and the relationship runs both ways, since the modes you find by reading are also what tell you the set is missing coverage.

Finally, it is not free. Fifty traces at five minutes each is a little over four hours of reading, plus an hour of writing up, which is most of a working day for one engineer. That cost is why this chapter is arguing at all: the day is easy to decline when a dashboard is already producing a number, and the number is what made two days of prompt work look reasonable.

What a trace buys is the difference between a description and a cause. A grid supports "wrong answer", which names the user's experience and points at whichever component the reader already suspected. A trace supports "the date window was built for the wrong year", which names one line of one function and is either true or false. Only the second kind of note can be counted with other notes, argued about, or checked afterwards, and the whole of the rest of this topic runs on notes of the second kind.

The setting so far has been an eval run, where a scorer hands you a list of case identifiers and says which ones failed. Production does not do that. There is no scorer marking conversations, no label on the row, and no queue of 87 cases waiting to be read — just several thousand conversations a day, of which some unknown fraction went badly. Getting a pool of failures out of that is its own problem, and every mechanism that produces one distorts what you will find.