Investigating a failure that left no stack trace

The ticket is four lines long and arrives on Thursday 5 March. The HR director at Calderwood Group has forwarded a screenshot: an employee asked the benefits assistant how much paid parental leave he was entitled to, and the assistant told him eight weeks. Calderwood raised that entitlement to sixteen weeks on 1 January, and the employee has already told his manager he expects to be back at his desk in mid-May. The ticket carries a conversation id, cv_8b41d2e7, and a date, Tuesday 3 March. Nothing else about that request is unusual. It returned HTTP 200 in 2.41 seconds, it produced a well-formed answer of 140 words with a citation block at the bottom, and it sits in the request log between forty thousand other conversations from the same day that nobody has complained about.

What that ticket starts is a post-mortem: a written investigation, carried out after a failure is over rather than while it is burning, that establishes what the system actually did, names the conditions that allowed it, and leaves behind a record someone else can use. The word is borrowed from medicine by way of operations engineering, and the borrowing is honest about the timing, since the examination is not going to help this patient and its whole value is what it teaches about the next one. The problem it solves is specific to systems that fail the way this one did. An assistant that answers wrongly does not crash, page anyone or move an error budget, so the natural life of a defect like Calderwood's is to be patched by hand for the customer who complained, closed as resolved, and then rediscovered eight months later by an engineer who has to work the mechanism out again from nothing, at roughly the cost of the first investigation.

The other half of the work is where those write-ups end up. A failure mode is a way the system can go wrong stated as a mechanism — superseded documents staying retrievable after a re-index, for instance — rather than as a symptom such as "the answer was wrong", and a collection of failure modes that people can search, assign and check is what the phrase institutional memory means once it is made concrete: the team's knowledge of how this particular system breaks, held somewhere other than in the heads of whoever happened to be on call at the time. You meet all of it in unremarkable places. A ticket arrives, someone spends a day inside the evidence, a review happens the following week, and an entry lands in an archive where it will either answer a stranger's question in September or sit unread. The day inside the evidence comes first, and it is the part that decides whether there is anything worth writing down at all.

The instinct is to open the assistant, type the question, and watch what comes back. Doing that on 5 March produces a correct answer — sixteen weeks, cited to the 2025 handbook, in 2.2 seconds. That result is worth almost nothing as evidence, because the system that answered on 5 March is not the system that answered on 3 March. The prompt template moved from bp-answer@v36 to bp-answer@v37 on 27 February as part of a citation-formatting change. The retrieval index for that tenant was rebuilt in place on 4 March, one day before the investigation, when an unrelated migration re-ran ingestion with a newer chunker — the component that cuts each source document into the passages, or chunks, that get indexed and later pasted into the prompt — which changed the boundaries of every chunk and therefore every similarity score the retriever computes. The served model identifier is the same string in both cases, but the provider rolled a serving-side update in the interval. Any one of those three would be enough to make a re-run tell you about today.

So the primary evidence for a failure like this is the trace of the specific requests that failed, carrying the prompt version, the resolved model version, the retrieval context that was actually assembled, and the tool calls that actually ran, all as they stood at 14:22 UTC on 3 March. Topic 15.16 established what those fields have to contain to be semantically useful during an incident. What this chapter is about is what happens when you come looking for them two days later, from a customer ticket rather than an alert, with no idea yet which subsystem is implicated.

The re-run answers a question nobody asked#

Reproduction is the reflex a backend engineer brings from every other kind of debugging, and it is well earned there. A null-pointer dereference in an order service reproduces because the code path is deterministic and the deployed artifact is the same one that crashed, so a failed reproduction is genuine evidence that you have the wrong input or the wrong hypothesis. In an assistant, the artifact is a composition of a prompt template, a model endpoint, an index, an embedding model, a set of tool schemas and a decoding configuration, and at least one of those changes most weeks without anyone thinking of it as a deploy.

There is also a second problem that is easy to miss. Even holding all of that fixed, the generation step is sampled. At a temperature of 0.3 — the decoding setting that controls how much randomness the sampler is allowed when it picks each next token, where 0.0 always takes the most likely one — the same prompt with the same context can return sixteen weeks on one call and eight on the next, if both figures are present in the context window. A single re-run that comes back correct is consistent with the failure being fixed, with it never having existed, and with it occurring on 30% of calls and you having drawn a good one. Two colleagues can run the same probe an hour apart and reach opposite conclusions about whether there is anything to investigate, which is how these tickets get closed as "cannot reproduce" while the underlying defect keeps serving.

The only thing a re-run genuinely establishes is the current behaviour of the current system, which matters later — after you have a cause and a mitigation, it is how you check the mitigation holds. Running it first, before you know what changed, converts a live investigation into an argument about whether the customer is mistaken.

The evidence is scattered, and each piece decays at its own rate#

An investigation like this rarely lives in one system. The conversation id is the join key, and the work is pulling the six things it can be joined to before the shortest retention window closes on you.

The order you pull these in is set by what expires soonest and what is cheapest to check, and it is worth being deliberate about because an investigation that wanders burns the window. Freeze first: before anything else, ask for a retention hold on the traces, request logs and index snapshots for that tenant, since a fourteen-day trace window and a three-day investigation leave less margin than it sounds. Then take the deploy and job history for the eight days before the failure, which is a two-minute query and immediately tells you whether a prompt, a model alias, an index rebuild or an ingestion run landed in the interval. Only then open the individual trace, because you now know which of its fields are suspicious. The support thread comes last and is read for what the user was trying to accomplish, not for the diagnosis, since the user's theory of what went wrong is a report of the symptom.

Evaluator scores deserve a note because they are the one source that can disagree with itself. The offline suite for this tenant scored 0.91 on groundedness — the measure of whether an answer's claims are supported by the context it was handed — for the week of 3 March, and the online judge, which is a second model scoring a sample of live outputs against a written rubric listing what a good answer has to do, put the parental-leave conversations at 0.88, both comfortably above their alert thresholds and both wrong about this failure in the sense that neither flagged it. That disagreement between a confident customer and a confident evaluator is itself evidence, and what to write when it happens is the subject of chapter 6. For now it belongs in the evidence pile with everything else, recorded rather than resolved.

The row that ends most investigations is the first one. Trace sampling at 10% is a reasonable default for latency debugging, where any hundred requests tell you the same story, and it is close to useless for post-mortems, where you need one specific request from nine days ago. The row that surprises people is the fourth. An index rebuilt in place has no history, so a question like "what did those 2024 chunks score against that query before the re-chunk?" becomes unanswerable the moment the rebuild completes — which, in this incident, happened on 4 March, the day before anyone went looking, and is the reason the trace's recorded scores are the only record of the ranking that produced the failure.

What survived, and what it showed#

Calderwood's tenant was in the 10% sample by luck, so cv_8b41d2e7 had a retained trace. Stripped to the fields that mattered, it looked like this.

{
  "conversation_id": "cv_8b41d2e7",
  "tenant_id": "t_calderwood",
  "ts": "2025-03-03T14:22:07Z",
  "prompt_template": "bp-answer@v36",
  "model": {
    "alias": "assistant-large",
    "resolved": "large-instruct-2025-01-22",
    "temperature": 0.3,
    "max_tokens": 800
  },
  "retrieval": {
    "query": "how much paid parental leave do I get",
    "k": 4,
    "chunks": [
      {"doc": "benefits-handbook-2024.pdf", "chunk": 118, "score": 0.842,
       "ingested_at": "2024-04-02T11:03:00Z", "doc_version": null},
      {"doc": "benefits-handbook-2024.pdf", "chunk": 119, "score": 0.804,
       "ingested_at": "2024-04-02T11:03:00Z", "doc_version": null},
      {"doc": "leave-policy-summary.pdf", "chunk": 12, "score": 0.771,
       "ingested_at": "2025-02-12T09:14:00Z", "doc_version": "2025.1"},
      {"doc": "benefits-handbook-2025.pdf", "chunk": 96, "score": 0.702,
       "ingested_at": "2025-02-12T09:14:00Z", "doc_version": "2025.1"}
    ]
  },
  "tools": [],
  "output_sha256": "3f9c…a1",
  "cited_docs": ["benefits-handbook-2024.pdf"]
}

Two of the four retrieved chunks came from a handbook that was superseded on 1 January, they outranked the current handbook by 0.14 and 0.10 of cosine score — the similarity between the query's embedding and the chunk's, running from 0 for unrelated text to 1.0 for the same direction, which is the only quantity this retriever ranks on — and the model cited the higher-ranked one. The doc_version: null on those two chunks is the loose thread. It says they were written into the index before the ingestion pipeline started stamping a version field, which turns out to be the whole story, and chapter 2 is where that gets stated as a cause rather than an observation.

From the trace alone you also get scope. Filtering the index for chunks with doc_version: null in that tenant returns 604 chunks across four superseded documents, and replaying the retrieval step over the tenant's logged queries for the period — which the request log retains for 90 days, even where full traces do not — puts at least one of those chunks in the context of 63 conversations between 13 February and 5 March. Nineteen of those produced an answer that contradicts the current handbook on a figure or a date. That is a number you can put in front of an account team, and it took about forty minutes to produce because the query text was retained even where the trace was not.

What the trace does not do#

A trace tells you what the system did. It does not tell you whether what the system did was wrong, and in this incident it happened to be easy only because a handbook figure is checkable against a document. Where the answer is a judgement — a tone, a recommendation, a summary that leaves something out — the trace shows a plausible context and a plausible output and settles nothing. Error analysis, from topic 3.6, is the discipline that decides whether an output is a failure and what to call it; the trace is the input to that decision rather than a substitute for it.

The trace also does not establish harm. Nineteen contradicting answers is a count of outputs, and the cost depends on how many of those employees acted on the number, which lives in leave requests and manager conversations that no system here can see. And it does not tell you what a user was actually trying to do, so the support thread stays evidence in its own right; the employee's original message said he was "planning around going back in May", which is why this ticket got escalated at all.

There is one more thing it cannot do, which is tell you the failure was singular. A trace is a record of one request, and a post-mortem written from one request tends to produce a fix scoped to one request — a hand-edit of the Calderwood index, a note to the account manager, a closed ticket. The replay over logged queries is what moves the finding from a story about an employee to a statement about 63 conversations and four documents, and the same replay across the other 339 tenants is what would tell you whether Calderwood is special. It is not, as it happens: eleven other tenants had chunks with doc_version: null still live, though only three of them had a superseded document that contradicts its replacement on a specific number, which is why nobody else complained.

The uncomfortable part of this investigation is that its success was luck. A tenant outside the 10% sample would have left a request log with query text, an index with no history, and no record of which chunks were assembled — enough to guess at retrieval, not enough to demonstrate it. The retention and pinning changes that would make the next one deterministic are, properly, a finding of this post-mortem rather than a prerequisite for it. Having established what the system did on 3 March, the next question is how to write that down so it means something to someone reading it in September, which is where the difference between naming a mechanism and labelling the model shows up.