Post-mortems for AI systems and building institutional memory of failure modes

Read these first

• Incident response for AI systems
• Error analysis

What you will know

• Run a post-mortem on a quality failure with no stack trace

  The evidence is usually distributed across prompts, model versions, retrieval context, traces, evaluator disagreement, and user reports.

• Write up a failure mode so the team recognises it next time

  A useful record captures the trigger, scope, detection gap, contributing causes, mitigation, and supporting evidence rather than merely labelling the model bad.

• Create a searchable failure register with owners, affected versions, and recurrence checks

  A document nobody can find, assign, or query during review does not prevent recurrence.

• Turn a post-mortem finding into an eval case

  Regression coverage must preserve the triggering input and expected behavior without overfitting to one example.

• Present a failure to stakeholders with evidence, impact, and corrective actions without blame or hand-waving

  Blame suppresses reporting, while vague assurances prevent decisions about rollback, guardrails, or accepting residual risk.

Chapters

1. 1

   Investigating a failure that left no stack trace

   With a conversation id and a complaint, the primary evidence is the trace of those specific requests: the prompt version, model version, retrieval context and tool calls actually in force at the time. Re-running the question today exercises a different system and proves nothing when it comes back fine. Most investigations stall because that trace was never retained, which is the thing you go and fix afterwards.

   Read chapter
2. 2

   A contributing cause names a mechanism, not a verdict

   'The model hallucinated' is a restatement of the symptom and teaches nobody anything. 'Retrieval returned the superseded policy document because the index was stale' can be checked, owned and prevented. A write-up that someone will recognise the failure from six months later carries the trigger, the scope of affected traffic, the contributing causes stated mechanically, the mitigation applied, and the evidence each rests on.

   Read chapter
3. 3

   The most valuable line is how long it took to notice

   In AI systems the outage is usually silent, so the interval between the first bad output and the first human noticing is the number that predicts the cost of the next one. Write it down, then write down why no monitor fired: the metric did not exist, was averaged over too much traffic, or fired and nobody owned the alert. That line converts a post-mortem into a detection change rather than a resolution.

   Read chapter
4. 4

   A finding becomes a regression case, not a memorised example

   Turning an incident into eval coverage means keeping the input that triggered it, adding variants around it, and asserting the property that actually failed rather than the exact string that came back. Pin the literal output and you get a test that passes on a paraphrase and fails on a harmless rewording. The variants are what stop the suite from memorising one customer's Tuesday.

   Read chapter
5. 5

   A register works when its checks run and its entries are owned

   Thirty entries in a document nobody can search, assign or query during design review prevent nothing. Each entry needs an owner, the versions it affected, and a recurrence check that executes somewhere. Judge the register by whether time-to-diagnose is falling and whether known failure modes stop coming back, not by whether every incident got written up, which measures diligence rather than effect.

   Read chapter
6. 6

   Blame suppresses the near-misses that are worth the most

   Naming the engineer who shipped the change buys one satisfying sentence and costs you every future report of something that almost went wrong, which is where the register's value actually sits. The same discipline applies when you face stakeholders: evidence, measured impact and corrective actions, without hand-waving. When two evaluators disagree and the customer is certain, record the disagreement itself, because it means your definition of correct is underspecified.

   Read chapter