A register works when its checks run and its entries are owned

The design review for the multi-corpus feature runs for fifty minutes in September and nobody opens the incident archive. That is not negligence; the archive is a wiki page titled AI Incidents 2025 with thirty entries in reverse date order, each a heading and a few paragraphs, and the question in the room — "if a tenant has two corpora with overlapping documents, what breaks?" — cannot be asked of it. The engineer who needs the answer would have to read thirty write-ups looking for anything about document versions, which takes an hour she does not have during a review, so she does what everyone does and relies on the two incidents she personally remembers. FM-2025-014 is not one of them, because she was on a different team in March.

A failure register is the standing collection of those write-ups, one row per failure mode rather than one page per incident, kept for as long as the system runs, and it earns the name when it can be interrogated at the moment someone is about to make the same mistake. That means three things about each entry: someone owns it, it records which versions of which components it applied to, and it carries a check that executes somewhere without a human choosing to run it. Thirty entries with those properties is a small database and a nightly job. Thirty entries without them is a document, and a document's only mechanism of action is that somebody happens to read it.

The fields that make an entry answerable#

Storage is unglamorous and it is most of the difference. The Calderwood entry from chapter 2 already has the content; what it lacks is a shape that supports the two queries that matter — "what do we know about X?" asked during design, and "has this come back?" asked by a job.

CREATE TABLE failure_mode (
  id              text PRIMARY KEY,          -- FM-2025-014
  title           text NOT NULL,
  status          text NOT NULL,             -- open|mitigated|guarded|retired|regressed
  owner           text NOT NULL,             -- a person, not a team alias
  first_seen      timestamptz NOT NULL,
  last_seen       timestamptz NOT NULL,
  components      text[] NOT NULL,           -- {ingestion,retrieval}
  affected        jsonb NOT NULL,            -- versions in force when it fired
  detection_gap   text NOT NULL,             -- absent|averaged|unowned|infeasible
  time_to_notice_h numeric,
  tags            text[] NOT NULL,           -- {document-versioning,staleness,multi-tenant}
  body            text NOT NULL              -- the write-up
);

CREATE TABLE recurrence_check (
  id         text PRIMARY KEY,               -- CHK-014-retrieval
  fm_id      text REFERENCES failure_mode(id),
  kind       text NOT NULL,                  -- test|alert|job_assertion
  locator    text NOT NULL,                  -- tests/regression/fm_2025_014.py::test_no_superseded_chunks_in_context
  last_run   timestamptz,
  last_result text                           -- pass|fail|skipped|missing
);

The affected column is the one that does unexpected work. Recording that FM-2025-014 fired under ingestion@1.9.2, bp-answer@v36 and index schema idx-3, and that the mitigation landed in ingestion@1.10.0, lets a query answer "which known failure modes could still be live in this deployment?" when someone is preparing a release. It also catches the case where a fix is reverted or a component is rolled back to a version that predates the mitigation, which happens more often than teams expect during a rollback under pressure.

With that shape, the design-review question becomes a query rather than an hour of reading.

SELECT id, title, status, owner, detection_gap
FROM failure_mode
WHERE 'document-versioning' = ANY(tags)
   OR components && ARRAY['ingestion']
ORDER BY last_seen DESC;

Twenty seconds, four rows, and one of them describes precisely what the multi-corpus design is about to reintroduce. The tags are the weak point, since they are human-assigned and drift, and the practical mitigation is a small controlled vocabulary — roughly thirty tags, reviewed quarterly — plus full-text search over body as a fallback for everything the vocabulary missed.

Searchability gets you halfway, and the remaining half is that somebody has to think to search. Two cheap mechanisms close most of that gap. The first is a line in the design-review template that asks which register tags the change touches, which forces the query to be run by the person writing the document rather than hoped for during the meeting; on this team it takes about four minutes and it is the mechanism that would have surfaced FM-2025-014 in September. The second is a triage bot that runs when an incident channel opens: it takes the first description someone types, embeds it, matches it against the body and title of every entry, and posts the top three with their ids, owners and current status. It is wrong often enough to be irritating and right often enough to matter — most of the 62% triage match rate in the table below comes from it, and the cost is a nightly embedding job over a few hundred kilobytes of text.

The maintenance cost is worth stating because it is what gets cut when a quarter goes badly. Writing an entry properly, with evidence links and scope numbers, takes two to three hours on top of the investigation. The recurrence check is usually the regression case from chapter 4 and is already being written. The sweep job took a day to build. The quarterly review — walking the open entries, retiring what is designed out, reassigning what has lost its owner — runs about ninety minutes for thirty entries. That is roughly a person-day a month across a team, which is cheap against a single twenty-day detection interval and expensive enough that it needs a defensible answer to "is this working".

An entry has a lifecycle, and most of them stall in the middle#

The two transitions back to open are the ones that keep the register honest. An entry that reaches mitigated and never acquires a check is an entry whose protection is that the fix stays in place, which nothing verifies, so after thirty days the job moves it back to open and notifies its owner. The same applies to an entry whose check exists and has stopped running — the @pytest.mark.skip from chapter 4 makes last_result read skipped, and rather than sitting unnoticed in a test report it reopens the entry that depends on it. That single link is what converted a decision made in a hurry, to silence a noisy test, into a visible decision to drop coverage for a known failure mode.

def sweep(conn) -> list[str]:
    """Nightly: reconcile check results, reopen anything unguarded."""
    rows = conn.execute("""
        SELECT f.id, f.owner, f.status,
               count(c.id) FILTER (WHERE c.last_result = 'pass'
                                   AND c.last_run > now() - interval '30 days') AS live
        FROM failure_mode f LEFT JOIN recurrence_check c ON c.fm_id = f.id
        WHERE f.status IN ('mitigated', 'guarded')
        GROUP BY f.id, f.owner, f.status
    """).fetchall()
    reopened = []
    for fm_id, owner, status, live in rows:
        if live == 0:
            conn.execute("UPDATE failure_mode SET status='open' WHERE id=%s", (fm_id,))
            notify(owner, f"{fm_id} reopened: no passing recurrence check in 30 days")
            reopened.append(fm_id)
    return reopened

Owners have to be individuals. A register whose owner column reads platform-team for eighteen entries produces exactly the behaviour you would predict, which is that the nightly notification goes to a channel and each entry is somebody else's. Ownership rotates when people change teams, and the sweep is a good moment to catch that, since a bounced notification is a signal that the entry needs reassigning rather than that the register is fine.

How you know it is working#

Thirty entries is enough to ask whether the thing has any effect, and the tempting measure is completeness: every incident has an entry, no gaps, the process is being followed. Completeness measures how diligent the team is about writing, and a register can be complete and useless — thirty carefully written entries, none of them queried, none of them guarded by a check, and the same class of failure arriving every quarter. What you want to know is whether knowing about these failures is changing anything.

Median time-to-diagnose is the measure with the most signal, because it is where institutional memory pays out first. An investigator who can search for "stale" and land on an entry describing the null-version mechanism, its trace fields and its job-log evidence starts with a hypothesis and a query to run, and the diagnosis in this system fell from a median of nine days in the first quarter to a day and a half by the third. Recurrence is the harder measure and the more important one, and it needs the register to be honest about what counts — a second incident from the same mechanism through a different code path is a recurrence, even if the original check still passes, and calling it a new failure mode makes the number look good while destroying its meaning.

It is equally useful to know what a failing register looks like, since the answer is rarely that entries stopped being written. Diagnosis times stay flat, the triage match rate sits near zero because nobody can find anything, check execution drifts down as tests get skipped, and recurrences arrive under new incident numbers because no one recognises them as the same mechanism. Any two of those in a quarter say the register is a filing cabinet, and the repair is not more entries. It is picking the three or four entries that describe the mechanisms most likely to return, giving each a live check and a named owner, and letting the rest sit unmaintained until they either prove useful or get retired. A small register that executes beats a complete one that does not, and the version of this that fails hardest is the one where a team responds to poor numbers by mandating write-ups for everything.

The one measure to watch defensively is the last. A near-miss is a fault somebody found and reported before it ever reached a customer, so an entry for one costs nothing except the writing. Their count going up is normally good news about the culture and can look like the system getting worse on a dashboard that counts entries. That relationship, and what happens to it when a post-mortem names a person, is what chapter 6 is about.

What a register is not#

It is not the failure taxonomy from topic 3.6. A taxonomy has labels applied consistently across hundreds of sampled conversations so that rates can be computed; a register has specific incidents with mechanisms, owners and checks. The two connect through tags — three register entries carrying staleness is a hint that the taxonomy should have a staleness label, and a taxonomy label whose rate is climbing is a prompt to investigate — and merging them produces a taxonomy too specific to count with and a register too vague to act on.

It is also not a compliance artefact, and treating it as one is the fastest way to kill it. The moment the register is reported upward as an incident count, the incentive shifts towards fewer entries, entries written to look controlled, and near-misses that never get logged. If an audit needs incident records, keep the audit view as a separate, derived read of the register rather than making the register itself the thing being graded.

Finally, a register is not permanent storage. Entries whose mechanism has been designed out — the null-version population, for instance, once the schema forbids null versions and the backfill is complete — should be retired with a note saying why, because thirty live entries can be reviewed and two hundred cannot. Retirement is a decision an owner makes with evidence, and the difference between retiring an entry and quietly ignoring it is entirely in whether the register records the reason.

All of this assumes entries get written, including the ones about things that almost went wrong and cost nobody anything, which are the entries that make the median diagnosis time fall. Whether people write those depends less on tooling than on what happened to the last person who did.