Nominating failures from production traffic

On an eval set the scorer decides which cases failed, and whatever its faults it applies the same rule to every case. In production nothing applies any rule. Conversations happen, they are logged, and none of them arrives labelled. To get a pool of failures to read, somebody has to nominate one, and the nomination is where the analysis is decided: read a pool assembled from complaints and you will produce a taxonomy of what annoys people, which is a legitimate finding and not the one you asked for.

That absence is worth taking literally before anyone works around it. A failure, for this purpose, is a mismatch between what the user was trying to accomplish and what the system gave them. Evaluating that predicate needs both halves of the comparison, and the record supplies only one. A log holds every token the system produced, every tool it called, every document it retrieved and every millisecond it took. It holds no field containing what the user was actually after, because no part of the system ever computed one — text arrived, a continuation came back, and at no point was an intention represented anywhere that could have been written down.

The user's words are evidence about that intention rather than a statement of it. "What's the return window on this?" is asked by a shopper who has already decided to send the item back and by one who is choosing between two products, and the same answer serves one and not the other. Even where the intention is unambiguous, whether the answer served it usually depends on facts that are not in the conversation at all — whether the figure quoted is the current figure, whether the order the user meant is the order the system looked up. A trace can be internally flawless and still be a failure, and nothing inside it says so. Failed is therefore not a column in the log, and it is not a function of the columns either.

Anything that selects conversations automatically is therefore selecting on something else. A nominator is a rule that fires on a property of the record — a rating field set to down, a handoff row, a question repeated within two turns, a non-zero error status, a judge score under a threshold — and it is chosen because conversations carrying that property fail more often than conversations that do not. That relationship is a correlation, not a measurement, and the two ways it comes apart are both routine. The rule fires on conversations where nothing went wrong, and it stays silent on conversations where something did. Neither gap is an accident of this month's data; both are properties of the rule, so neither closes by running the rule over more traffic.

What comes out of those rules is the population every later number is about. Modes get named from it, counted in it and projected out of it into monthly traffic, and each of those steps inherits whatever the rules let through. The mechanical part — the query that assembles the rows — is the least consequential part of the design and reliably absorbs most of the effort.

Every surface sees a different failure#

The surfaces available to most teams fall into six families, and their blind spots are systematic rather than random.

Each of these is a real signal and none of them is a sample of failures. A pool assembled from thumbs-down alone answers the question "what makes users complain". A pool assembled from tool errors answers "where does the plumbing break", which existing alerting probably answers already.

The third column of that table is not a list of unlucky exceptions, and each entry follows from what would have to be true for the surface to fire at all. A thumbs-down needs three conditions to hold at once: the user noticed the answer was wrong, cared enough to act on it, and was still in the session when they noticed. The first condition is the one that does the damage. Somebody asks a question because they do not know the answer, and not knowing the answer is exactly the state in which a wrong one is indistinguishable from a right one. An answer that is fluent, correctly formatted and confidently stated suppresses the signal through the same property that makes it expensive. Negative CSAT carries the same condition and adds one more, since the survey has to be answered by somebody who has already left.

Escalation needs the user to still want something after the answer arrived, so what it cannot see is everyone the answer satisfied — correctly or otherwise — along with everyone who gave up quietly rather than asking for a human. Retries and rephrasings fire on a weaker condition, and the weakening is worth naming because it explains why they reach further. Rephrasing a question requires only that the user noticed the response did not address what they asked, and that is a judgement anybody can make about an answer without knowing what the correct answer would have been. Detecting a false answer needs the truth; detecting a non-answer does not. Rephrasings therefore reach modes that no complaint-based surface will ever reach, and stop dead at the fluent answer that addressed the question and got it wrong.

A product-routed handoff or a guardrail trip fires on a rule somebody wrote in advance, which means the population it cannot see is defined by an author's imagination on the day the rule was written. It is closer to a measurement of the routing policy than of the conversations. Tool errors are the most trusted of the six surfaces and the narrowest: a non-zero status, a timeout, an empty result set and a schema violation are all statements about whether the call completed in the shape the caller expected. None of them is a statement about whether what came back was true. A retrieval that returns three well-formed, confidently written chunks describing last year's policy returns status ok, and the machinery has no vocabulary for what is wrong with it.

The judge is the subtlest of the six, because it feels like a general-purpose scorer and it is not: it flags cases matching criteria somebody wrote down, so it can only ever nominate failures that were already imagined. Underneath that there is a second blindness, and it is the more dangerous of the two. A judge handed the conversation and a rubric, without the retrieved chunks beside it and without access to the policy the answer cites, has nothing to compare the answer against except its own sense of what a good answer looks like. Fluent, complete, well-structured and internally consistent is what a good answer looks like. That is also the exact description of the worst failures on the audit below, which the judge scored four out of five.

The belief worth taking apart here is that a nominator's output is a sample of failures. It is a sample of the failures that trip that nominator, and the difference between those two things is a selection step that can be written down. Suppose one mode leaves a tool error in half the conversations it spoils, while a second mode, equally common in production, leaves one in twenty. Every conversation of either kind that errors enters the pool, and nothing else enters at all. The first mode therefore arrives in the pool ten times as often as the second while the two are level in production, and the pool's composition is the real composition with every mode multiplied by its own chance of tripping the rule.

Reading more of that pool does not touch the multiplication. A longer read estimates the pool's contents more precisely, and the pool's contents are the reweighted population, so the extra precision accumulates around the wrong quantity. A mode that never trips the rule has a multiplier of zero and is not rare in the pool but absent from it, at any sample size, for any reader. This is the property a base rate has to escape rather than average away (chapter 5, where a mode's count out of fifty becomes a rate over conversations only by being projected through a failure rate the random arm supplied): a denominator taken from the pool carries the multipliers with it, and a denominator taken from a random draw does not.

Combining the surfaces helps, and it does not solve the problem, because the union of six biased filters is still a biased filter. What the union changes is the multiplier, not its existence — a mode's chance of entering the pool becomes its chance of tripping at least one of the rules, which is still a property of the mode rather than a constant across modes. The confident wrong answer that nobody complains about trips none of the six. Fluent, well-formatted, plausible and wrong is the profile of the failure that costs the most and nominates itself the least.

The random arm is what makes the rest interpretable#

The only mechanism with no opinion about what a failure looks like is a uniform random draw from all traffic. Having no opinion has a precise meaning here: the chance of being selected is the same for every conversation and depends on no property of any of them, so the multiplier from the paragraphs above is one constant shared by every mode. Multiplying every mode's share by the same constant leaves their proportions where they were. That is the whole reason the composition of a random sample estimates the composition of traffic while the composition of a nominated pool cannot, and it is a fact about the drawing procedure rather than about the size of the draw.

Take thirty conversations at random, read them the way chapter 1 sets out — from the first turn forward, to the point where the system held what it needed and produced something wrong anyway — and judge each one yourself. That judgement is the step with no machinery underneath it, and there is no arrangement that removes the person from it, because the label being supplied is exactly the quantity the record does not contain. Somebody has to reconstruct what the user was after and decide whether they got it. Two readers will not always agree, which is what a written codebook is for (chapter 4, where each mode gets a definition and a boundary case so that two people coding the same traces produce the same counts).

Most will be fine. The ones that are not give you two things nothing else can: an estimate of the overall failure rate on real traffic, and an estimate of how many failures the nomination surfaces are missing. If six of thirty random conversations turn out to be failures and only one of the six carried any signal from any surface, the nominated pool is seeing roughly one failure in six, and the modes it over-represents are the loud ones. Run the same check quarterly and you can watch coverage move as the product changes.

Both of those estimates are coarse, and saying so is more useful than dressing them up. Six failures out of thirty is one in five, and a draw that happened to contain five or seven would have moved that rate by a fifth in either direction with nothing about the product being different. Thirty conversations settle whether coverage is bad and cannot settle it per surface, because six failures spread over five surfaces leaves one or two apiece, and a recall computed from one case is not an estimate of anything.

Two hundred conversations is enough for the per-surface figures, and at four to five minutes each that is a day and a half of reading, which most teams can afford once a quarter. A random audit of 200 conversations on a shopping assistant produced 16 failures by hand judgement, and cross-referencing those 16 against the nomination surfaces gives two figures for each: precision, the share of the conversations a surface flagged that turned out to be genuine failures, and recall, the share of all the genuine failures that the surface flagged.

Every row is two divisions, and the two divisions have different denominators for a reason worth spelling out. Thumbs-down fired on two conversations and both were genuine failures, so its precision is two divided by two, everything it flagged. Its recall divides that same two by the sixteen failures the audit found, which is one in eight and appears in the table as 13% once rounded. The judge fired on twenty-one of which nine were genuine, a precision a little over two in five, and those same nine against sixteen give a recall of just over a half.

The precision column can be computed by any team with a flagged queue and somebody to read it, since both of its numbers come from the flagged conversations themselves. The recall column cannot. Its denominator is the count of genuine failures among the two hundred, and that count exists only because two hundred conversations were drawn without reference to any surface and then judged one at a time. Nothing done to the nominated pool produces it, however much of the pool gets read, which is the argument for the second arm compressed into one column of a table.

Two things fall out of that table. The union catches three failures in four, which is better than any single surface and still means a quarter of failures are structurally invisible to the nomination machinery. And thumbs-down, the surface teams reach for first because it is the most explicit signal of user dissatisfaction, has the worst recall of the six — high precision, almost no coverage. A month of analysis built on thumbs-down alone studies 13% of the problem while believing it is the whole of it.

The union row is also not the sum of the rows above it, and the gap carries information the individual rows do not. Adding the five firing counts gives fifty firings spread over thirty-eight distinct conversations, so somewhere between six and twelve of those thirty-eight tripped more than one surface. The same addition over genuine failures gives twenty-six flags raised against twelve real failures, a shade over two flags apiece. That duplication is why a candidate pool has to be collapsed to one row per conversation before it is counted.

The four failures nobody caught are worth reading individually rather than as a statistic, because they define the shape of the blind spot. On this audit they were variations of one thing: a fluent, correctly formatted, confidently stated answer that was factually wrong, with no tool error, no user pushback and a judge score of 4 out of 5. Run that profile past the conditions set out above and every surface fails to fire for its own reason. Nothing errored, so there is no status for the machinery to catch. The user did not rephrase, because the answer addressed the question they asked. They did not escalate, because they had what they came for. They did not press thumbs-down, because from where they were sitting the assistant had been helpful. And the judge scored it well, because on every property the judge could see it was a good answer. That is the profile the entire nomination apparatus is built to miss, and knowing it is the profile tells you what to write the next judge criterion about.

The two arms do different jobs and both are needed. The nominated pool is where the modes come from, because it concentrates failures and lets you spend a reading budget on cases that are mostly interesting. The random draw is where the denominators come from, because it is the only sample whose base rate means anything — a rate computed over a nominated pool is a rate over the complaining minority, and multiplying it by monthly traffic produces a number that is confidently wrong.

Assembling the pool#

The mechanical part is a query. What matters is that each candidate carries the source that nominated it, because you will want to know later whether a mode was found only through one channel.

-- candidate pool for the July read, one row per conversation,
-- with every surface that nominated it
with nominated as (
  select conversation_id, 'feedback'   as source, created_at
    from message_feedback where rating = 'down'
  union all
  select conversation_id, 'escalation', created_at
    from handoffs where reason in ('user_requested', 'agent_gave_up')
  union all
  select conversation_id, 'rephrase', created_at
    from conversation_signals where repeated_intent_within_turns <= 2
  union all
  select conversation_id, 'tool_error', created_at
    from tool_calls where status != 'ok' or result_count = 0
  union all
  select conversation_id, 'judge', created_at
    from judge_scores where score < 3
),
random_arm as (
  select conversation_id, 'random' as source, created_at
    from conversations
   where created_at >= now() - interval '30 days'
   order by random() limit 30
)
select conversation_id,
       array_agg(distinct source) as sources,
       min(created_at)            as first_seen
  from (select * from nominated union all select * from random_arm) u
 where created_at >= now() - interval '30 days'
 group by conversation_id;

The random arm sits inside the same query rather than beside it, for a reason that is easy to miss. A recall figure needs to know, for every randomly drawn conversation, which surfaces fired on it. Aggregating the sources over the union does that: a conversation drawn at random that had also tripped the judge comes back tagged with both sources, while one that tripped nothing comes back tagged random alone. Those two cases are the caught and uncaught columns of the audit table, produced by the query rather than reconstructed by hand afterwards.

Deduplication is not cosmetic. A single bad conversation frequently trips three surfaces — the tool errored, the user rephrased, then thumbs-downed — and if you count rows rather than conversations, the modes that produce visible machinery failures appear roughly three times as often as they should. The audit above shows the same effect at a smaller scale, with twenty-six flags standing for twelve real failures. Collapsing to one row per conversation with an array of sources fixes the count and preserves the information.

Every surface in that query also carries a threshold, and score < 3 is the visible one. A stricter threshold flags fewer conversations, of which a larger share are genuine, so precision rises and recall falls; a looser one trades the other way and spends reading time to do it. The size of the pool is therefore a setting rather than a finding. Two more fields are worth carrying from the start, the intent or topic classification and the week the conversation happened in, because both are what the sampling design downstream stratifies on (chapter 3, where the fifty traces to be read are drawn inside cells defined by intent and by week, with a floor that keeps the small cells from vanishing); adding them at query time costs nothing, and recovering them after drawing a sample means redrawing it.

The failures you cannot nominate#

Retention is the first limit. Most teams keep full traces for 14 or 30 days and aggregate metrics for a year, and the asymmetry is a storage decision: a metric is a handful of numbers per day whatever the traffic does, while a trace is kilobytes per conversation. A mode that started three months ago is consequently visible in a metric and unreadable in a trace, which leaves you able to date its onset and unable to say what it is. If you are going to do this work, traces have to be logged before you need them, at the level of detail chapter 1 sets out — the whole exchange with its tool calls and retrieved chunks, not the reply. Logging only the final message is the same as not logging.

Sampling at the logging layer is the second. Systems under cost pressure often log full traces for 1% or 5% of traffic and only summaries for the rest, and a nomination query that joins against full traces will silently restrict the pool to the sampled slice. That is survivable if the sampling is uniform and fatal if it is not — logging "all conversations that errored, plus 1% of the rest" produces a store where machinery failures are a hundred times over-represented. That factor is the same multiplication as before, and it is short enough to check. An errored conversation reaches the store with certainty. A clean one reaches it once in every hundred attempts. The ratio between those two chances is precisely what the store's composition gets multiplied by, and the distortion is already there before any nomination query runs.

Access is the third. Conversations contain names, addresses, order numbers and sometimes payment details, and reading fifty of them is a legitimate use that still needs to happen inside whatever your data policy allows. Redaction pipelines that strip entities before storage interact badly with this work, because "the model used the wrong name" is invisible once every name is [REDACTED]. Redaction breaks the failure predicate itself: a failure is a mismatch between two values, and stripping one of them from the record makes the comparison impossible rather than merely harder. A workable compromise is pseudonymisation that is consistent within a conversation, so you can still see that the model referred to the customer by the wrong consistent token. That recovers the mismatches internal to the conversation and none of the ones needing the real value, so a transposed order number stays invisible.

There is also a limit no logging fixes. Some failures are not in the conversation at all. A user who asked a question, received a fluent answer and never came back has produced a trace that reads perfectly, and only a downstream signal — a return, a chargeback, a repeat contact on a different channel a week later — reveals that anything went wrong. That is the missing half of the comparison in its hardest form, absent not only from the record but from the whole system until the consequence lands somewhere else. Joining conversations to downstream outcomes is a project of its own and worth knowing you are not doing.

The pool you end up with is a designed object rather than a query result. Its sources are recorded per candidate so that a mode found through one channel only can be recognised as such; its biases are known because the random arm measured them; and its coverage is a number — three failures in four, on this product, this quarter — rather than an assumption. Everything downstream inherits those properties, which is why the design is worth an hour before the reading is worth a day. None of that design makes the nominators into measurements. It makes them into proxies whose distortion has been measured, which is the strongest thing available given that no field anywhere in the record says a conversation failed.

What you have at the end is roughly four thousand candidates from the last month, each tagged with the surfaces that nominated it, its intent and its week, plus the random draw sitting beside it as a calibration arm. That is more than anyone will read. Which fifty come out of it determines what the entire analysis will conclude, and drawing them badly is easy enough that two engineers working from the same pool can reach opposite conclusions about the same product.