Four hundred copies of one failure is weight, not coverage

Yesterday's candidate queue holds 900 thumbs-down. Four hundred of them come from one enterprise customer whose contracts team uploads a document format the parser mangles, so every question about a Schedule B addendum comes back with the wrong effective date. One hundred and twenty are the same retrieval bug that was fixed on 11 March, still arriving from clients that had not refreshed. The instinct in the room — real failures are precious, keep them all — produces an evaluation suite in which 44% of the cases are one failure at one customer, and from then on every model comparison, every prompt change, and every retrieval experiment is scored mostly on how well it handles Schedule B addenda. A change that helps that one parser looks like a five-point global improvement. A change that helps everything else looks like noise.

Coverage is a count of distinct behaviours the suite can distinguish, and 400 records of one behaviour contribute one behaviour's worth of coverage and 400 records' worth of weight. Those are separate quantities, and conflating them is the single most common way a production-fed evaluation suite goes wrong, because the conflation is invisible in the only number people look at, which is how many cases the suite contains.

What has to be in a case before it is worth anything#

A thumbs-down is a pointer into a log. Turning it into an evaluation case means capturing enough that the failure can be reproduced next month, on a different model version, by someone who was not there — and, separately, recording what the right answer was, which is the part teams skip.

{
  "case_id": "case_2026_03_18_0417",
  "provenance": {
    "trace_id": "tr_7c19ba03",
    "captured_at": "2026-03-18T09:41:22Z",
    "signal": "thumbs_down",
    "signal_comment": "effective date is wrong, it's the addendum date not the master",
    "tenant_hash": "t_5f2a",
    "cluster_id": "clu_addendum_effective_date"
  },
  "input": {
    "request_class": "document_qa",
    "user_message": "When does the Schedule B addendum take effect?",
    "conversation_prefix_ids": ["msg_88a1", "msg_88a4"]
  },
  "context": {
    "retrieved": [
      { "doc_id": "doc_9931", "chunk": 4, "content_sha256": "b41c…", "score": 0.81 },
      { "doc_id": "doc_9931", "chunk": 7, "content_sha256": "0ae7…", "score": 0.74 }
    ],
    "index_version": "contracts@2026-03-11",
    "tools_called": []
  },
  "generation": {
    "prompt_template": "doc_qa@17",
    "model": "claude-sonnet-4-6",
    "params": { "temperature": 0.2, "max_tokens": 800 },
    "output": "The Schedule B addendum takes effect on 1 January 2025…"
  },
  "expectation": {
    "assertion": "states effective date 14 February 2026",
    "must_not": ["cites the master agreement date as the addendum date"],
    "decided_by": "legal-ops@example.com",
    "decided_at": "2026-03-19",
    "confidence": "high"
  },
  "redaction_state": "pseudonymised",
  "retention": { "policy": "eval-derived-90d", "expires_at": "2026-06-16" }
}

Three parts of that record do the work. The context block with content hashes is what makes the case reproducible: without it, a rerun in June retrieves whatever the index holds in June, the failure does not appear, and the case silently starts passing for reasons that have nothing to do with the model. The generation block pins what produced the failure, so a regression can be dated to a template or a model version. And the expectation block, with a named human and a date, is what separates an evaluation case from a bug report — a case whose expected outcome is "not that" cannot be scored, and a suite full of them turns every run into an argument.

The cluster_id is assigned at capture rather than discovered later, and how it is assigned determines everything downstream. A small model is given the user's message, the free-text comment where one exists, the changed span where the signal was an edit, and the current list of cluster names with one-line descriptions, and asked to pick an existing cluster or propose a new one. Proposals go into a weekly review where a human either accepts the new cluster or merges it into an existing one, which keeps the taxonomy from fragmenting into 400 clusters of one case each. Assigning at capture is what makes the caps in the next section enforceable in real time; a pipeline that clusters retrospectively has already admitted the 400 before it knows they are the same thing.

Severity is scored at the same moment and on a fixed three-point scale tied to consequence rather than to how annoyed the user sounded: a wrong figure or date that a user would act on is 3, a materially incomplete answer is 2, and a stylistic or formatting complaint is 1. That score is what the selection sorts by, and it is the reason the three survivors of the addendum cluster are the three worst rather than the three earliest. Scoring severity from the comment text alone is unreliable in a specific direction — enterprise users write mild comments about severe failures and self-serve users write furious ones about formatting — so the score is derived from what changed in the answer, with the comment as one input among several.

The expectation is also the step that costs money, because someone with domain authority has to decide what the right answer was. On a contracts product that is legal ops, at maybe eight minutes per case. That cost is the reason the pipeline has to be selective before it reaches a human, and it is the reason the 900 cannot all be kept even if storage were free.

What happens to the 900#

The rules below are applied in order, and each one is stated as a rule rather than a judgement so that the same pile processed by a different engineer produces the same output. Two of them lean on the cosine similarity between request embeddings, which is the usual way of asking whether two requests mean the same thing: each request is turned into a vector, and a cosine close to 1 means the two vectors point in nearly the same direction and the requests are near-paraphrases of each other.

Sixty-four cases out of 900 candidates, at roughly 8.5 minutes of expert time each, is nine hours of review for one day's queue, which is still more than most teams can sustain and is why the caps end up tighter than the table suggests after the first month. The proportion is the point: the pipeline's job is to compress a volume signal into a behaviour map, and a compression ratio somewhere between 10:1 and 20:1 is normal on a queue dominated by a small number of live failures.

The 41 in the fifth row deserve attention because they are the ones a naive pipeline poisons itself with. A user who thumbs-down an answer that correctly declines to give legal advice has produced a signal about the product, and writing an eval case that asserts the model should have answered would encode a policy the company does not hold. Sorting those out requires a human, which is another reason the review step cannot be automated away.

The six parked cases in the last row are a category worth having rather than an inconvenience. They are reproducible failures where the correct behaviour is genuinely undecided — should the assistant answer a question about a clause in a document the user has access to but the requesting team does not, and should it say so or stay silent — and forcing an expected outcome onto them under time pressure encodes whichever answer the reviewer preferred that afternoon. Parked cases sit in a queue with the question written out, get resolved when product or legal makes the call, and enter the suite afterwards. A pipeline with no parking lot resolves these by quietly dropping them, which means the hardest questions in the product are the ones the evaluation suite never asks.

Deduplication of the evaluation set is not deletion of the production record. All 900 traces stay in the trace store under whatever retention policy applies to them, and the 400 remain exactly the evidence that this parser problem is large and urgent. The two artifacts answer different questions: the trace store answers how often and to whom, and the eval set answers does the system still behave correctly on this shape of input. If the suite genuinely needs to reflect production frequency — and sometimes it does, when you are estimating a user-facing defect rate rather than checking for regressions — the right mechanism is a weight on the case at scoring time, carried as a field and recomputed monthly from the trace store, so that frequency is a number you can inspect and change rather than a duplication you can only discover by counting rows.

from collections import Counter

NEAR_DUP = 0.92          # cosine on the request embedding
PER_CLUSTER_CAP = 5
PER_TENANT_CLUSTER_CAP = 3

def select(candidates, existing_cases, embed):
    kept = []
    per_cluster = Counter(c.cluster_id for c in existing_cases)
    per_tenant_cluster = Counter((c.tenant_hash, c.cluster_id) for c in existing_cases)
    seen = [embed(c.user_message) for c in existing_cases]

    for cand in sorted(candidates, key=lambda c: -c.severity):
        if per_cluster[cand.cluster_id] >= PER_CLUSTER_CAP:
            continue
        if per_tenant_cluster[(cand.tenant_hash, cand.cluster_id)] >= PER_TENANT_CLUSTER_CAP:
            continue
        v = embed(cand.user_message)
        if any(cosine(v, s) >= NEAR_DUP for s in seen):
            continue
        kept.append(cand)
        seen.append(v)
        per_cluster[cand.cluster_id] += 1
        per_tenant_cluster[(cand.tenant_hash, cand.cluster_id)] += 1
    return kept

The caps are compared against the cases already in the suite rather than against today's batch, which is what stops a cluster from growing by five a day forever. Sorting by severity before applying the caps means the three survivors of a 400-record cluster are the worst three rather than the first three the query happened to return.

Keeping the set honest as it grows#

The nine hours of expert review per day's queue is not a number any team sustains, and what happens in practice is that the caps tighten until the queue fits the reviewer. On this product the arrangement that held was two hours a day from one legal-ops reviewer, which supports about fourteen cases, so the per-cluster cap dropped from five to three and the pipeline began holding candidates from clusters already at cap rather than surfacing them. The consequence is a backlog of candidates that are never reviewed, and the honest way to run that backlog is to expire it — candidates older than thirty days are discarded, because a failure mode that is still occurring will resupply the queue and one that has stopped is not worth a case. Teams that instead accumulate an unbounded candidate queue end up with a number on a dashboard that only ever goes up and that nobody can act on.

Two numbers say whether the suite is still a map rather than a pile. The first is the count of distinct clusters, which is the actual coverage figure; on this product the suite reached 640 cases across 58 clusters. The second is the share of the suite held by its largest cluster, which is the concentration figure, and it is capped at 5% by policy — when a cluster exceeds it, cases are retired from that cluster rather than blocked from entering, so that the constraint holds as the suite changes shape.

Both numbers can be gamed by clustering badly, and this is the honest limitation of the whole approach. Clusters here come from embedding the request and the failure description and running an agglomerative pass — starting with every case in a cluster of its own and repeatedly merging the two closest clusters until the nearest pair is further apart than a tuned threshold — and that threshold determines the coverage number directly: loosen it and coverage looks worse, tighten it and 58 clusters becomes 140 without a single new behaviour being discovered. The defence is to pin the clustering parameters alongside the suite version and to re-cluster only on a deliberate schedule, so that a change in the coverage number always corresponds to a change in the cases rather than in the method.

A third number is worth tracking for a different reason: the rate at which incoming candidates land in clusters that did not exist before. When that rate falls, the system may genuinely be running out of new failure modes, or the sampling may have narrowed onto the traffic you already understand — a pipeline fed only by thumbs-down from a handful of tenants stops discovering anything once those tenants' problems are fixed. Distinguishing the two is what the random labelled sample from chapter 1 is for, since it is the only source of candidates whose selection does not depend on anyone having complained.

The 64 cases that survived yesterday's queue are now sitting somewhere, and where they are sitting is the next problem. Each one contains a customer's question, chunks of a customer's contract, an internal document id, and a free-text comment written by a named individual at a named company. The team's first instinct was to commit them to the repository next to the code, which is convenient, reviewable, and versioned, and which fails every obligation the source log carries.