A year-old suite tests the problems you had a year ago

Four quarterly reports in a row put the suite's weighted pass rate between 0.86 and 0.89. Over the same twelve months the product added a document-upload flow, a Slack integration and a Spanish-language rollout; conversation length went from a median of 3 turns to 9; and the assistant's CSAT, the average of the satisfaction ratings users leave on a five-point scale after an interaction, fell from 4.4 to 3.8. The suite recorded none of that, because a coverage audit run in the thirteenth month found that of 112 cases, 71 had been written in the first six weeks of the suite's life, the median case age was 14 months, and the three most common support complaint categories of the last quarter mapped to zero cases between them.

The number was not wrong. Weighted pass rate over those 112 cases really was 0.87, measured correctly, with an honest scorer. It described a product that no longer existed, and its stability — the thing that made it feel trustworthy in the quarterly report — was the clearest available evidence that it had stopped tracking anything.

Comparison suites have a freeze incentive that product datasets do not#

There is a general version of this problem and a specific one. The general version is familiar from building a golden dataset: any fixed set of cases drifts away from a moving product, and the answer is additive refresh with versioning and a re-baseline so that historical numbers stay interpretable. All of that applies here unchanged and is not worth restating.

The specific version is that a suite built for cross-model comparison has a structural reason to resist change that a product-quality dataset does not. Its whole job is to make a model released next year comparable to one released last year, and every case you add breaks the direct comparability of the numbers on either side of the addition. That creates a quiet, reasonable-sounding argument against every proposed addition — if we change the set now, we lose the trend — which is correct in isolation and, applied consistently, produces exactly the four flat quarters above. The argument is strongest for the people who care most about measurement quality, which is why suites maintained by careful engineers rot in this particular way more often than sloppy ones do.

Resolving it requires being explicit that the suite serves two purposes with different time constants. Comparability across models is served by holding a set fixed within a comparison, which is a matter of running both systems on the same version on the same day rather than never adding cases. Currency is served by adding cases, and the cost of adding them is one re-baseline run of the incumbent — eleven minutes and about $2.40 at this suite's size. Stated that way, the trade is between a trivial recurring cost and an instrument that goes blind, and it stops being a close call.

Intake with quotas, not intake when someone complains#

The previous chapter's route into the suite was three annoyed engineers in a standup. That works and cannot be depended on, because it fires only when a disagreement is loud enough to reach a meeting. A standing intake makes case addition a scheduled activity with a quota, which is what gets cases written for failures nobody found dramatic.

Stratifying the traffic sample by intent share means drawing the eight cases so that each kind of request contributes in proportion to how often it actually occurs, which is what stops a month's intake from being eight variations on whichever flow the person doing the sampling happens to work on. Quotas are small on purpose. Eight sampled cases and six escalation cases a month is 168 cases a year against 112 existing, which is more churn than the suite can absorb without the retirement column doing real work, and the retirement quota is the one people skip. Twelve cases in this suite tested a v1 upload flow removed in March; they passed on every system, contributed nothing to any comparison, and inflated every pass rate by about a point.

def coverage_audit(cases, traffic_90d):
    """Which of today's traffic does the suite have any case for?"""
    covered = 0.0
    for intent, share in traffic_90d.items():
        n = sum(1 for c in cases if c.intent == intent and not c.retired)
        if n >= 3:
            covered += share
        else:
            print(f"GAP {intent}: {share:.1%} of traffic, {n} cases")
    print(f"covered: {covered:.1%} of traffic")

# month 13:
# GAP document_upload:   19.4% of traffic, 0 cases
# GAP multi_turn_ops:    22.1% of traffic, 1 case
# GAP es_ES_any_intent:  11.8% of traffic, 0 cases
# covered: 41.3% of traffic

A suite covering 41% of production traffic is producing a number about a minority of what the system does, weighted as though it were everything. Running that audit takes an afternoon to write and thirty seconds a month thereafter, and it converts the vague sense that the suite might be stale into three named gaps with traffic shares attached, which is the form a gap has to be in before anyone will schedule work against it.

Telling a disciplined streak from a rotted bar#

Three quarters of rejections is the situation where this matters most, because the two explanations look identical from inside. Either the releases genuinely did not clear a bar that still describes the product, or the bar stopped describing the product and is now rejecting everything by construction. Five diagnostics separate them, and none of them requires deciding in advance which answer you want.

The first is the distribution of rejection reasons. Five candidates rejected across three quarters, each failing a different gate, is a bar with several live constraints. Five candidates all rejected on the same gate — say, all five on the 3-second TTFT ceiling — means one number is doing all the work, and that number deserves an audit against the current product rather than another application.

The second is case age. If no case has been added in two quarters while the product shipped features, the suite is measuring an older product, and the rejections are about that older product. The month-13 audit above answers this in one line.

The third is a counterfactual re-run, and it is the most informative of the five. Take the model you were using a year ago, run it against today's suite version, and look at what it scores. If the year-old model now fails cases it used to pass, the bar has moved with the product and the streak is discipline. If the year-old model still passes everything, the bar has not moved in a year, and a bar that has not moved while the product changed is rejecting candidates for failing to improve on a target nobody is aiming at any more.

suite@v14 (today's cases), all systems re-run 2026-08-03:
  model-2025-06 (a year ago)   0.612    was 0.874 on suite@v6
  model-2025-11 (incumbent)    0.831
  candidate-2026-07            0.848    delta +1.7 pts  (band +/-3.5)

That output is a healthy result. The year-old model loses 26 points against today's cases, which means the suite has genuinely become a harder and different test, so the incumbent's lead is real and the candidate's +1.7 is a real non-result rather than an artefact of a frozen bar. Had the first row read 0.869, the correct conclusion would have been that three quarters of rejections say more about the suite than about the models.

The counterfactual has a practical obstacle, which is that year-old models get deprecated and the endpoint you want to re-run against stops existing. The defence is to store outputs, not just verdicts. Every run writes the full generated text into the results table alongside the timing and cost, so a model that was retired in March can still be re-scored in August against cases written in July, provided the cases' inputs are among the ones it was run on. That works for the case set as it stood at the time and fails for cases added afterwards, which limits the counterfactual to the overlap. Keeping a frozen snapshot of a small open-weights model — one whose parameters you can download and run on your own hardware, so no vendor can retire it out from under you — as a permanent bottom-of-the-range reference covers the rest, since it can be re-run against anything forever and gives every version of the suite a fixed low anchor. Storage for all of this is trivial — 400 runs of 90 cases at a few hundred tokens each is well under a gigabyte — and it is the sort of thing that is free to start doing today and impossible to backfill.

Retirement needs a process too, because the twelve upload-flow cases were not removed by anyone deciding to remove them; they were removed when a monthly pass asked which cases referenced a feature flag that no longer exists in the codebase. That query is a join between case metadata and the flag registry, it takes seconds, and it found four more cases pointing at an endpoint deleted in January. Retired cases move to an archive with the date and reason rather than being deleted, which is what allows an old score to be reconstructed later — a pass rate of 0.87 on a version containing twelve free passes is a different number from 0.87 on one without them, and knowing which you are looking at requires the archive.

The fourth is external calibration. If a candidate has been adopted widely by teams doing similar work, and your suite says it is worse, your suite should be able to explain the discrepancy in specific terms — the multi-turn coverage gap from the previous chapter is that kind of explanation. An unexplained divergence from everyone else is not proof you are wrong, and it is a strong prompt to go and look.

The fifth is the production signal the suite is supposed to predict. A suite whose score is flat while CSAT falls six points, or while escalation rate rises, has broken the correlation that made it worth running. Checking that link once a quarter costs a query and is the only diagnostic that catches the case where every case is current and the scorer is measuring the wrong property.

What refreshing does not fix#

Adding cases does not make the suite representative in any statistical sense, and it should not be described that way. Even at 250 cases with a monthly intake, this instrument is a curated sample weighted deliberately toward past failures, which is the right design for comparing systems and the wrong one for estimating product quality. The two artefacts stay separate, and the temptation to merge them — since both contain cases and both produce a pass rate — leads to a set too slow to run on release day and too skewed to trust as an estimate.

Refreshing also does not fix a scorer that has drifted. Cases added in month 13 with a judge prompt written in month 1 inherit whatever that judge does badly, and the judge's own version history has to be re-baselined on the same schedule as the cases.

The larger thing it does not fix is that all five diagnostics need history. The rejection-reason distribution needs the rejections written down with their reasons; the counterfactual needs last year's model still runnable and last year's scores still stored; the external calibration needs a record of what you read and when. Those records do not exist by default, and reconstructing them from memory in the thirteenth month produces the version of events that most flatters whoever is reconstructing them.