Diagnosing judge drift

Judge drift is a judge's effective standard shifting over time while its scores continue to be reported as one series. It has two causes that produce identical evidence: the judge changed, or the rubric it applies changed. A frozen anchor set re-scored under a new judge detects both and distinguishes neither, because in each case the same archived outputs receive a different verdict than they originally did.

That ambiguity matters most where a re-score is usually run, which is after something has gone wrong and somebody wants a cause. What follows sets out the explanations that fit, the evidence that separates them, and the two things that come after a diagnosis: an incident to be investigated on its own terms, and a run of past decisions taken against a standard that has since moved.

Five explanations, one observation#

The third row is the one people skip, and it is not a failure mode at all — it is the normal, healthy case. Rubrics get stricter on purpose. If an edit since the archived month added a criterion requiring appeal windows to be stated explicitly, then those outputs failing today is exactly correct behaviour from today's judge, and it is also completely uninformative about whether the old judge did its job, because those outputs were never written against that criterion and the old judge was never asked to check it.

The difference a re-score returns has a sign, and the sign says which instrument is more permissive rather than which one is right. That limitation follows directly from what the re-score computes. It takes two verdict sets over the same outputs and reports how they differ, and a difference between two things says which is larger; deciding which is correct needs a third thing outside both of them. Nothing in the re-score supplies one. This is the same shape as two thermometers disagreeing — the reading tells you which runs hotter and cannot tell you which is calibrated — and the analogy breaks in a way that matters here, because a thermometer has a physical quantity behind it that both are approximating, whereas two rubric versions may be describing genuinely different standards, so there may be no single correct answer for a third instrument to reveal. What a human standard supplies is not the truth about quality but a reading of each rubric text taken independently of the judge applying it. Every row in that table is settled that way.

Recovering the specification when the judge is gone#

The trick that makes this tractable is that the old rubric text is recoverable even when the old judge is not. The prompt lives in version control; git show gets you the text as it read on the day of a release. The model behind it is gone, but the specification is not, and the specification is what humans can apply.

When the old text genuinely is not recoverable — the prompt lived in a notebook, or in a hosted playground, or in a config table that only ever holds the current row — two fossils are usually left. Pull request descriptions and review comments often quote the criterion being changed, which gets you fragments. More usefully, if the judge ever emitted a reason alongside its verdict, those reasons are a record of the criteria it was applying: read forty archived verdicts and the rubric it was working from reassembles itself in outline, including which criterion it cited most and which it never mentioned. That reconstruction is good enough to design the labelling task around and not good enough to quote as the original specification, and the difference should be stated wherever the number is reported.

So take 60 outputs from the archive and label them twice, by two people, blind to both judges' verdicts: once against the old rubric text, once against today's. Then compare four things.

from statistics import NormalDist


def paired_shift(a: list[bool], b: list[bool]) -> tuple[float, float]:
    """Net change in pass rate between two verdict sets over the same items,
    with the standard error from the discordant pairs only."""
    n = len(a)
    a_only = sum(x and not y for x, y in zip(a, b))   # passes under a, fails under b
    b_only = sum(y and not x for x, y in zip(a, b))
    shift = (b_only - a_only) / n
    stderr = (a_only + b_only) ** 0.5 / n
    return shift, stderr


shift, se = paired_shift(human_march_rubric, human_today_rubric)
print(f"{shift:+.1%} ± {1.96 * se:.1%}")   # -15.0% ± 9.8%

Every line of that function is doing one job and the middle two are the ones that look wrong until they are explained. a_only counts the items that pass under the first rubric and fail under the second, b_only the items that move the other way, and the shift is their difference over the number of items — which is just the change in pass rate, arrived at by counting movers rather than by subtracting two rates. The standard error is where the paired design shows up: only a_only and b_only appear in it, so every item that received the same verdict under both rubrics is excluded. Those items are excluded because they carry no information about the size of the change. An output that passes under both texts tells you the two texts agree about it; it does not tell you how far apart they are. All the evidence about the gap lives in the items that moved, so the uncertainty in the gap is the uncertainty in how those items split between the two directions.

On one such run the labels came back as follows. Under the older rubric, the humans passed 52 of the 60 archived outputs, or 87%. Under today's rubric, on the same 60 outputs, they passed 43, or 72% — nine items pass under the old specification and fail under the new one, and none move the other way, which is a 15-point paired shift well outside its own noise. Run the function above on those two label sets and every figure falls out of it: nine movers one way and none the other, so the shift is minus nine over 60, the 15 points; the square root of nine is three, and three over 60 is a standard error of five points, so the band either side is a shade under ten points and does not reach zero.

The old judge had passed 55 of those 60, or 92%; its disagreement with the old-rubric labels is a net three items, five points, with a standard error of 4.4 points, which is not distinguishable from zero at this sample size. That standard error is larger than the net difference it accompanies, and the reason is instructive: a net of three items can sit on top of rather more items that moved in both directions, and the error is set by how many moved rather than by how far apart the two totals ended up. A comparison where five items go one way and two go the other has the same net as one where three go one way and none go back, and it is much less certain. Today's judge passed 74% against the humans' 72% under today's rubric.

Read together, those four numbers say something quite specific: both judges were roughly faithful to the rubric they were given, and the rubric moved by about fifteen points' worth of standard. Each judge was compared against people applying the same text it was applying, and each came within a few points of them, which is the only comparison that can convict a judge of anything. Set either judge against the other rubric's labels and it looks badly wrong, which is the shape a drift investigation mistakes for a finding. Two very different pass rates are both approximately correct measurements — of different things. The comfortable conclusion, that the old judge was lenient, is the one explanation the evidence actively argues against, and its comfort is the reason it needs the most evidence rather than the least.

The escalation is a separate question#

Letting a large re-score gap absorb an incident is the most common way both get mishandled, and the reason it happens is that the two investigations arrive at the same moment and one of them has a number attached. A drift finding is quantified, dated and satisfying; a specification gap is a sentence saying nobody thought of this. The first will be volunteered as the explanation for the second unless somebody checks whether it can be. Check whether either rubric version has a criterion that covers the behaviour that caused the escalation. If today's does and the older one did not, the escalation is a specification gap that has since been closed, and the honest account is that at the time nobody had thought to check for this. If neither version covers it, the drift finding is irrelevant to the escalation entirely and the judge is not the story — the rubric never encoded the thing that hurt the customer, and no amount of recalibration will find the next one like it. That case is common and it is the one where a drift investigation is most likely to be used as a decoy.

Either way, the escalated output becomes a permanent eval case with a hand label attached, and it joins the anchor set so that every future judge change is scored against it. An incident that does not end with a new case in the eval suite has been resolved rather than learned from.

The releases decided in between#

Between March and August, five releases went through the same gate. Each of those decisions used a number produced by whichever judge existed that week, and the versions were not recorded, so the series is four short series in a trench coat.

The repair is mechanical and takes a day. Pin one judge version — today's is the defensible choice, since it is the one you can still validate against humans — and re-score the archived outputs behind all five gate decisions under it.

Read that table for what it can support. The right-hand column answers "does this release meet the standard we hold today", which is a live product question about code currently in production, and three releases fail it. It does not answer "was the decision to ship wrong at the time", because the standard the decision was taken against is the one in the left-hand column and that standard has since moved fifteen points' worth. Presenting the right-hand column as a list of past mistakes repeats the original error in the opposite direction, and the people who made those calls will notice.

The two columns also converge as the dates approach today, which is a check that the pinned re-score is behaving. March loses seventeen points, April eight, June nine, July four, and August none at all — because August's gate score was produced by a judge close enough to today's that re-scoring changes almost nothing. A table where the gap failed to close toward the present would be evidence of something other than accumulated rubric edits, most likely a serving model that moved under an alias, which is the fourth row of the table of explanations above.

Publish two artifacts: the corrected series with version boundaries drawn on the chart, and a short list of the releases whose shipped behaviour no longer meets the current bar, framed as work to do rather than as judgement of the past. The June feature-flag rollout is the one that needs an actual decision, because it is still enabled, it was marginal even against the old bar, and the appeal-window behaviour that produced the escalation lives inside it. Resist the urge to restate the history quietly: a corrected chart that appears without commentary invites the team to assume the old numbers were lies, which is both wrong and corrosive. The old numbers were measurements taken with an instrument that changed without anyone recording the change, which is a process failure rather than a credibility failure, and saying so is what keeps the next person willing to run the re-score that produced the correction.

The controls that were missing#

The honest answer to "Should this have been caught?" names the specific missing control rather than describing the incident as unfortunate. A frozen anchor set re-scored on every judge change would have surfaced the fifteen-point standard shift the week it landed, for the price of a few dollars and six minutes of continuous integration. A judge version stamped on every score would have made the four-way split in the series visible in any query anyone ran. Both were skipped because a judge prompt did not look like the kind of thing that needs versioning, and the cost of that judgement is one un-audited rollout and a customer escalation nobody could date.

What changes is a short list, and it is the same list whether you are recovering from an incident or setting a judge up for the first time. Archive inputs, outputs, and every field the judge saw, from today, because everything else in this topic depends on being able to re-run a verdict. Freeze an anchor set now and add a new one each quarter without retiring the old. Stamp the assembled-prompt hash, the model identifier and the harness version onto every score you write. Run the anchor re-score in continuous integration on any change that alters that hash, and require a human to read the flipped outputs. Spend fifty stratified hand labels a quarter validating the judge against people, and measure the human-human ceiling once so you know what a good agreement number looks like on your rubric. Run the presentation swap on every pairwise comparison and publish the flip rate beside the margin. And when a probe for verbosity or self-preference returns a number, write it down next to the metric it distorts, because the correction is not applied by having measured it.

Drift runs in both directions, and the direction a re-score reports is a fact about permissiveness rather than about correctness. The evidence that resolves it is cheap and specific: the old rubric text out of version control, sixty archived outputs, two labellers, and four numbers instead of two. The most likely finding is the unglamorous one — both judges did what they were told and the instructions changed underneath them — which is repaired by recording version boundaries rather than by recalibrating anything.

None of the practices in this topic make a judge correct. They make a judge accountable, which is what lets a number survive contact with a quarterly review, an incident, and the person who asks five months later what exactly you were measuring. All of it assumes the judge's inputs are what you think they are — the product's output and the evidence about it, and nothing else. The next topic takes that assumption apart.