Validating an LLM judge

An LLM judge is a model prompted with a criterion and asked to score another model's output — the automated stand-in for a human reviewer. Like any measurement instrument it has a configuration, a population it was calibrated against, and an accuracy that belongs to the pair rather than to the instrument alone.

This chapter is about the two failures that make a judge's headline agreement figure uninterpretable, both independent of how good the judge is. The first is that the number was produced by iterating against the same items it is reported on, which is fitting. The second is that it was never placed beside the agreement between the humans whose labels it is scored against, which is the highest score any measurement of the same construct can reach.

Ninety-one percent measured the fit#

Take a judge built the way most of them are built. Two annotators label 300 outputs of a retrieval assistant for "is this answer supported by the cited passage", and a team runs a judge prompt over those 300 items, revises where it differed from the labels, and repeats until agreement reaches 91%.

The prompt has parameters — the phrasing of the criterion, the few-shot examples included, which are labelled items pasted into the prompt as worked demonstrations, the tie-break instruction, the output format — and twelve revisions is twelve gradient steps taken by hand against a set of 300 items. Reconstructing what the revisions did produces this trajectory:

The held-out column was computed afterwards, on items the prompt had never been shown. The first three revisions generalise, because each one encoded a real distinction that recurs in the data. The last two do not, and v12 is worse on new items than v9 was, because the three examples added at the end were selected from the residual errors on the tuning set and taught the judge about those particular items. A judge shipped on the strength of the left-hand column is 15 points worse than its headline in production, and nothing in the process could have surfaced that.

The mechanism that turns a sensible-looking loop into a fitting procedure has three steps and none of them involves anybody doing anything wrong. First, each revision is chosen by looking at where the judge currently disagrees with the labels, which means the evidence for every change comes from those 300 items and no others. Second, two very different kinds of change look identical from inside that view: a rule that captures a real distinction recurring throughout the data, and a rule that happens to resolve the particular items in front of you. Both raise the number on the tuning set by the same amount, so the process has no signal that separates them and accepts both. Third, only the first kind raises the number anywhere else. Twelve revisions accumulate a mixture of the two, the tuning column climbs on both and the held-out column climbs on one, and the distance between the columns is the share of the work being done by rules that fit these items.

The last two revisions are where the mixture tips, and the reason is in how the examples were chosen. By revision nine the judge is already right on most of the set, so the errors that remain are the ones the earlier, general rules failed to cover. That residue is not a random sample of the task — it is selected for being unlike the rest of it, which is exactly the property that makes an example teach something that does not transfer. So the highest-risk move in the whole procedure is the one that feels most productive: taking the items you are still getting wrong and showing them to the model. Reading the two columns across v9 and v12 says so plainly. Two more points were gained on the tuning set and six were lost on new items, which is a trade nobody would have accepted had the right-hand column existed at the time.

The discipline that prevents this is the same one used for any fitted artifact. Split the labelled set before touching the prompt, work only against the development slice, and open the held-out slice once, at the end, to produce the number that goes in the report. If you go back and revise after seeing the held-out result, that slice is now a development slice and you need a new one.

import random

def split_reference(items, dev_frac=0.5, seed=20260401):
    rng = random.Random(seed)
    pool = sorted(items, key=lambda x: x["id"])
    rng.shuffle(pool)
    cut = int(len(pool) * dev_frac)
    return {"dev": pool[:cut], "holdout": pool[cut:]}

Splitting 300 items in half leaves 150 for tuning and 150 for validation, which is a thin holdout when the interesting class is rare — 150 items at 12% prevalence contains 18 positives, and a judge's recall on 18 positives has a 95% interval roughly 20 points wide. The reference set has to be sized for the holdout, not for the tuning, so the practical figure is a set with at least 100 positives in the held-out half.

Every agreement number needs its ceiling printed beside it#

The second flaw is independent of the first and would remain even with a clean holdout. An agreement figure between a judge and a set of human labels has no interpretation until you know how well the humans agreed with each other on those same items.

On the 300 items, the two annotators had agreed with each other on 88% of them. A judge reporting 91% agreement against labels whose own consistency is 88% has not beaten the humans; it has fitted the resolution procedure that turned two disagreeing labels into one. If the pipeline resolved ties by taking annotator A's label, the judge learned to imitate annotator A, including the 6% of items where A was wrong.

Why that is a ceiling rather than a coincidence takes one step to see, and the step is about what the judge is being scored against. The reference labels are not the truth; they are one annotator's opinion on the items where two annotators differed, and both annotators' shared opinion on the rest. On the twelve items in every hundred where the pair disagreed, a decision had to be made about which label went into the reference, and whatever rule made it — take A, take the more experienced reviewer, take the first one entered — is now part of the yardstick. A judge scoring highly on those items is a judge that has learned to reproduce that rule. It gets no credit for being right when the rule was wrong, and it is penalised for it. So a judge which agreed with the truth perfectly would score, on this reference set, somewhere close to the rate at which the reference agrees with the truth, and the humans' agreement with each other is the best available estimate of how often that is. Exceeding it is not an achievement to celebrate. It is a signal that the judge has fitted the tie-break, and the fitting is the thing the split-and-hold-out discipline is there to expose.

The legal-clause extraction product from chapter 5, on the limits of automated evaluation, shows the same problem in a form that is easier to miss, because there the raw numbers look reassuring:

The 94% that a substitution proposal rests on is two points below the human ceiling on raw agreement, which sounds like a rounding error, and 0.28 of kappa below it, which is most of the distance to nothing.

Those two gaps describe the same pair of rows and differ by more than an order of magnitude, which is the strangest thing in the table and the most useful. The reason is the chapter-two mechanism applied twice over. Raw agreement on a clause-typing task is mostly the free baseline: both parties assign the common types to the common cases, and that costs no discrimination. Kappa strips that baseline off and reports what is left, and what is left is a much shorter scale — so the same two points of raw difference, measured against a ceiling of a few points rather than a hundred, become a large fraction of everything that was available. The practical instruction is to distrust any comparison of two agreement figures that are both near the top of the range, because the visible distance between them understates the distance between the things they measure by whatever factor the free baseline sets.

The bottom two rows are where the substitution decision actually lives, and they only exist because the reference set kept the rare types and the adjudicated contested items tagged as separate strata. Read them as a sequence rather than as three separate numbers. Kappa goes 0.58 across all clause types, 0.31 on the six rare ones, 0.24 on the contested items — falling by roughly half from the headline to the hard tail, while raw agreement over the same three rows falls by about a third of its distance to nothing. A single averaged figure conceals that gradient completely, and the gradient is the decision: nobody is proposing to automate the easy majority and worry about the tail, but that is what the 94% is silently describing.

What a judge validation record contains#

Instruments carry calibration certificates that say what was measured, against what, when, and under which configuration. Storing that record beside the judge is what makes the number auditable six months later, when the person who ran the validation has moved teams.

{
  "judge_id": "grounding_v4",
  "judge_model": "claude-sonnet-4-6-20260214",
  "prompt_sha": "b71f2ac",
  "temperature": 0,
  "reference_set": {
    "id": "grounding_holdout_2026q1",
    "n": 420,
    "positives": 118,
    "guideline_version": 2,
    "adjudicated": true,
    "adjudicator": "legal-ops-lead",
    "strata": ["common_types", "rare_types", "contested_v1"]
  },
  "human_ceiling": {"raw": 0.96, "kappa": 0.86, "positive_specific": 0.79},
  "judge_scores": {
    "overall": {"raw": 0.94, "kappa": 0.58},
    "rare_types": {"raw": 0.91, "kappa": 0.31},
    "contested_v1": {"raw": 0.62, "kappa": 0.24}
  },
  "product_model_at_validation": "internal-extractor-3.2",
  "validated_at": "2026-03-11",
  "expires_on_change_of": ["product_model", "prompt_sha", "guideline_version"]
}

The reference set that this record points at is not the raw output of two annotators. It has been adjudicated, in the sense established in chapter 4, on diagnosing a disagreement set: every item where the two disagreed was resolved by someone with authority over the definition, with a one-line reason recorded, and the items keep a flag saying they were contested. Unadjudicated labels put the pipeline's tie-break rule into the ground truth, and the judge then gets scored on how well it reproduces a tie-break rule.

The positive_specific field in the human ceiling is there because of what chapter 3, on positive-class agreement, established: on a skewed label the overall figures hide the only comparison that matters. A reference set whose annotators shared a quarter of their positive labels gives a judge a ceiling so low that no judge score computed against it means anything, which is the reason not to ship such a set as ground truth in the first place.

Agreement is not the number you report to the business#

A judge can agree with humans 94% of the time and still misreport the metric the team actually tracks, because agreement is symmetric and error is not. Take 1,000 outputs where the true unsupported rate is 12%, so 120 bad answers. Suppose the judge misses 45 of those 120 and falsely flags 15 of the 880 good ones. Total disagreement is 60 items, so agreement is 94%. The judge reports 90 unsupported answers, which is a 9.0% failure rate against a true 12.0%, understating the thing the release gate is checking by a quarter.

Work the two counts separately, because the reason one number hides inside the other is entirely in how they combine. Agreement adds the two kinds of mistake together: 45 misses plus 15 false alarms is 60 items on which the judge and the labels differ, and 60 out of a thousand is six in a hundred, so agreement is 94% regardless of which kind those 60 were. The reported rate subtracts them. Everything the judge flags is either a real failure it caught — 120 minus the 45 it missed, which is 75 — or a good answer it flagged wrongly, of which there are 15. Seventy-five plus fifteen is 90, and 90 out of a thousand is the 9.0% the code prints. So the two error types cancel in the count the gate reads and accumulate in the count the report quotes. A judge with 45 misses and 45 false alarms would have worse agreement and report the true rate exactly, and a judge with no false alarms at all and the same 45 misses would have better agreement and understate by more. Agreement is not a weaker version of the right number. It is a number that moves in a different direction. What a gate on a rare failure needs is a figure with no cancellation in it — the real failures caught over the real failures that existed, with the false alarms kept out of it entirely.

def reported_vs_true(n, true_pos, fn, fp):
    flagged = (true_pos - fn) + fp
    return {
        "agreement":     1 - (fn + fp) / n,
        "true_rate":     true_pos / n,
        "reported_rate": flagged / n,
        "recall":        (true_pos - fn) / true_pos,
    }

print(reported_vs_true(n=1000, true_pos=120, fn=45, fp=15))
# {'agreement': 0.94, 'true_rate': 0.12, 'reported_rate': 0.09, 'recall': 0.625}

Recall of 0.625 is the sentence that belongs in the release report, and it does not appear anywhere in "94% agreement". Whenever a judge feeds a gate on a rare failure, publish its recall and precision on the positive class from the held-out set — recall being the share of the real failures it flags, precision the share of its flags that are real failures — and treat the raw agreement figure as a sanity check rather than as the result.

Validation expires when the thing being measured changes#

Keeping a judge honest across a model swap is where the validation record earns its cost. When the extraction model behind the legal product was swapped, judge-versus-paralegal kappa on the standing held-out set fell from 0.58 to 0.41 within a week. The cause was mundane: the new model phrased uncertain extractions with hedging language, and the judge's prompt contained an example that treated assertive phrasing as a signal of a well-grounded clause. The generator changed; the instrument did not; the number quietly moved.

Notice what that episode says about what a validation figure is a claim about, because the intuitive reading — the judge got worse — is the wrong one and leads to the wrong repair. Nothing about the judge changed. Its prompt, its model and its temperature were identical on the day the number fell and the day before. What changed is the population it was measuring, and the judge's accuracy was never a property of the judge alone: it was a property of the judge applied to the outputs of a particular generator. A worked example that treats assertive phrasing as evidence of grounding is a good rule against a generator that hedges only when it is genuinely unsure, and a bad one against a generator that hedges as a matter of style. Neither the rule nor the judge is wrong in isolation. The pairing is. That is why the trigger table below is keyed on changes to the product model as well as to the judge, and why a validation record that does not name the product model it was measured against is recording an incomplete claim.

Three mechanisms hold that table up. The validation job re-runs the judge over the frozen held-out set on every change to the product model, the judge model, the judge prompt, or the guideline version, and blocks the deploy when kappa falls more than 0.05 below the recorded baseline. The held-out set is refreshed quarterly by adjudicating a fresh sample from recent production traffic and retiring a slice of the old one, because a set frozen for two years stops resembling the traffic the judge sees. And a small standing human sample continues in production regardless of judge scores, which is where a failure mode nobody has written a criterion for gets discovered.

The ceiling the whole arrangement inherits#

A judge validated this way can be trusted to reproduce human labels on the distribution the reference set covers, and it cannot be better than those labels. If the reference set's human ceiling on the positive class is 0.79, no configuration of prompt and model gets credited above it, even one that is genuinely more accurate. That limitation is real and only partly addressable: sending the judge-versus-human disagreements back to an expert for blind re-adjudication will occasionally show the judge was right and the annotator wrong, which raises the ceiling honestly, and it costs expert hours per disagreement. LLM-as-a-judge vs. human evaluation: Why together is better is a reasonable survey of how teams combine the two rather than choosing.

The last row of the trigger table matters as much as the gate above it. When judge agreement drops, the first response is often to edit the judge prompt until it comes back, which is the same fitting mistake at a different level. The drop is a disagreement set, and it gets diagnosed the way any disagreement set does.

The number attached to a judge is therefore a claim about an instrument, a reference set, a guideline version and a product model considered together. Split the labels before the prompt is touched, adjudicate them so the figure is not a score against a tie-break rule, and print the human ceiling and the positive-class recall beside it so it can be read.

Every mechanism in this chapter — the quarterly refresh, the re-adjudication of disagreements, the standing production sample, the strata that keep rare types visible — draws on the same finite resource, which is the time of the people qualified to make the judgement. On the legal product that resource is a paralegal; on a clinical product it is a clinician, and there is one of her. Chapter 7, on planning an evaluation around reviewer capacity, designs the programme backwards from that constraint.