LLM-as-judge

Read these first

• Building a golden dataset

What you will know

• Build a judge that scores an output against a rubric

  Probe it with deliberately flawed answers and inspect whether scores track the rubric rather than fluency, verbosity, or agreement with the judge’s own preferences.

• Choose between pointwise and pairwise, and justify it

  Account for scale drift, position bias, ties, and the extra inference cost that pairwise comparisons introduce.

• Write a rubric another person would apply the same way

  Use observable pass and fail conditions with boundary examples, so independent graders can reproduce decisions instead of relying on taste.

• Compare reference-guided and reference-free grading on outputs with valid alternative answers

  A reference can improve factual anchoring while unfairly penalizing correct answers that use a different method or wording.

• Calibrate the judge against human labels and reproduce its failure cases on a held-out set

  Agreement on easy examples is insufficient; disagreements, prompt sensitivity, and systematic severity or leniency reveal whether the score is trustworthy.

Chapters

1. 1

   Rubric reliability

   Two colleagues hand-grading 40 outputs against a 5-point helpfulness scale agree on 24, and that number is a verdict on the rubric rather than on them. Replace scale points with observable conditions a grader can check, attach a borderline example showing pass and fail for each, and re-measure — because a judge applying an ambiguous rubric consistently is reproducible and still invalid.

   Read chapter
2. 2

   Probe sets for judge bias

   A probe set is a handful of constructed pairs whose rubric-correct verdict you know because you built them, differing in exactly one property the rubric does not mention. Handing a judge a polished wrong answer beside a blunt right one costs pennies and tells you whether it is scoring your criteria or the writing — and it is the diagnostic most often skipped, because an unprobed judge looks fine.

   Read chapter
3. 3

   Pointwise and pairwise judging

   Scoring one output at a time against a fixed scale gives you a level that drifts; showing a judge two answers and asking which is better gives you a direction that does not. The swap that makes head-to-head honest doubles the inference cost and buys immunity to scale drift — and a win rate still cannot say whether either system is good enough to ship.

   Read chapter
4. 4

   Position bias and tie policy

   Running every pair twice with the answers swapped leaves two verdicts per input, and the pairs that disagree are exactly what the swap was designed to expose. Four defensible ways of counting them produce headline win rates twelve points apart, so the policy has to be pinned before the run — along with whether the judge is offered an explicit tie at all, which changes its behaviour rather than the arithmetic.

   Read chapter
5. 5

   Anchoring a win-rate series

   Making win rate against the previous release the permanent metric builds a chain of comparisons with no fixed point, and nothing in the series can detect gradual absolute decay because the baseline moves with you. A frozen baseline, a length-matched subset and a rotation with an overlap window are what turn four cheerful numbers into a trend somebody can read.

   Read chapter
6. 6

   Reference-guided grading

   Handing a judge a known-good answer alongside the candidate anchors the semantics the question left unstated — and smuggles in one particular implementation as though it were required. The rubric has to say which dimensions of divergence are permitted, execution should decide row equality wherever a warehouse is available, and dropping the reference trades a countable false-fail rate for an uncountable false-pass one.

   Read chapter
7. 7

   Auditing false fails

   A suite reporting 68% pass hides a correction you can compute. Hand-audit a random sample of the failures, find that 11 of 50 returned exactly the reference's rows, and scale that rate back across the whole failure set — roughly seven points, putting the real pass rate near 75% with an interval as wide as the correction itself. The same arithmetic applied only to failures is the most common way it gets misused.

   Read chapter
8. 8

   Judge agreement and base rates

   A judge that passed everything unconditionally would score 85% on a sample drawn from production, so headline agreement mostly measures the easy majority. What decides whether a judge can gate anything is its agreement on borderline cases and whether its errors lean one way — because a consistently lenient judge reports a pass rate that rises exactly when quality falls.

   Read chapter
9. 9

   Judge versioning

   A quarterly mean going 3.8 to 4.1 across a judge-prompt change compares two instruments rather than two quarters. Re-score the stored outputs with the current prompt and the difference is the only movement the product earned — which is why every score carries a fingerprint of everything that can change a verdict, and why a floating model alias is the one change no fingerprint catches.

   Read chapter