Mid · 8 chapters
Judge design rules
Binary pass/fail per criterion rather than 1–5 scales (mid-scale scores are noise), one criterion per judge call, reasoning before verdict, a judge model from a different family than the generator, judge cost as a line item
These rules prevent attractive evaluation numbers from hiding inconsistent rubrics, correlated model bias, and an unbudgeted inference bill.
Read these first
What you will know
Use binary pass/fail per criterion instead of a 1–5 scale, and explain why the middle of a scale is noise
Ordinal scores invite inconsistent calibration between runs and judges; binary outcomes make disagreement and threshold changes visible, at the cost of losing fine-grained ranking.
Split a composite rubric into one criterion per judge call
Separate calls let failures be attributed to a specific requirement, although they multiply latency and token spend when a response has many criteria.
Have the judge reason before it verdicts, and say why the order matters
An early verdict can anchor the subsequent rationale, producing post-hoc justification rather than an independent assessment.
Pick a judge model from a different family than the generator
Different training data and tuning reduce correlated stylistic preferences, but they do not remove shared benchmark leakage or other common blind spots.
Budget the judge as a line item rather than an afterthought
Per-criterion calls multiplied across regression suites, retries, and release gates can make evaluation inference a material part of operating cost.
Chapters
- 1
Reading a score histogram
Six hundred outputs scored against a written rubric, and 71% land on a 3 or a 4 while the quarterly mean drifts between 3.6 and 3.9. That clustering is not a quality signal — it is what a judge produces when the middle of the scale has no criterion attached to it, and a histogram is the diagnostic that tells you so before the mean has fooled anybody.
- 2
Binary criteria
Swapping a five-point overall score for pass or fail on each named criterion makes disagreement and threshold changes visible, and makes a failure traceable to a specific requirement. The cost is real and specific: you can no longer rank two answers that pass everything, which is what pairwise comparison is for — and multiplying the all-pass rate by five to get a number out of five is not a summary, it is a fabrication.
- 3
One criterion per call
One judge call returning six pass/fail fields tends to return six fields that agree with each other, because every field after the first is decoded with the preceding verdicts in context and all of them are drawn from a single impression of the answer. Splitting the rubric into one criterion per call separates the correlations that are facts about the answers from the ones the call structure invented — and an instruction to judge independently cannot.
- 4
Output field order
Whether `verdict` or `reasoning` comes first in a judge's output schema is not a style preference. Emitting the verdict first produces it with no intermediate computation, and everything generated afterwards is conditioned on a decision already made — a justification rather than the work behind it. Reasoning first costs latency and tokens and buys a verdict the rationale actually constrains.
- 5
Same-family judges
When the assistant runs on model X and the judge is X's larger sibling, the judge reports 94% pass while blinded human review of the same outputs comes out at 71%. Siblings share preferences over form and style through common training data and tuning, so 'it never wrote these outputs' does not rule the effect out — and three cheap measurements split the gap before anyone spends a month rewriting the rubric.
- 6
Cross-family judges and shared blind spots
Switching the judge to a different vendor removes the part of a judge-versus-human gap that came from siblings rewarding each other's form, and leaves the rest exactly where it was: both models were pretrained on overlapping web corpora, both may have absorbed the same benchmarks, and a claim neither family knows to be wrong is one neither will flag. Knowing which half the swap buys is what lets you price it.
- 7
Costing a judge harness
Five criteria as separate calls, reasoning first, across 400 golden examples, on every push of roughly 25 pull requests a week: fifty thousand judge calls weekly before anyone opens the invoice. Prompt caching, deterministic checks, content-keyed caching and a stratified per-push subset take a $2,993 week to $63 — and each one gives up something specific that is worth naming before it is traded.
- 8
Repairing an inherited harness
A release-gating harness that breaks every rule at once — eight criteria in a single 1-to-5 call, verdict fields ahead of the rationale, judge and generator the same model, no cost line anywhere — cannot be fixed in one branch. The ordering of the changes matters more than the list, a frozen labelled baseline has to come first, and the pass rate falling from 92% to 61% is a measurement rather than a regression.
External resources
- LLM-as-a-judge: a complete guide to using LLMs for evaluationsevidentlyai.com
Full design guidance: binary criteria, single-criterion prompts, reasoning first.
- How to build LLM-as-a-Judge evaluators that hold up in productionarize.com
Concrete judge prompts and the iteration loop to make them stable.
- Binary vs Likert Scales: Why We Use Yes/No Evalsellamind.com
The argument for pass/fail over 1-5 scales, with the failure cases.
- LLM as a Judge Evaluation Guide | Promptfoopromptfoo.dev
This is a substantive, runnable explainer that directly teaches pass/fail rubrics and clear criteria with reasoning, though it does not clearly recommend cross-family judges or treat judge cost as a distinct budget line.
- LLM Evals: Common MistakesHamel Husain
Why Likert scales and multi-criteria judge prompts fail, and what to do instead
- LLM as a Judge Evaluation Guide | Promptfoopromptfoo.dev
This is a substantive, runnable guide that teaches pass/fail rubrics, judge reasoning, calibration, model separation, multi-judge patterns, and cost-related tradeoffs, though it is documentation rather than a video and does not emphasize every listed rule equally.