Human evaluation

Read these first

• Building a golden dataset

What you will know

• Write annotation guidelines precise enough for two people to agree

  Operational criteria, examples, edge-case rules, and tie-breaking procedures prevent hidden interpretation differences.

• Measure agreement between reviewers and act on a low number

  A low kappa or alpha may reflect class prevalence, ambiguous labels, or sampling effects, so adjudication should find the cause before relabeling.

• Decide which tasks cannot be judged by a model at all

  Automated judges can miss subtle safety, factuality, and policy failures, especially when the evaluator shares the system’s biases.

• Budget expert review time as real project cost

  Review capacity controls throughput and release cadence; unplanned specialist queues turn evaluation into schedule risk.

• Turn human labels into the ground truth a judge is validated against

  Without a held-out, adjudicated reference set, judge scores can look precise while tracking annotator artifacts rather than user outcomes.

Chapters

1. 1

   Writing an annotation guideline

   An annotation guideline that leaves a case undefined is not one instrument with noisy readings, it is two instruments sharing a scale — which is why two reviewers scoring the same 200 support answers on a bare 1-to-5 helpfulness scale landed on the same number 41% of the time. Covered here: writing each level as a condition a reviewer can check, anchoring it with a real output, converting every pilot disagreement into a rule, and stating a precedence order for when two criteria pull in opposite directions.

   Read chapter
2. 2

   Cohen's kappa and raw agreement

   Two reviewers label 1,000 outputs for policy violations, agree on 96% of them, and Cohen's kappa comes back at 0.14 — which is not a bug in the implementation. Violations are rare, so agreement earned by both defaulting to 'clean' is nearly free, and kappa subtracts precisely that baseline. Covered here: what chance correction measures that raw agreement conceals, where the coefficient and its interpretive labels came from, and why the gap between the two numbers is the finding rather than a technicality.

   Read chapter
3. 3

   Positive specific agreement

   Kappa tells you the agreement is thin; it does not tell you what to ask for next. Positive specific agreement does: the share of the flags two annotators share, computed on the positive class alone, reported as a fraction anybody in a meeting can read. It is the number to demand beside a 93%-with-kappa-0.21 pair. Covered here are the formula, how many positives a sample has to contain before the estimate holds still, which coefficient answers which question when there are more than two annotators, and the threshold values that turn a reference set from shippable into not.

   Read chapter
4. 4

   Diagnosing annotator disagreement

   The items two annotators disagreed on are the most informative subset of a labelled set, and the instinct to bin them and relabel with better-trained people throws that information away. Covered here: pulling the contested items out as a batch, slicing them by content, structure, annotator and time, reading which of four signatures the pattern matches, and the schema change, sampling fix or re-anchoring that each signature implies — plus the cheap re-pilot that says whether the diagnosis was right before the full relabel is paid for.

   Read chapter
5. 5

   The limits of LLM evaluation

   Substituting an automated judge for human reviewers is safest on form and least safe on truth: a judge sharing the generator's training data shares the blind spot that makes a false domain claim read as fine. Covered here: the independence property that decides which review tracks can be handed over, why the safest tracks should have been code rather than a judge, why a failure whose signature is missing content is invisible to any judge, and why a random 5% audit of a rare high-severity miss reports clean by construction.

   Read chapter
6. 6

   Validating an LLM judge

   Tuning a judge prompt against 300 labelled items until it agrees 91% of the time, then reporting the 91%, measures the fit rather than the accuracy — and the figure was never placed beside its ceiling, which is how often the two humans agreed with each other on the same items. Covered here: holding out a reference set before touching the prompt, adjudicating it, quoting the human-human rate beside every judge number, reporting recall rather than agreement when a judge gates a rare failure, and re-validating when the product model is swapped.

   Read chapter
7. 7

   Planning around reviewer capacity

   One clinician, four hours a week, roughly 150 outputs per release on a two-week cadence: that arithmetic sets the evaluation plan whether or not anybody writes it down. Covered here: measuring realised capacity in minutes rather than items, costing a sampling plan against it, spending scarce hours on mandatory strata and judge-flagged items instead of a uniform subsample, putting the constraint on the schedule where it is visible before a release date, and what a second reviewer actually costs.

   Read chapter