Labeling and annotation operations

Read these first

• Human evaluation

What you will know

• Write annotation guidelines two people apply consistently

  Include boundary cases, examples, escalation rules, and an adjudication path; otherwise reviewers silently invent different labels.

• Measure and act on reviewer agreement

  Break agreement down by class and example type, since an acceptable aggregate score can conceal systematic disagreement on rare but important cases.

• Budget annotation as funded work with a named owner

  Estimate volume, reviewer throughput, adjudication overhead, and rework, then assign accountability for missed quality or delivery targets.

• Use active learning to label the examples that matter most

  Compare the selection strategy with random or production-like sampling and verify that it improves the target metric rather than merely choosing uncertain noise.

• Run a labeling pilot before scaling the operation

  Use a small representative batch to measure time per item, defect rates, agreement, and cost before committing to the full dataset.

Chapters

1. 1

   A guideline is written out of the disagreements it resolves

   Two reviewers labelling support tickets as billing, technical or account agree on 91%, and nearly every disagreement is one case: a customer locked out because their subscription lapsed. The document needs that exact ticket worked through, an explicit precedence rule for when two categories both apply, a named adjudicator, and a path that writes each ruling back as the next example. Without those, reviewers quietly invent their own labels.

   Read chapter
2. 2

   Ninety-five percent agreement and a kappa of 0.31 are the same data

   On 1,000 transcripts with a binary flag that is 96% negative, raw agreement of 95% and Cohen's kappa of 0.31 are both true — two reviewers blindly labelling everything negative would score 96% raw and near-zero kappa. The same skew operates inside an aggregate: kappa of 0.82 across 12 classes can rest entirely on the two classes carrying 80% of the volume. Break agreement down by class and example type before anything is signed off.

   Read chapter
3. 3

   Two reviewers applying the same wrong rule agree perfectly

   A model sits flat at 88% for three months across four rounds of fresh data while inter-reviewer agreement holds steady at 0.86, and the team concludes the labels are fine and the ceiling is the model. Agreement measures consistency between reviewers, not correctness, and a shared misreading of the guideline produces exactly this picture — which is also why a per-item contract with an agreement floor buys consistency and never buys correctness. Scaling the model from here fits the systematic labelling error faster and more confidently, and the plateau does not move.

   Read chapter
4. 4

   Agreement is consistency; a gold standard is correctness

   The instrument that tests the previous chapter is a reference sample re-labelled by someone with authority over the guideline, rather than by the same reviewer pool reading the same document. Measure reviewers against that gold set instead of against each other, and run an error analysis over the model's residuals to check whether the labels on the examples it keeps failing are actually right. This is also what turns an agreement finding into an action with a price: revise the guideline and re-label the affected items, retrain or replace reviewers, split or retire a class, or escalate to adjudication.

   Read chapter
5. 5

   A pilot exists to produce throughput, defects and adjudication load

   Before committing to 120,000 items, spend three hundred: draw them the way the full corpus will be drawn, have everyone who will work the project label them independently, time each item, and adjudicate the disagreements. You leave with throughput, defect rate, agreement and adjudication load. A pilot run on clean examples by the people who wrote the guideline produces none of those honestly, which is how estimates land at half of actual.

   Read chapter
6. 6

   Adjudication and rework are the lines that blow the estimate

   Forty thousand items at a 90-second median is the easy arithmetic, and it is the arithmetic that misses. Absent are resolving disagreements with a senior reviewer, re-labelling everything already done each time the guideline is revised, and reviewer ramp. This chapter works through cutting double-labelling from 100% to 10%, reading a median that drifts from 75 to 110 seconds, and why the work needs a funded owner accountable for a date rather than leftover team capacity.

   Read chapter
7. 7

   Uncertainty sampling cannot tell hard from undecidable

   Switching the next batch from random to uncertainty sampling and watching accuracy rise four points settles nothing without the arm that trains on the same number of randomly chosen items over the same period. The second surprise is agreement collapsing from 0.87 to 0.55 on the selected batch: the acquisition function cannot separate hard-but-decidable items from truncated transcripts and junk, so accepting the batch whole trains on noise no model can learn. And 79% moving to 79.4% on a 40-class problem settles nothing either — it sits inside run-to-run variance, the test set was drawn under the sampling policy you are replacing, and the aggregate hides the classes the new labels were meant to help.

   Read chapter