Writing an annotation guideline

Human evaluation is the practice of having people read a system's outputs and record a judgement about each one under a written procedure, so that the judgements can be counted, compared across releases, and used to decide whether something ships. It exists because the properties that decide whether a support bot is worth deploying — whether an answer resolves the user's problem, whether it asserts something false, whether it refuses when policy says it should — have no automatic test. A unit test can assert that a function returns 3; no assertion in a test suite establishes that a paragraph is helpful, so someone reads it and says so. The moment more than one someone is reading, the two readings have to be made comparable, and everything else in this topic depends on that having been done properly.

What makes them comparable is the annotation guideline, also called the rubric or the labelling spec: the written instructions telling a reviewer which question they are answering, what each available score or category means in terms they can check, and how to resolve the cases where the right answer is not obvious. Each judgement a reviewer records is a label — one score, category or flag attached to one output — and the people producing them are called annotators or reviewers interchangeably. Before the real labelling starts, a pilot round puts two of them over the same small batch of outputs independently, which is where a guideline's gaps first appear as two people writing down different numbers for the same text. This chapter is about writing the guideline so that those gaps get closed rather than averaged.

A backend engineer meets this work in three places: standing up the evaluation for a feature nothing in the existing metrics can score, inheriting a golden dataset assembled by someone who has since moved on and being asked whether its numbers can be trusted, or reading a release report where a human agreement figure sits beside an automated score. All three depend on how precisely the guideline was written, because a guideline that leaves a case undefined produces labels two people would have assigned differently, and no statistic computed downstream recovers what each of them meant.

Where two reviewers diverged#

A support-bot evaluation ran its pilot on 200 answers with two reviewers, one from support operations and one from the machine-learning (ML) team, working through the same batch independently over three days. They produced the identical score on 82 of them, or 41%. They were within one point on 79%. On twelve items they were three or more apart, including one answer that received a 5 from one reviewer and a 2 from the other.

Those three figures describe three different populations and are worth converting into counts before any of them is interpreted. Eighty-two exact matches out of two hundred is a little over two in five. A hundred and fifty-eight items — that is 79% of two hundred — landed within one point, which means the two reviewers were adjacent but not equal on about seventy-six of them, since the within-one figure contains the exact matches inside it. And twelve items sat three or more apart. Read downwards, the shape is a pair who mostly hover near each other and occasionally diverge completely, which is not the shape a pair produces when one of them is simply careless. Carelessness is spread evenly. A dozen items where two people are at opposite ends of the same scale is a pattern, and a pattern has a cause you can go and find.

Pulling the twelve worst disagreements apart is faster than arguing about the aggregate. The answer that scored 5 and 2 was this one, in response to "I was charged twice for March — can you refund the duplicate?":

I understand duplicate charges are frustrating. Our billing system sometimes creates a second authorisation hold that drops off within 3–5 business days. If the second line is still on your statement after 5 business days, contact billing@ and they will issue a refund.

The support-operations reviewer gave it a 5: the explanation is correct, the timeline is correct, and it is what a good agent would say. The ML reviewer gave it a 2: the user asked for a refund and the bot did not initiate one, so the user's task was not completed. Both readings are defensible under "use your judgment", which means the scale was never measuring one thing.

The rest of the disagreements sorted into four groups, and every group is a question the guideline had not answered.

None of these is a hard question once someone decides. All of them are unanswerable by a reviewer sitting alone with a five-point scale at 4 p.m. A guideline is the same kind of object as a test specification: its job is to make the outcome reproducible by someone who was not in the room when it was written, and it fails in the same way a spec fails, by leaving a case undefined and letting each implementer resolve it privately.

The claim in this chapter's opening — that an undefined case produces two instruments rather than one noisy instrument — is the one worth being able to derive, because everything anyone does about a low agreement figure follows from which of the two it is. Take noise first. A noisy instrument is one whose readings scatter around a common value: both reviewers are answering the same question, and each of them is a bit inconsistent about where the line falls, so their errors point in no particular direction. That kind of variation has a useful property. It averages out. Add a third reviewer, or have each reviewer label each item twice, and the mean of the readings converges on the value they are all scattered around, because the deviations cancel.

Now take the other case. The support-operations reviewer is answering "is the information correct" and the ML reviewer is answering "was the user's task completed", and on the billing answer those two questions have different right answers — a 5 and a 2. Neither reviewer is scattered; each is being perfectly consistent about the question they think they were asked. The deviation between them is a fixed offset on a whole class of items rather than a wobble, so it does not cancel, and adding reviewers does not help: a third reviewer answers one question or the other, which adds weight to one side rather than converging on anything. Averaging a 5 and a 2 lands halfway between three and four, and that is not a compromise between two readings. It is a value no reviewer would assign, on a scale where nobody has said what it would mean.

That difference is testable and the test costs nothing, which is what makes it the first thing to do rather than an academic distinction. If the variance is noise, relabelling a subset with the same guideline produces a different disagreement set the second time, because which items fall on the wrong side of a wobbly line is itself random. If it is a definitional split, the same items are contested again, and they are contested in the same direction — the same reviewer high, the same reviewer low, on the same kind of case. Reading the disagreements is how you tell, and it is why the twelve worst were pulled apart above before anything was concluded about the 41%.

Every level needs a criterion someone can point at#

The repair is to replace the adjectives with conditions a reviewer can check against the text in front of them. "Helpful", "clear" and "high quality" are all judgments about the reviewer's internal state. "Contains a next action the user can take without leaving the chat" is a property of the string. Two reviewers can disagree about the first indefinitely and will rarely disagree about the second.

The asymmetry there is not about one phrasing being more precise than the other, and treating it as a matter of precision leads to the wrong repair, which is longer adjectives. It is about where the thing being judged lives. "Contains a next action" names a property of an object both reviewers are looking at, so a disagreement about it is a disagreement about that object and can be settled by pointing: here is the sentence, or there is no such sentence. "Helpful" names a property of the reviewer's response to the object. Both reviewers can inspect the answer as carefully as they like and never reach the other's response, because the response is not in the answer. A disagreement about it has nothing to be settled against, which is why those arguments run long and end in seniority rather than in evidence. The rule that falls out is simple to apply while writing: for each level, ask what a reviewer would point at to defend the score. If the answer is a span of text, the level is written. If the answer is "well, it just isn't very helpful", it is not.

Here is the rewritten scale for the same support bot. The task was fixed first — helpfulness means the user's stated request is resolved — and every level is expressed as a condition on the answer relative to that request.

Three properties of that table are doing the work. Each level names an observable condition rather than a feeling. Each level carries a worked example drawn from the pilot, so a reviewer who is unsure which of two levels applies has two anchor texts to compare against rather than two adjectives. And the examples are real outputs from the system under test, which matters because invented examples are always cleaner than production traffic and therefore anchor the scale in the wrong place.

The same content goes into the annotation tool as a machine-readable spec, versioned alongside the golden set it applies to, so that a label can be traced to the guideline revision that produced it:

task: support_answer_helpfulness
version: 4
definition: >
  Helpfulness means the user's stated request is resolved by this answer.
  Score the answer as delivered, not the answer you would have written.
scale:
  5: {resolves: full, gaps: none, facts: all_correct}
  4: {resolves: full, gaps: one_closable_by_user, facts: all_correct}
  3: {resolves: none, information: correct, next_step: other_channel}
  2: {resolves: none, scope: narrower_than_request}
  1: {facts: any_incorrect, or_policy: violated, or_scope: different_question}
anchors_file: anchors/support_helpfulness_v4.jsonl
precedence: [factual_accuracy, policy, resolution, completeness, tone]
edge_cases: rules/support_helpfulness_v4.md

The anchors_file holds the worked examples with their scores and a one-line reason each. Reviewers open it at the start of every session; it takes four minutes and it is the cheapest agreement intervention available, because it re-anchors the scale before the drift of the previous session carries forward.

The edge cases are found, not predicted#

The four clusters in the pilot are not a list anyone would have written in advance. That is the structural argument for running a pilot round on 40 to 60 items before the real labelling starts: the purpose is to generate disagreements while they are still cheap, then convert each one into a rule. A pilot that produces perfect agreement has usually been drawn from the easy part of the distribution and has taught you nothing.

For the support bot, the four clusters became four rules, each stated as a condition and a resolution rather than as guidance:

## Edge-case rules (v4)

1. **Correct information, no resolution.** Cap at 3. The scale measures
   resolution; a correct explanation that leaves the user to act elsewhere
   is a 3 regardless of how well written it is.

2. **Narrower scope than asked.** If the answer addresses a strict subset
   of the request, score 2. If it addresses the request plus extra
   material, ignore the extra unless rule 3 applies.

3. **Correct answer with an added unsupported claim.** Score 1. An answer
   containing a false statement is a 1 even if the rest resolves the
   request. Accuracy has precedence over resolution.

4. **Policy-correct refusal.** Score 4 if the refusal names the correct
   channel, 2 if it refuses without a route. Refusals are never 1 unless
   the refusal itself was wrong (the bot could have answered).

5. **Ambiguous user request.** If the request admits two readings and the
   answer resolves one of them, score against that reading. Flag the item
   with `ambiguous_request` rather than lowering the score.

Rule 3 is the one that surprises people, and it is where the ordering of criteria stops being cosmetic. Two of the pilot's disagreements were answers that resolved the request and also asserted something false about the refund window. One reviewer averaged the good and the bad into a 3; the other let the error dominate and gave a 1. Both behaviours are reasonable and they cannot coexist in one dataset, so the guideline has to say which one the project wants. The precedence list in the spec above is that decision written down: factual accuracy outranks policy, which outranks resolution, which outranks completeness, which outranks tone. When two criteria pull in opposite directions, the higher one in the list decides the score and the lower one is recorded in the reviewer's note.

A tie-break has to be a procedure, not an exhortation. "Use your best judgment when criteria conflict" reintroduces exactly the variance the rest of the document removed. "The first criterion in the precedence list that the answer fails sets the ceiling" is a rule a reviewer can apply at 4 p.m. on a Friday and get the same result as they would have got on Tuesday morning.

A precedence list rather than a set of weights is a deliberate choice and the reason is worth having, because weights are the more sophisticated-looking option and they are the wrong one here. Weights ask the reviewer to combine — this answer is strong on resolution, weak on accuracy, so what does that come to — and combining requires each reviewer to decide privately how much weaker than what, which is the private resolution the whole document exists to remove. A precedence order asks a different question, and it asks for a single comparison rather than a computation: does this answer fail accuracy, yes or no. If it does, nothing below accuracy in the list can change the score, so no combining ever happens and there is nothing left for two reviewers to weigh differently. The cost is real and should be stated: a precedence order throws away information about how good the answer was on the criteria it never reached, which is why the guideline sends that information to the reviewer's note rather than deleting it.

Re-run on the same 200 items with the rewritten guideline, the two reviewers matched exactly on 74% and were within one point on 96%. Both movements are worth converting: exact matches went from eighty-two of the two hundred to a hundred and forty-eight, so sixty-six items that two people had scored differently now receive the same score, and the within-one figure went from a hundred and fifty-eight to a hundred and ninety-two. The remaining exact-match failures were concentrated on the 4-versus-5 boundary, which is where a scale with fine gradations always pays for itself least.

Reliability is not validity#

A guideline of this kind makes reviewers reproducible. It does not make them correct, and the two get conflated constantly. If the precedence list is wrong for the product — if, say, tone genuinely matters more than completeness for this audience — then two reviewers will now agree beautifully on a score that does not predict whether users came back. Agreement is a property of the measurement process; validity is a property of the relationship between that measurement and the outcome you care about. A rubric can only be checked for the first by inspecting labels, and the second has to be checked against something outside the labelling process entirely, such as escalation rates or repeat-contact rates for the answers each level was assigned.

The other thing a guideline cannot do is remove ambiguity that is real. Rule 5 above does not resolve ambiguous user requests, it quarantines them: the reviewer scores against one reading and tags the item. Some categories of judgment are irreducibly contested, and forcing a rule onto them produces agreement by suppressing information rather than by clarifying it. When 15% of a sample carries the ambiguous_request tag, that is a finding about the product's input distribution and it belongs in the report, not averaged into a mean helpfulness score of 3.6.

There is also a limit on how much specification a document can carry before it stops being read. The v4 spec above is roughly a page and a half including anchors. Guidelines that run to eight pages get skimmed, and a rule that is not read is worse than a rule that was never written, because the project now believes the case is covered. Refining and Reusing Annotation Guidelines for LLM Annotation is worth reading on the loop that keeps this bounded: pilot, cluster the disagreements, add the smallest rule that resolves the cluster, re-pilot, and delete rules that no disagreement has invoked in two rounds.

The guideline is the instrument, and the reviewers are people operating it. When two of them produce different numbers for the same text, the first hypothesis worth testing is that they were handed different instruments, because that is the hypothesis that costs a written paragraph each to check and repairs the whole programme when it turns out to be true. Observable conditions at every level, anchors drawn from real outputs, a rule for each case a pilot surfaced, and a precedence order for conflicts is the whole of the repair, and on this support bot it moved exact agreement from 41% to 74% without a minute of reviewer training.

What none of the numbers in this chapter — 41%, 74%, 96% — has been checked against is how much of that agreement two reviewers would have reached by accident. On a five-point scale with a flat distribution that baseline is low and the improvement is real: two people assigning scores at random across five levels used equally often land on the same level one time in five, so the starting 41% was already well clear of it and the 74% is further clear again. On the labels most evaluation programmes care about, where the interesting class shows up in a few percent of traffic, the baseline is high enough to swallow the entire number: chapter 2, on chance-corrected agreement, takes a pair of reviewers who agreed on 96% of a thousand items and works out why the statistic that corrects for chance came back at 0.14.