Annotator disagreement

Two people applying the same relevance rubric to the same query-document pairs will not produce identical labels, and the standard summary of how close they came is Cohen's kappa, an agreement statistic that discounts the agreement two labellers would reach by guessing alone, so that 0 is chance and 1 is perfect. A kappa of 0.61 is usually described as respectable, and on a first labelling exercise it is the number that ends the discussion and unblocks the set.

It should not, because the statistic is an average and the information is in the distribution. Disagreement spread evenly across every kind of query means two people are making independent small errors. Disagreement concentrated in one class, splitting the same way every time, means the rubric never told either of them what to do with that class, and each supplied the missing clause from their own working life, consistently and invisibly.

Two faithful readings of one pair#

The clearest instance from one such exercise was the pair (query: how do I paginate results?, passage: the reference section for GET /v1/orders). That passage is a parameter table. It lists page_size as an integer with a default of 25 and a maximum of 100, and cursor as an optional opaque string, each with a one-line description. It does not say that the response body contains a next_cursor, and it does not say that you pass that value back in cursor on the following request until it comes back null.

A technical writer who owns the reference pages labelled it relevant, on the grounds that it documents the parameters pagination uses. A backend engineer who supports the SDK labelled it irrelevant, on the grounds that a developer holding only that page writes a loop that either never terminates or silently processes the first 25 of 4,000 orders.

Both of them were applying the rubric from chapter 1, where each dimension is written as a test an annotator can apply, faithfully. The answerability test says a reader holding only this passage can state the answer without guessing, and they disagreed about what the answer is. The writer read the question as "which parameters control pagination", for which the page is complete. The engineer read it as "what sequence of calls retrieves every order", for which the page is missing the one step that makes it a sequence. Nothing about either person's diligence explains the split. The rubric never said which of those two readings a how do I query denotes.

Why the third annotator makes it worse#

The reflex when two labellers disagree is to add a third and take the majority. Inter-annotator disagreement is normal, 2-of-3 resolves it, agreement statistics improve, and the set is unblocked by lunchtime. The procedure is sound when the disagreements are noise — attention lapses, misread passages, a mis-clicked radio button — because averaging independent errors is exactly what majority vote is for.

These disagreements are not noise. Every one of the twenty-seven procedural pairs splits along the same axis, in the same direction, for the same reason. A third annotator does not sample from a distribution of random errors; they arrive holding one of the two readings, and whichever one they happen to hold decides all twenty-seven labels at once. The vote produces a set that looks clean, with kappa comfortably above 0.7 after adjudication, and the label on the most commercially important query class in the corpus has been settled by which person was free on Thursday afternoon. Worse, the signal that told you something was wrong has been consumed: the disagreement was the only artefact pointing at the gap, and majority vote deletes it.

It is worth naming the property that makes majority vote useless here, because it is the property that makes most procedural defences useless. Run the vote over a set whose disagreements are genuine noise and it returns a clean set with a higher kappa. Run it over a set whose disagreements are a hole in the rubric and it returns a clean set with a higher kappa. The output is the same in both cases, which means the output carries no information about which case you were in. A check whose result does not move when the thing it exists to detect is present rather than absent is not a weak check. It is a ritual. Turning it back into a check means reading something that does differ between the two cases, and the thing that differs is where the disagreements land.

The distinguishing question is cheap to ask. When a pair splits, ask each annotator which dimension decided their label — which is what the mandatory decided_by field is for. If they cite different dimensions, or the same dimension with different readings, the rubric is underspecified and no amount of voting will fix it. If they cite the same dimension with the same reading and still disagree, one of them made a mistake on this pair, and adjudication is the right tool.

The middle branch is worth keeping. A handful of genuinely ambiguous pairs exist in every corpus — a passage that is half an answer, a query whose intent is unrecoverable from the string — and forcing a label onto them adds noise to every number you compute afterwards. Dropping them costs a percent of the set and is honest, provided you record the count, because a set that has quietly shed 15% of its hardest cases will report a recall figure that no production system can reproduce. The mechanism behind that is short enough to state in full. The pairs that get dropped are the ones no rule in the rubric decided. A pair no rule decided is, by construction, a pair sitting on a boundary, and boundaries are where a retriever is most likely to be wrong. Removing them therefore lifts the measured score without anything about the system having changed, which is why the count has to be recorded next to the score rather than in a commit message.

Naming the criterion and paying for the relabel#

The missing criterion in the pagination case is a narrower form of answerability that the original rubric collapsed. Procedural queries ask for a procedure, and a passage answers a procedural query when a reader can execute the task end to end from it. Written as a test an annotator can apply:

dimensions:
  answerability:
    test: >
      A reader who sees ONLY this passage can state the answer without
      guessing. Mentioning the topic is not answering it.
    procedural_clause: >
      For queries of the form "how do I X", the passage must let a reader
      produce every step of X, including the ordering of steps and the
      termination condition where one exists. A passage that documents the
      inputs to a step, without stating the sequence, fails this test.
      Label such a passage `partial` when it is the reference page for the
      operation, `irrelevant` otherwise.

The partial carve-out is not a hedge. A reference page that lists the right parameters is genuinely useful as supporting context alongside a guide that states the loop, and a system that retrieves it in slot three next to the pagination guide in slot one is behaving well. Labelling it flatly irrelevant would train a reranker to suppress a page that belongs in the answer, which is the opposite failure and just as expensive.

The numbers before and after are what justify the exercise to whoever paid for it, with recall@5 standing for the share of queries whose answer the live retriever placed somewhere in its top five results — chapter 4, on ranking families and the metric that compares them, makes that precise.

The last row is the one that matters and the one nobody expects. The rubric fix did not change the retriever at all; it changed what counts as a hit, and the reported quality of the live system dropped by fifteen points. That fifteen points was always there — users had been experiencing it as documentation answers that looked right and did not work — and the measurement had been agreeing with the retriever's mistake because both of them were treating topical presence as sufficient.

The row above it says why the last one moved, and the arithmetic between the two is worth doing rather than accepting. Thirty-one pairs lost their relevant label, twenty-four of them down to partial and seven to irrelevant. Recall@5 scores a query as a hit when at least one relevant passage sits in its top five. Demoting a pair can therefore cost a query its hit, and it can equally cost nothing at all, because a query that had two relevant passages in the top five still has one. The fall in recall is bounded above by the thirty-one demotions and has to land below it, and fifteen points is the shape that produces. A fall larger than the demotions would have meant something other than the relabel had changed underneath the comparison.

The first two rows are two views of one tally, and the distance between them is not an error either. Cohen's kappa takes the observed agreement, subtracts the agreement two labellers would have reached by guessing, and divides by the room that was left — agreement beyond chance, over the beyond-chance agreement that was available to reach. Run that backwards on the before column. An observed 84.5% with a kappa of 0.61 pins the chance agreement at about six pairs in ten, which is what a pooled set should give: most pairs in a pool are plainly irrelevant, both annotators say so, and neither of them needed the rubric to get there. Now run it backwards on the after column. An observed 96.0% with a kappa of 0.89 pins the chance agreement at about six in ten again. The baseline held while the agreement moved, and that is the check worth caring about — had the chance figure jumped between the columns, the improved kappa would have been the label distribution shifting rather than the two people converging.

The third row is where the average in the first two comes apart. Before the fix, twenty-seven of the seventy-six procedural pairs split, which is better than one pair in three. Every other pair in the set is one of the remaining hundred and twenty-four of the two hundred, and between them they account for the four disagreements left over from thirty-one — fewer than one in thirty. Those two rates differ by more than a factor of ten, and one kappa computed across both of them reports neither. That is the case against reading the headline number at all. The set holds one population the annotators understood identically and one they did not, and averaging the two produces a figure that describes no population in the set.

Checking that the criterion was actually applied#

A written criterion and an applied criterion are different things, and the gap between them is invisible in agreement statistics, because two annotators who both ignore the recency clause agree with each other perfectly. Three cheap mechanisms close most of it.

Seed control pairs into the batch. Twelve pairs whose correct label you have already established, unmarked and shuffled into the two hundred, cost the annotators four minutes and tell you whether each criterion is live. Make at least three of them turn on recency specifically: a superseded policy page that states a clean, confident, wrong number. An annotator who marks that pair relevant has not internalised the recency test, whatever the rubric says, and you have found that out for the price of twelve pairs rather than after launch.

Audit the decided_by distribution against something you know independently. Every document in the corpus carries an effective_date, and you know which documents have a superseded_by field set. Any pair involving a superseded document should have been decided by recency; if it was labelled irrelevant but decided_by says authority, the annotator got the right answer for the wrong reason, which will not generalise to the next superseded document that happens to sit in an authoritative location.

def audit_recency(labels: list[dict], docs: dict[str, dict]) -> list[dict]:
    """Pairs where the corpus says recency should have decided, and it didn't."""
    suspicious = []
    for row in labels:
        doc = docs[row["doc"]]
        if doc.get("superseded_by") is None:
            continue
        if row["label"] == "relevant" or row["decided_by"] != "recency":
            suspicious.append({
                "annotator": row["annotator"],
                "pair": (row["query"], row["doc"]),
                "label": row["label"],
                "claimed": row["decided_by"],
                "expected": "recency",
            })
    return suspicious

Re-label a sample blind. Take 10% of each annotator's pairs, strip the labels, and give them back a week later. Self-agreement below about 0.85 means the criterion is being applied by feel rather than by test, and no amount of cross-annotator agreement rescues that, because a person can be reliably inconsistent in a way that a second person mirrors.

The smallest set worth having, and its resolution limit#

A hundred queries is the number to start with, sampled from real traffic rather than invented. Pool the top five results from every ranker you are comparing, deduplicate, and you land at roughly 650 distinct pairs. At fifteen to twenty seconds per pair for someone who knows the domain, that is under four hours of work. Have both annotators label the same 30 queries so you can compute agreement, and split the remaining 70. One afternoon each, and you have a set with a stated rubric, measured agreement and an audit trail.

Every figure in that paragraph is a multiplication, and they are worth seeing rather than trusting. A hundred queries, taken five deep, from two rankers is a thousand pairs before deduplication, so the six hundred and fifty that survive say the two rankers are returning three or four of the same documents inside each top five. Six hundred and fifty pairs at twenty seconds apiece — the slow end of the stated rate — is thirteen thousand seconds, a little over three and a half hours, which is where under four hours comes from. Splitting the work is what turns that into an afternoon. Both annotators label the same thirty queries and each takes half of the remaining seventy, so each of them covers sixty-five. Six hundred and fifty pairs over a hundred queries is six and a half pairs per query, sixty-five queries is therefore a little over four hundred pairs each, and four hundred pairs at twenty seconds is under two and a half hours. The same rate prices the twelve control pairs from the section above at four minutes, which is the four minutes already quoted there — the two estimates were made with one assumption rather than guessed separately.

That number sets the resolution you can expect. A hundred queries measures a recall figure to roughly ±0.09 at 95% confidence, which is enough to detect the difference between 0.62 and 0.81 and not enough to adjudicate 0.74 against 0.76. Knowing that in advance stops the meeting where two systems that are statistically identical get argued about for an hour.

Where nine points comes from decides how much a larger set would buy, so it is worth one derivation. A rate measured over a hundred queries has a standard error equal to the square root of the rate, times one minus the rate, divided by the number of queries. Somewhere near seven in ten — which is where these recall figures sit — that expression lands a little under five points, and a ninety-five percent interval is about two standard errors wide, which is the nine. The number of queries sits under the square root. The interval therefore shrinks with the square root of the set size rather than with the set size, so halving nine points to four and a half takes four hundred queries rather than two hundred. A hundred is the recommended starting point because the next genuinely different resolution costs four times as much, and the intermediate sets buy almost nothing.

The two comparisons in that sentence sit deliberately either side of the line, and checking them is how the interval becomes usable rather than memorised. Add nine points to 0.62 and subtract nine points from 0.81: the first lands just below the second, so the two intervals fail to touch by a hair and the difference between those systems is real. Do the same to 0.74 and 0.76 and each interval swallows the other's centre several times over, so there is no difference in there to recover however long the meeting runs. The rule that falls out is that a gap has to exceed twice the interval before it is worth arguing about, and 0.81 against 0.62 clears that bar by the narrowest margin the arithmetic allows.

Which is the shape of the argument when the head of support says the assistant goes live in two weeks and there is no time to label anything, because the demo answers were great. The demo answers being great is the weakest evidence in the room: those questions were selected, consciously or not, because they produced good answers, which makes the demo set a sample with maximum survivorship bias. Answering with a request for a two-week labelling sprint concedes the frame, since it accepts that measurement costs two weeks when it costs six hours.

The cost column closes exactly, which is the first thing to check on a plan of this shape: half an hour, an hour, three hours and half an hour is five hours, with nothing parked in a row that reads and then integrate. The clock in the left column runs from nine in the morning to half past three, so the day is one working day whichever of the two figures gets quoted, and the difference between the column and the clock is the hour the column does not book. A plan whose rows sum to less than its total has an unnamed task in it, and the unnamed task is always the one that slips.

The binary label is the compression that makes this fit in a day. It gives up the graded scale, the partial category and the per-dimension audit, and it still answers the only question that matters before launch: how often is the evidence the model needs actually in front of it, and how often is the top result a document the company has already replaced.

The thing to refuse to ship without is that superseded-document count on real queries. Not a benchmark, not a target — a number, measured on questions customers actually asked, for how many of them the assistant would answer out of a document that is no longer in force. If it comes back zero, launch with confidence and a good story. When it came back on one policy corpus it was 9 out of 100, and every one of those was a customer being told something the company would not honour.

Those two numbers do different jobs, and both fall out of the same binary pass over the same pairs. Recall@5 counts the queries where the evidence the model needed was somewhere in front of it. The superseded count says nothing about whether the evidence was present and everything about what was sitting next to it. A query can be inside the good part of the first number and inside the nine of the second at the same time, because a top five is wide enough to hold a policy and the document it replaced, and a model reading both has no field to tell it which is which. That asymmetry is why the second number is the one to gate a launch on. The first improves when more documents are put in front of the model, and the second gets worse when they are.

Disagreement, then, is the cheapest diagnostic in this whole topic, and it only works if you look at where it lands rather than at how much of it there is. Two labellers, one shared class of query, and a decided_by field are enough to find a hole in a rubric that no amount of careful writing would have closed in advance, because the hole is only visible once two people with different working lives apply the same sentence to the same pair.

Nine out of a hundred is a rate, and rates invite the wrong response, which is to tune something until the rate drops. Before tuning anything it is worth understanding why the retriever produced those nine, because the answer is not that the model was poorly trained or the index poorly configured. A similarity score is structurally incapable of representing the constraint that was violated, and the sharpest way to see that is to build a query where the single closest passage in the corpus is not merely a weaker answer than the best one, but definitively wrong.