Diagnosing annotator disagreement

A disagreement set is every item in a labelled batch where two annotators recorded different labels, pulled out as a batch of its own. It is usually treated as waste — the residue an adjudicator has to grind through before the labels can be used — and that framing is what makes teams reach for a relabel. Read the other way it is the highest-density evidence the labelling round produced, because every item in it is a case where the written guideline admitted two readings, and those are exactly the cases a new guideline has to decide.

Two hundred contested items is a dataset#

The set is 800 items from a retrieval-augmented product, meaning one that answers a user's question by first fetching passages from a document store and then having a model write an answer from them. Each item is an answer plus the passages that were retrieved for it, labelled grounded or not_grounded — that is, every claim in the answer traceable to those passages, or not — by two annotators. Raw agreement is 75.0%, Cohen's kappa is 0.38, and positive specific agreement on not_grounded is 0.56.

The 200 items the pair disagreed on are already sitting in the table. Pull them out as a batch and attach every covariate you have, meaning every property recorded alongside an item that could plausibly explain why it was hard — the length of the retrieved context, the query type, the answer length, which annotator flagged it, and the timestamp of the label. Then compare the disagreement rate inside each slice against the rate outside it.

select
  case when ctx_tokens > 800 then 'long_ctx' else 'short_ctx' end as slice,
  count(*)                                            as items,
  sum(case when a_label <> b_label then 1 else 0 end) as disagreements,
  round(100.0 * sum(case when a_label <> b_label then 1 else 0 end)
        / count(*), 1)                                as pct
from golden_grounding
group by 1;

Running that grouping over a handful of covariates on this set produced the following:

Read the table in pairs, because it is three separate partitions of the same 800 items stacked on top of each other rather than six independent groups. The first two rows split the set by whether the answer was partially supported, and 190 plus 610 is 800; the second pair splits the same 800 by context length; the third splits it by which half of the job the label came from. Within each pair the disagreement counts add back to 200, which is the check worth running on any table of this kind, since a slicing that does not reconstruct the total has either dropped items or double-counted them.

Three effects are visible and they are of different sizes. Partial support is the dominant one: items where the passages support some of the answer's claims but not all of them disagree at 62%, against 13% everywhere else, and those 190 items account for 118 of the 200 disagreements. That last figure is where the word "dominant" is earned rather than asserted. The partial-support slice is under a quarter of the items and holds nearly three fifths of the disagreements, so it is over-represented among the contested items by a factor of about two and a half. Long contexts disagree at roughly twice the base rate, which is a smaller effect and is partly explained by partial support being more common when more passages are retrieved. The batch effect is the weakest of the three in the disagreement column, and it becomes clearer when you look at annotator B's flagging rate directly rather than at disagreements: B labelled 39% of the first batch not_grounded and 14.5% of the last, while annotator A stayed between 29% and 31% across all four. B's threshold moved during the job.

Why the flag rate shows the drift more sharply than the disagreement rate does is worth a moment, because it generalises to every per-annotator effect you will look for. A disagreement is a joint event: it requires one annotator to say one thing and the other to say another, so anything that moves one annotator is registered only to the extent that the other one stayed put, and the count is diluted by every case where B's drift happened to carry B toward A rather than away. A flag rate is a single annotator's own behaviour measured against nothing. B going from flagging two items in five to flagging one in seven is a movement of that annotator alone, visible at full size, and it needs no assumption about what A was doing. The rule to carry: when the hypothesis is about one person, measure that person, and use the disagreement set only to find out that there was a person-shaped hypothesis worth having.

A caution about reading these slices: the covariates are correlated with each other, and the first table invites you to treat three findings as three independent causes. Long contexts retrieve more passages, more passages give the answer more claims to make, and more claims is exactly the condition that produces partial support. Cross-tabulating, which means recomputing the rate for one covariate separately within each level of the other, removes most of the length effect — among partial-support items, long and short contexts disagree at 64% and 60% respectively, so length is mostly a proxy.

The logic there is the one step in this chapter that is worth being able to reproduce cold, because without it a covariate table generates as many false leads as findings. The problem is that the two slicings overlap: long-context items are disproportionately partial-support items, so when the table reports that long contexts disagree at twice the rate short ones do, it cannot say whether length is doing anything or whether it is simply pointing at the partial-support items hiding inside it. Cross-tabulating settles that by removing the competition. Look only at partial-support items, so that every item in the comparison is partial-support and that variable can no longer differ between the two sides, and then split those by length. If length were doing work of its own, the two figures would still separate. They come back at 64% and 60% — four points apart where the uncontrolled comparison was twenty apart — so once partial support is held fixed, knowing the context length tells you almost nothing more. Length was a marker for partial support rather than a cause of disagreement, which is worth catching before somebody writes a rule about long contexts and finds it changes nothing.

The same procedure run the other way is what promotes the batch effect from suspicion to finding. Annotator B's declining flag rate could equally have been B being handed easier items later in the job, which would show up identically in a table sliced only by batch. Recomputing B's flag rate within the partial-support slice alone holds item difficulty fixed the same way length was held fixed above, and the decline is still there — so it is B who changed and not the work. Two real causes, then, one of them dominant, and a third pattern that dissolves under inspection.

Three signatures, three different fixes#

Each of those patterns implies a different defect, and only one of them is fixed by hiring better annotators.

For this set, the dominant finding is that grounded was doing the work of two different questions. Reading twenty of the partial-support disagreements makes the split obvious: annotator A was answering "is every claim in this answer supported by the passages", annotator B was answering "is the answer's main claim supported". Both are legitimate labels for a retrieval product and they diverge on exactly the 190 items where an answer is mostly right with one unsupported aside. No amount of training makes two people converge on a question that was never chosen between.

The schema change that follows is small and specific:

task: grounding
version: 2
labels:
  main_claim_supported:  [yes, no]        # the claim the user asked about
  all_claims_supported:  [yes, no, n/a]   # n/a when the answer has one claim
derived:
  grounded: main_claim_supported == yes and all_claims_supported != no
rules:
  - Judge each claim against the retrieved passages only. Background
    knowledge you happen to have does not count as support.
  - An answer that hedges ("this may vary by region") is supported if the
    hedge is consistent with the passages, unsupported if it invents a
    condition the passages do not mention.
  - Context over 800 tokens: read all passages before labelling. Do not
    label from the top-ranked passage alone.

This is the part that better-trained annotators cannot reach. Training transmits a guideline; it does not repair one. A calibration session on the v1 definition would have produced a room full of people who each privately resolved the partial-support case in whatever way the examples happened to suggest, and the resulting agreement would depend on whether the session's examples happened to include an answer with an unsupported aside. If they did, agreement climbs and nobody knows why; if they did not, the new pair reproduces the old number and the conclusion is that annotation is just hard. The mechanism that lifts agreement here is the schema change, and the training round is how it gets distributed.

The drift finding produces a different, smaller change: annotators re-read the anchor examples at the start of each session, batches drop from 200 items to 80, and annotator B's last two batches get re-labelled rather than the whole set, because that is where the movement is.

The loop this sits inside#

Adjudication is not a step at the end of labelling, it is the cycle that produces a usable guideline in the first place, and each pass should be cheap enough to run more than once.

The re-pilot on 60 items is the step people skip, and it is the one that stops the loop from being expensive. Sixty items at two annotators is about 2.5 person-hours — a hundred and twenty labels at the seventy-five seconds an item this task takes is a hundred and fifty minutes — and it answers whether the schema change moved agreement before anyone commits to the full relabel. Kappa is the right statistic to read at that size only because the schema change was expected to move it a long way: at sixty items an interval on kappa is wide, so a re-pilot of this kind can distinguish 0.38 from 0.71 and cannot distinguish 0.38 from a value a few hundredths above it. Design it to answer the question it can answer, which is whether the change did something large, and not the one it cannot, which is how much.

On this set, the v2 schema re-piloted at kappa 0.71 on main_claim_supported and 0.66 on all_claims_supported, which is the evidence that justifies spending the relabel budget. Had it come back at 0.42, the money would have been saved and the diagnosis would have been wrong, which is also worth 2.5 hours to know. Set that against what it is buying insurance on: the full relabel is 800 items twice over, which at the same seventy-five seconds is sixteen hundred labels and a little over thirty person-hours. The re-pilot costs about a thirteenth of that, which is the sort of ratio that makes an argument about whether to run it more expensive than running it.

Shared bias, and the disagreements that do not resolve#

The 600 items where the pair agreed are not garbage and should not be thrown out with the contested ones. They are, however, only as trustworthy as the assumption that two annotators agreeing means both were right, and that assumption fails precisely when both share a systematic misreading. It is worth seeing why no statistic computed from the pair can detect that, since teams keep hoping one will. Every coefficient in this topic is built from the two annotators' labels and nothing else, so all of them are measuring whether the two agree. Shared bias is the case where they agree and are both wrong, which produces exactly the readings a well-run study produces — high agreement, high kappa, a comfortable positive-class figure. The information that would separate the two situations is not in the table, because it is a fact about the world and the table contains only opinions about it. That is why the test has to bring in a third party who did not learn the same habits, and why it cannot be replaced by recomputing anything.

Testing it is cheap: sample 60 of the 600 agreed items, have the domain owner label them blind against the v2 schema, and compare. If the owner matches the pair on 55 or more, the agreed labels carry forward under the new schema with a note. If the owner disagrees on more than about 10%, the agreement was shared bias and those labels need redoing too — which is a much more serious finding than the kappa was, and one the kappa could never have surfaced.

The 200 disagreements have a second life as well. After adjudication they are the hardest, most discriminating items in the set, which makes them the part of a reference set that actually separates a good automated judge from a mediocre one. Keeping them tagged as contested_v1 lets you report judge performance on the easy majority and on the hard tail separately, and the gap between those two numbers is usually larger than anything else in the report.

The common shortcut for resolving a contested item is to add a third annotator and take the majority. That produces a label for every item and destroys the diagnosis, because a 2-1 vote records no reason and the pattern in the reasons was the entire finding. Adjudication as described here has an adjudicator — someone with authority over the definition, usually the person who owns the guideline — resolve each contested item and write one line saying which rule decided it. Those lines are what get clustered. The labels are a by-product; the coded reasons are the output.

It is also true that some disagreement does not go away. If the contested items had been spread evenly across every covariate with no cluster anywhere, the honest conclusion would be that "grounded" as posed has a fuzzy boundary at the resolution the annotators can perceive, and the responses available are to coarsen the label until it is decidable, to replace it with something more directly observable such as "every numeric claim appears verbatim in a passage", or to keep it and publish the ceiling. Publishing the ceiling means every downstream number carries the note that human agreement on this label is 0.55, so no automated system will be validated above that. That is an uncomfortable line in a report and it is more useful than a relabelled set that lands at 0.38 again.

The general shape holds beyond this one set. A disagreement rate is a summary; the disagreements themselves are evidence, and they are cheap to read because there are never very many of them relative to the batch. Slice them by content, by structure, by annotator and by time, act on the signature that dominates, and prove the fix on sixty items before buying the full relabel. Those steps cost hours where the alternative costs days, and only one of the two ends with an explanation.

With a diagnosed guideline and an adjudicated reference set, the next question is how much of this reviewing has to keep being done by people at all. Chapter 5, on the limits of automated evaluation, separates the parts of a review a model can take from the parts where handing it over removes the only check that was catching the failure.