Adjudicating labeller disagreements

Labelling is the step where a golden dataset acquires its ground truth, and it is the step most often run as though the answer were already in the cases and somebody just had to write it down. Two competent people working from the same brief will produce different verdicts on a predictable fraction of any real set. Carelessness is rarely the cause. The brief did not decide the question, so each of them decided it privately, in different directions, in exactly the places where deciding it matters.

This chapter is about running that pass so the disagreements become an asset. It covers the mechanics that affect the numbers more than people expect — what a brief has to contain, why labelling has to be blind in three directions, and what a four-hour sitting does to the last cases in it — then the agreement statistic and the ceiling it puts on every downstream measurement, then the adjudication itself, whose output has to be a rule rather than a verdict. The last part is the consequence nobody plans for: new rules apply to cases already labelled, the set's score moves without the system changing, and the only defence against reading that as a regression is a versioned set with the previous release re-run against it.

What a brief has to decide, and how the pass is run#

A one-page brief saying "mark a case as pass if the assistant handled the conversation correctly" delegates the entire question to the labeller. A brief that is doing its job states the decision procedure rather than the goal: which document is authoritative when the trace and the policy conflict, whether a correct answer delivered in the wrong turn counts, what to do when the customer's request is impossible, and — most importantly — what to do when the brief does not cover the case, which is to stop, tag it, and move on rather than to invent a rule privately. The first version of a brief is always incomplete, and its quality is measured by how many disagreements it converts from silent divergence into explicitly flagged unknowns.

Two mechanical details affect the numbers more than people expect. Labelling has to be blind in three directions: labellers must not see each other's verdicts, must not see the production outcome the case was captured from, and must not know which system version produced the answer they are judging. Showing any of those pulls the labeller toward agreement with something other than the evidence, and it is particularly damaging when the set is being used to compare a new release against an old one, since a labeller who knows which is the new one will find reasons. The other detail is order and fatigue: cases should be presented in randomised order rather than grouped by slice, so that a labeller does not calibrate to a run of similar cases, and sessions should be capped at about ninety minutes, because the last cases in a four-hour sitting are labelled visibly faster and with more variance than the first.

The three blinds guard against three different things, and the third is the one that does lasting damage. Seeing another labeller's verdict is anchoring: the second opinion stops being a second measurement and becomes an edit of the first. Seeing the production outcome imports a verdict that was produced by the system under test, so the label and the thing the label is supposed to judge stop being independent, which is the property the whole set is built on. Knowing which version produced an answer is worse than either, because it does not add noise, it adds direction. Noise widens an interval and washes out over enough cases. A systematic lean toward the release somebody expects to be better moves the estimate itself, and it gets more confident as the set grows rather than less.

What the agreement number is actually telling you#

Raw agreement of 86.7% on 150 cases overstates how aligned two labellers are, because a large share of it comes free from the base rate. If roughly 75% of cases pass, two people guessing independently at that rate would agree 62.5% of the time by chance alone.

That figure is two multiplications and an addition, and doing them is what makes the rest of this section readable. Two people who each pass three quarters of cases, deciding independently, both pass any given case three quarters times three quarters of the time, which is nine sixteenths. They both fail it one quarter times one quarter of the time, which is one sixteenth. Agreeing means either of those happened, so the two add: ten sixteenths, or 62.5%. Multiplying is allowed only because the two are assumed to decide independently, which is exactly the situation being described — two people guessing. Nothing about competence entered the calculation anywhere, and that is the point of it.

Cohen's kappa removes that floor, rescaling agreement so that agreement at the chance rate scores 0 and perfect agreement scores 1:

def cohens_kappa(both_pass, both_fail, a_only, b_only):
    n = both_pass + both_fail + a_only + b_only
    p_o = (both_pass + both_fail) / n
    p_a = (both_pass + a_only) / n          # labeller A's pass rate
    p_b = (both_pass + b_only) / n          # labeller B's pass rate
    p_e = p_a * p_b + (1 - p_a) * (1 - p_b)
    return (p_o - p_e) / (1 - p_e)

print(round(cohens_kappa(both_pass=104, both_fail=26, a_only=14, b_only=6), 2))
# 0.64

Working that function on its own arguments is worth doing once, because kappa is quoted a great deal more often than it is followed. Both labellers agreed on a hundred and four passes and twenty-six fails, so observed agreement is a hundred and thirty of a hundred and fifty cases — the 86.7% above. One labeller passed a hundred and eighteen of the hundred and fifty and the other a hundred and ten, so their individual pass rates are a little under four fifths and a little under three quarters. Those two multiplied together give the chance they would both pass a case by luck; the two fail rates multiplied together give the chance they would both fail one; adding the two gives chance agreement of just under two thirds.

Kappa is then a single ratio between two distances. The room available above chance is one minus chance agreement, which is about a third of the scale — because two thirds of it was going to be agreement whatever anybody did. The distance actually covered is observed agreement minus chance agreement, a bit under a quarter of the scale. Dividing the second by the first asks what share of the available room these two labellers crossed, and that is the 0.64 the code prints. The rescaling is what makes kappa comparable across tasks with different base rates, and it is also why kappa falls on a lopsided task: a set where 98 cases in a hundred pass leaves almost no room above chance, so two labellers can agree on nearly everything and still score poorly, which is a fact about the set rather than about them.

A kappa of 0.64 is respectable by the conventions people quote and it carries a consequence that is easy to miss: it puts a ceiling on every downstream measurement. If two humans applying the same brief disagree on 13% of cases, then an automated scorer that reproduces one of them perfectly will look 87% accurate when checked against the other, and no amount of judge engineering can push measured accuracy above the noise the labels themselves contain.

That ceiling is one substitution long, and walking it is more convincing than accepting it. Suppose a scorer reproduced one labeller's verdict on every one of the 150 cases. Then the scorer's agreement with the other labeller is the two labellers' agreement with each other, because the scorer's verdicts and the first labeller's are the same list. A perfect scorer therefore measures 86.7% accurate, and a scorer measuring higher against that second labeller is agreeing with them in places the first labeller did not — which is not better, it is idiosyncratic in a different direction.

The claim needs its condition attached, because as stated it holds only for a scorer measured against one person's raw verdicts. Once the disagreements have been adjudicated into rules and the affected cases relabelled, the reference has stopped being one person's reading and the ceiling rises accordingly. That is not a loophole in the argument; it is the reason the rest of this chapter exists. What cannot be beaten by judge engineering is the noise in whichever labels you are actually comparing against, so the way to raise the ceiling is to make better labels rather than a better judge. Teams routinely spend a fortnight tuning a scoring prompt from 0.84 to 0.87 agreement with the labels, when the labels agree with each other at 0.87, which means the last three points were fitted to one labeller's idiosyncrasies. Measuring inter-annotator agreement, which is the rate at which two people applying the same brief to the same cases arrive at the same verdict, is what tells you where to stop.

The comparison worth holding is a code review where two reviewers leave contradictory comments on the same function. The useful response is not to average them, and not to let the more senior one win by default, because in both cases the disagreement recurs on the next pull request and the next after that. The useful response is to decide the question once, write the decision into the style guide, and let the guide settle it silently from then on.

Adjudicating in a room, and writing down the rule#

What matters about the procedure is that the output is a rule rather than a verdict, because a verdict settles one case and a rule settles every case like it, including the ones nobody has captured yet. Twenty disagreements on a 150-case set will usually collapse into five or six causes:

The rules go into the labelling guide as numbered, dated entries with the case that prompted them attached, so that a labeller six months from now can see not only what the rule says but what it was invented to resolve:

- id: LG-11
  added: 2026-03-19
  supersedes: null
  question: "The answer is correct but omits a caveat the returns policy mentions."
  rule: >
    A missing caveat is a FAIL only where the policy text uses "must" for that
    caveat. Where the policy says "should", the case is a PASS and is tagged
    caveat_missing so the rate is trackable separately.
  prompted_by: ["ret-0031", "ret-0044", "ret-0058"]
  decided_by: ["dmoreno", "ashaikh"]
  escalated_to: null
- id: LG-12
  added: 2026-03-19
  supersedes: null
  question: "Correct answer that also states the delivery address unprompted."
  rule: >
    Disclosing any account field the customer neither asked for nor needed to
    receive the answer is a FAIL, independent of correctness.
  prompted_by: ["ma-0007", "ma-0011", "acc-0009"]
  decided_by: ["dmoreno", "ashaikh"]
  escalated_to: null

Not every disagreement should get a rule. Where the ambiguity is in the product rather than in the labelling — a returns policy that genuinely does not say whether a customer already refunded outside the window is entitled to a second exception — forcing a verdict encodes one engineer's guess as ground truth and makes every future case of that kind inherit it. The case is tagged blocked-on-policy, excluded from the scored set, and sent to the policy owner. Discovering that a policy is undecided is a real output of building an eval set, and it happens more often than people expect, because labelling is the first activity that forces someone to apply the policy to specific inputs rather than read it.

The rules apply backwards, and the score moves#

New rules do not only decide the disagreements. They also change the correct label for cases in the agreed majority, where both labellers had been consistently applying an assumption the rules now override. A rule like LG-12 is the expensive kind: if both labellers had been passing answers that volunteered a delivery address, every such case in the agreed set is now a failure.

Finding those cases is a query rather than a re-read, and only because the case format keeps rule_applied and an ambiguity note on each row:

affected = [c for c in cases
            if c["labelling"].get("rule_applied") in (None, "LG-02")
            and "address" in c["labelling"].get("ambiguity", "").lower()]
print(len(affected))     # 12 to re-check, not 130

The consequence is that the set's pass rate moves without any code shipping, any prompt changing or any model moving. The definition of a pass got sharper, and the number followed.

What happened there is a change of instrument rather than a change of measurement, and the distinction is the whole reason the next paragraph is necessary. A pass rate is a count of cases meeting a definition, divided by the cases considered; editing the definition edits the numerator while the system under test sits still. Every case labelled before a rule existed was scored on a different instrument from every case labelled after it, which is why applying the rules backwards is mandatory rather than tidy — a set half-labelled under LG-12 and half not is one instrument carrying two scales. It is also why the version stamp belongs on the set rather than on the run. The run is perfectly reproducible; what moved was the ruler. Anyone comparing the new figure against last month's and concluding the assistant had regressed would be reading a change in the ruler as a change in the thing measured. The only defence is that every score is reported against a stated version of the set, so v3 and v4 are never compared directly, and the release that produced the earlier number is re-run against v4 to establish what it would have scored under the new rules.

Two labellers is not the only arrangement#

Double-labelling everything costs twice the labelling budget to buy an agreement statistic. The cheaper arrangement is to double-label a stratified 20–30% sample, measure kappa on that, and single-label the rest, which gives you the ceiling estimate for about a third of the extra cost. It is the right default once a labelling guide has matured, because most of the value of the second pass is in the rules it produces and a mature guide produces fewer new ones.

What you cannot do is single-label everything and skip the agreement measurement entirely. Then the set has an unknown noise floor, and every subsequent argument about whether an automated scorer is good enough is unresolvable — there is no number to compare its accuracy against, and a scorer tuned to 0.91 against labels of unknown quality might be excellent or might be fitted to one person's habits.

Three labellers with majority voting is the arrangement to be most careful with, and it is the one that sounds most rigorous. It produces a verdict on every case without anyone ever articulating the rule, so the same disagreement recurs on every refresh, the guide never grows, and the cost is paid again each time.

The mechanism is that a vote resolves a disagreement at the instant it appears and therefore never represents it. What a disagreement carries is a location: it marks the exact place where the brief was silent. A vote consumes that information and emits a verdict, so the silence stays in the brief and produces the same split next quarter on different cases — a recurrence a written rule would have made avoidable. Two labellers plus an adjudication meeting produce fewer verdicts per hour and a guide that is faster to apply every time it is used, which is a different shape of cost curve rather than a smaller number on the same one.

Majority voting is defensible where labelling is genuinely subjective and no rule could settle it — perceived tone, for instance — and it is a poor fit for anything a policy document could decide.

The agreement number is a measurement of the instrument rather than of the system, and it is the number that bounds everything built on top of the set. What produces it is not the 130 cases both labellers called the same way; it is the twenty they did not, and the value of those twenty is realised only if each one leaves behind a numbered rule that the next labeller inherits. The rules then apply backwards, the score moves, and the set has to be versioned for that movement to be legible as what it is.

What exists now is 149 scored cases, a versioned labelling guide with six rules, a recorded cost per case, and a stamp reading v4 on the nineteenth of March. It describes the assistant as it stood in March, against the policy as it stood in March, over the traffic mix of March — and that stamp is going to matter more than anything else about it, because the product does not stop in March.