Cohen's kappa and raw agreement

Chance-corrected agreement is any measure of how often two people labelling the same items match each other, adjusted for how often they would have matched by coincidence given how liberally each of them uses each category. Cohen's kappa is the standard one for exactly two raters, and it reports the share of the headroom above that coincidence baseline which the pair actually captured: 1.0 for perfect agreement, 0 for agreement no better than two independent coin flips weighted the same way.

This chapter is about what happens to that statistic when the interesting class is rare, which is the condition almost every safety, compliance and quality label operates under. Both figures are correct answers to different questions, and reading only one of them either ships a label set that identifies nothing or bins one that was fine.

What the thousand items actually contained#

A trust-and-safety team labelled 1,000 production outputs for "contains a policy violation" with two reviewers working independently against a guideline that had already been through a pilot round. Raw agreement came back at 96.0%. The same job also computed Cohen's kappa, and that printed 0.146.

Raw agreement collapses a 2×2 table into a single fraction, and the fraction is the same for many tables that mean very different things. Expanding it takes one query:

The 960 agreements that produce 96% are 956 items both reviewers called clean and 4 items both called violations. Against those 4 joint flags there are 40 items that exactly one reviewer flagged. Put plainly, on the class the label exists to detect, the reviewers picked the same item 4 times out of the 44 that either of them thought worth flagging.

Nobody reads the table that way from the 96%, because 96% is arithmetically dominated by the cell that required no discrimination at all. Consider what a reviewer who clicked "clean" on every item without reading any of them would score against reviewer A's real labels. They would match A on every one of the 978 items A called clean and miss all 22 A flagged, which is agreement of 97.8% — better than the two real reviewers managed with each other. A measurement that a reviewer who read nothing can beat is not measuring reading.

The baseline kappa subtracts#

That last observation is the whole of chance correction. Cohen's kappa asks what fraction of items two raters would agree on if each of them kept their own overall rate of saying "violation" but distributed those flags independently of the other's, and then reports how much of the remaining headroom the observed agreement captured.

def cohen_kappa(both, a_only, b_only, neither):
    n = both + a_only + b_only + neither
    p_o = (both + neither) / n                 # observed agreement
    a_yes, b_yes = (both + a_only) / n, (both + b_only) / n
    p_e = a_yes * b_yes + (1 - a_yes) * (1 - b_yes)   # expected by chance
    return p_o, p_e, (p_o - p_e) / (1 - p_e)

p_o, p_e, k = cohen_kappa(both=4, a_only=18, b_only=22, neither=956)
print(f"observed {p_o:.4f}  expected {p_e:.4f}  kappa {k:.3f}")

observed 0.9600  expected 0.9531  kappa 0.146

Three lines of that function carry the whole idea, and each is worth building rather than reading. p_o is the easy one: the items the pair agreed on, over all the items, which is the 96% already in hand.

p_e is the modelled floor and it is the line to slow down on. The question it answers is: if each reviewer kept their own habit of flagging but chose which items to flag with no reference to the other, how often would they land on the same label anyway? There are exactly two ways for that to happen. Both say violation, or both say clean. Multiplying the two flagging rates gives the first, and multiplying the two clean rates gives the second, and multiplying is allowed only because the model assumes the two choices are independent — that is the entire content of "by chance" here, and it is an assumption rather than a fact about the reviewers. Adding the two products is allowed for a different reason: both-violation and both-clean cannot happen on the same item, so the two ways of agreeing never overlap and their chances add without double-counting anything.

Put the table's own marginals through it. Reviewer A called 2.2% of items violations and reviewer B called 2.6%. Two independent coin flips weighted that way both say violation about six times in ten thousand — a small number multiplied by a small number is a very small number — and both say clean just over ninety-five times in a hundred, since a shade under ninety-eight in a hundred multiplied by itself is a shade under ninety-six in a hundred. Those two add to 95.31%. Almost all of the floor is the second term, which is the arithmetic statement of why a rare class is a problem: when both reviewers say "clean" almost always, "both said clean" is almost certain whether or not either of them looked.

So the observed 96.00% is 0.69 percentage points above a floor that costs nothing to reach, and the ceiling above that floor is only 4.69 points wide. Kappa is the ratio of those two quantities, 0.0069 divided by 0.0469, and the division is what makes the statistic mean anything. The numerator is what the pair earned above the floor. The denominator is what there was to earn — the distance from the floor up to perfect agreement, which is one minus the floor. Dividing turns an amount into a share of the available headroom, which is why kappa reads as 1.0 for a pair who took all of it and 0 for a pair who took none. The reason it comes out so much smaller than the raw figure is that it is measured on a scale 21 times shorter: the raw number is scored out of a hundred points, and this pair was scored out of four and seven tenths.

The expected-agreement term carries an assumption that is easy to accept without noticing. It models the two reviewers as independent given their own flagging rates, which is a reasonable description of two people working through separate queues and never comparing notes. It is a poor description of two people who sat next to each other, asked "what did you put for the one about refunds?", or worked from a shared spreadsheet where the other's answer was visible. Correlated reviewers inflate observed agreement without inflating the modelled baseline, so kappa rewards the contamination. That is an argument for keeping the two passes genuinely blind — separate assignments, no shared sheet, no in-flight discussion — and for treating any calibration conversation as a separate, logged event that ends before the measured round begins.

One more quantity from the same table bounds how much of the shortfall is even fixable. Reviewer A flagged 22 items and B flagged 26. If every one of A's flags had been inside B's set, agreement would have been 99.6% and kappa would have been 0.915. That figure is worth reconstructing, because it is the check that stops the whole diagnosis being an artefact. Hold both reviewers' flagging habits exactly where they are — A flags 22 items, B flags 26 — and rearrange only which items they chose. The best available arrangement puts all 22 of A's flags inside B's 26, which leaves four items B flagged alone and none A flagged alone, so the pair disagrees on four items out of a thousand and agrees on 99.6%. The floor does not move, because the floor depends only on the two flagging rates and those were held fixed. Feed the new observed agreement through the same division and the answer is 0.915.

That number is the ceiling this pair could have reached without either of them changing how liberally they flag, and it matters because there is a real phenomenon in which lopsided marginals — the row and column totals of the table, meaning how often each reviewer used each category — mathematically cap kappa, so that a low value says more about the shape of the table than about the reviewers. This is not that case. The pair achieved 0.146 out of an attainable 0.915, and dividing one by the other says they captured about a sixth of what was available to them. There is real headroom, and the pair is using 16% of it.

Where the statistic came from, and what its labels are worth#

Chance-corrected agreement predates machine learning by decades. William Scott proposed pi in 1955 for coding open-ended survey responses, using a single pooled estimate of how often the category was used; Jacob Cohen published kappa in 1960 with the modification that each rater keeps their own marginal, so a rater with a systematically heavier hand is not penalised as disagreement in the expected-agreement term. That choice is why Cohen's kappa is the default for exactly two raters and why it stops being the natural tool the moment you have three, which is territory the next chapter covers.

The difference between those two choices is small on paper and decides something real, so it is worth seeing what each does to this table. Scott's pooled estimate would take the flags of both reviewers together — 22 and 26, pooled over two thousand opportunities — and use one flagging rate for both sides of the multiplication. Cohen keeps A's 2.2% for A and B's 2.6% for B. The consequence is what happens to a rater with a systematically heavier hand. Under a pooled estimate, a reviewer who flags twice as often as their partner is partly charged for it, because the pooled rate sits between the two and the model expects each of them to flag at that middle rate; the excess shows up as disagreement the coefficient did not expect. Under Cohen's version the model already knows each reviewer's habit and asks only whether they picked the same items, so a heavier hand is absorbed into the baseline instead of being scored as a failure. Which of the two you want depends on whether "one of them flags more than the other" is a finding you want the statistic to report or a fact about the pair you want it to control for.

The interpretive labels that get quoted alongside kappa — "slight", "fair", "moderate", "substantial", "almost perfect" — come from a 1977 paper by Landis and Koch on observer agreement for categorical data, and the authors described the cut-points as arbitrary at the time. They have been reproduced so often that a 0.146 gets read as "slight agreement, but agreement nonetheless", which is a more comforting sentence than the data supports. Nothing about the labelling task determines that 0.61 is where "substantial" starts. The number that matters is whether the labels are reliable enough for the decision they will feed, and that depends on how the labels are used, not on a table from a biostatistics journal.

Three label sets with the same raw agreement#

Raw agreement and kappa answer different questions, and the informative move is to read them together rather than picking whichever one supports the plan. Three label sets, all with the same 96% raw agreement, show how far apart the underlying situations can be:

The first two rows differ only in how the 40-odd flagged items overlap, and that difference is invisible in the headline. In the second row the pair disagrees on the boundary of a category they both broadly recognise, which is the situation a guideline revision fixes. In the first row there is no evidence that either reviewer can identify the positive class at all, and the plausible explanations are of a different kind: the definition of "violation" may be under-specified in a way the pilot did not reach, one reviewer may be applying a much broader reading than the other, or the sample may contain so few genuine violations that the 44 flags are mostly borderline material where any reading is defensible.

The third row is a reminder that kappa is not intrinsically pessimistic. When both classes are well represented and the reviewers actually track each other, chance-corrected agreement lands close to the raw figure, because the free baseline is only 50-ish percent instead of 95. Run the floor calculation on that row and it comes out at exactly one half: A flags about half the items and B flags half, so both-violation is about a quarter and both-clean is about a quarter, and a quarter plus a quarter is a half. The pair observed 96% against a floor of one half, which leaves forty-six points earned out of fifty available, and forty-six over fifty is the 0.92 in the column. The same raw figure, scored against a floor less than half as high, becomes a completely different verdict. A statistic that reads 0.92 on balanced data and 0.15 on skewed data with the same raw agreement is behaving exactly as designed.

Shared bias, unequal reviewers and a wide interval#

Kappa measures reliability, which is whether the measurement process is repeatable. It says nothing about whether the thing being measured is the right thing. Two reviewers who have both internalised the same wrong reading of the policy will produce a kappa of 0.9 and a label set that systematically misses a whole category of violation. Reliability is necessary for validity and does not imply it, and the only way to check the second is to compare the labels against an outside consequence — an appeal that was upheld, an incident that was filed, a regulator's finding.

It also does not tell you which reviewer to believe. The table above is symmetric; nothing in it distinguishes the 18 items A flagged alone from the 22 items B flagged alone. If A is a specialist and B is a generalist, the disagreements may be almost entirely A catching real violations that B missed, and kappa reads exactly the same as it would if both were guessing. Sorting that out requires a third pass over the disagreements with an adjudicator, which is the subject of chapter 4, on diagnosing a disagreement set before paying to relabel it.

A third limitation is more mundane and bites often: kappa on 1,000 items with 44 flags is estimated from a very small effective sample, and its confidence interval is wide. A bootstrap — drawing a new sample of 1,000 items from these 1,000 with replacement, recomputing kappa, and repeating it a few thousand times to see how far the answer moves — puts the 95% interval at roughly 0.03 to 0.29, so the difference between "no agreement" and "some weak agreement" is inside the noise.

What a bootstrap is doing there deserves its own sentence, because it looks like a trick and is not. The question an interval answers is how much the estimate would have moved had you happened to label a different thousand items drawn from the same traffic, and you cannot answer that by labelling another thousand, since labelling another thousand is what you are trying to avoid. The bootstrap substitutes the sample you have for the population you cannot draw from: resampling those thousand items with replacement — so some appear twice and some not at all — manufactures a batch that differs from the original in the way a genuinely new draw would, and recomputing kappa on each such batch traces out how far the answer wanders. Where the wandering comes from is visible in this table. Almost every resampled batch contains a slightly different number of the four joint positives, and kappa's numerator is built almost entirely from those four items, so a batch that happens to draw three of them and one that happens to draw six produce visibly different answers. Four is a small number of items to hang a coefficient on, and an interval running from 0.03 to 0.29 is that fact expressed as a range.

Reporting 0.146 to three decimal places implies a precision that 4 joint positives cannot support. Counting on Consensus: Selecting the Right Inter-annotator Agreement Metric is the right reference for choosing a coefficient once you know your label distribution, and it is candid about how much these statistics wobble on skewed data.

This is also why "kappa is just conservative on subjective tasks, 0.2 is normal" is a dead end when someone offers it. It is a claim about the statistic when the evidence points at the data. If a task is genuinely so subjective that two trained reviewers cannot converge on the positive class, then the labels are not a reference set, and the honest conclusion is that the current label definition is not measurable rather than that the metric is harsh.

The pair of numbers establishes that the agreement on this label is thin and roughly how thin, and neither figure alone would have said so. The 96% is a true statement about a table whose mass sits in one cell; the 0.146 is a true statement about the four items that cell does not contain. Read together they say the reviewers can both recognise a clean output and cannot yet demonstrate that they recognise the same violations, which is a finding about the guideline rather than about either person.

What the pair cannot do is settle a decision, because the report as it stands cannot answer whether these labels are good enough to serve as the reference set everything downstream gets validated against. Chapter 3, on positive-class agreement, takes the same shape of data — 500 items, 93% raw agreement, kappa 0.21 — and works out which single additional number decides that.