Two reviewers applying the same wrong rule agree perfectly

The intent classifier has been at 88% on the held-out set since March. There have been four rounds of data collection since then, each adding 15,000 labelled examples from an outsourced team at $0.42 an item, so the project has spent $25,200 and three months to move accuracy from 88.0% to 88.1%, with 87.9% and 88.2% in between. The quality dashboard shows inter-reviewer agreement at 0.86, computed per class, stable across all four rounds, with no class below 0.7. The team's conclusion at the retro is that the data is fine and the ceiling is the model, so the next quarter's proposal is a larger encoder and a better tokeniser.

The picture they are reading is also exactly what a shared, stable, systematic labelling error looks like. Agreement asks whether two reviewers reach the same answer. It has nothing to say about whether the answer is the one the project needs, and two people applying the same wrong rule from the same guideline agree with each other completely, forever, at no cost in throughput.

The rule that produced the plateau#

The vendor's reviewers had settled on a heuristic in week two, and it was a reasonable one. The guideline for order_status said it covered "any enquiry about the state, location or timing of a specific order", and virtually every ticket that quotes an order number is asking about a specific order in some sense, so the working rule became: if there is an order number in the body, it is order_status. That rule is fast, it is unambiguous, and two reviewers applying it will never disagree. It absorbed a slice of returns ("I want to send back order 44192, it doesn't fit"), most of shipping_damage ("order 51023 arrived with the box crushed"), and a portion of billing_dispute ("I was charged twice for order 39877").

The held-out set does not use that rule, because it was built at project kickoff by the two engineers and the support lead, before the vendor existed, and its labels follow the resolution-ownership convention the taxonomy was designed around. So the training corpus and the evaluation set now encode two different definitions of order_status, and they differ on roughly 12% of items. A model trained on 60,000 examples of the vendor's convention learns the vendor's convention very well and is then marked against a set that disagrees with it on 12% of items, which puts a hard ceiling at 88% no matter how much more of the same data arrives.

That is why the number is not merely low but flat. A model short of data improves when you add data; the curve bends. A model that has fully learned a rule which contradicts the test labels on a fixed slice sits still, because each new round of 15,000 examples reinforces the same contradiction at the same rate. The remarkable stability of 88.0, 88.2, 87.9, 88.1 across three months is itself the signal, and it was read as evidence of a robust measurement.

Sorting the residuals in twenty minutes#

The cheapest way to test this does not require any new labelling. Take the model's errors on the held-out set and ask whether they concentrate on a feature that a reviewer might plausibly have keyed on.

import re
from collections import Counter

ORDER_RE = re.compile(r"\b(order\s*#?\s*\d{5,}|#\d{5,})\b", re.I)

errors = [ex for ex in heldout if model.predict(ex.text) != ex.label]
print(len(heldout), len(errors))
# 3000 357   -> 88.1% accuracy

sliced = Counter(
    (bool(ORDER_RE.search(ex.text)), ex.label, model.predict(ex.text))
    for ex in errors
)
for (has_order, gold, pred), n in sliced.most_common(6):
    print(f"order_no={has_order!s:<5} gold={gold:<16} pred={pred:<16} n={n}")

# order_no=True  gold=returns          pred=order_status     n=148
# order_no=True  gold=shipping_damage  pred=order_status     n=112
# order_no=True  gold=billing_dispute  pred=order_status     n=71
# order_no=False gold=cancel_request   pred=order_status     n=9
# order_no=False gold=returns          pred=account_update   n=8
# order_no=True  gold=password_reset   pred=order_status     n=5

Of 357 errors, 331 are the same error: an item containing an order number, predicted order_status, labelled as something else in the held-out set. That is 93% of the model's residual mass in one direction, on one surface feature. A model whose mistakes are that structured is not a model that has run out of capacity; it has learned something specific and is being marked wrong for it consistently.

The complementary check takes about as long. Sample 50 training items that contain an order number and are labelled order_status, and read them. If a third of them are people asking to return a product, the vendor's convention is confirmed directly from the corpus, without a single new annotation.

A third check costs an afternoon of GPU time and settles the "not enough data" hypothesis on its own terms. Train the same recipe on 15,000, 30,000, 45,000 and 60,000 examples and plot the curve. On this project it came back 88.0, 88.1, 88.0, 88.1, which is a flat line from the first round onward. A model genuinely short of data produces a curve that bends — something like 84.2, 86.5, 87.8, 88.4, still rising and visibly decelerating — and the shape tells you roughly how much another round would be worth. A line that is flat from 15,000 examples says that whatever the last 45,000 items contained, the model had already extracted all of it from the first 15,000, which is exactly what happens when every round encodes the same convention and the convention is the thing being measured against.

Neither check proves the held-out labels are the correct ones. Both conventions are defensible in the abstract, and it is entirely possible that the vendor's rule is the better one and the kickoff labels are the defective ones. What the checks establish is that there are two conventions in the pipeline, which is a fact about the data and not a fact about the model, and no amount of encoder size resolves it.

What a healthy agreement number does and does not rule out#

The mistake worth naming is treating agreement as a summary of data quality rather than as a summary of one specific property of the data. When a stakeholder asks "are the labels good", 0.86 answers a narrower question than the one asked, and the gap between the two questions is where three months went.

The row that matters is the fourth one, and it is the only row where the healthy agreement number is not neutral evidence but evidence pointing the wrong way. A reviewer pool that has converged on a crisp, wrong rule scores better on agreement than a pool that is faithfully wrestling with genuinely hard boundaries, because wrestling produces disagreement and a crisp rule does not. Agreement rewards the failure mode.

The other thing to be clear about is that this is not an argument against measuring agreement. The previous chapter's audit — the one that found 26 of 36 disagreements sitting on a single ticket shape — is only possible because agreement is measured and the disagreements are kept. Agreement is a working instrument for the class of defects that produce inconsistency. It is blind to the class of defects that produce consistency, and those are the ones that are stable enough to survive four rounds and cheap enough for a vendor to standardise on.

How the rule got in, and why nobody saw it#

The order-number heuristic was not smuggled in. It appeared in week two, in a one-page onboarding sheet the vendor's team lead wrote for new joiners, under a heading called "quick wins" alongside four other shortcuts that were all perfectly sound. The sheet was never sent to the client, because nothing in the engagement asked for it and there was no obvious reason to; it was training material for the vendor's own staff, of the kind any competent operations lead produces.

The unresolved queue that would have caught it stayed empty for a different reason. The engagement had no adjudication path back to the guideline owner, so a reviewer with a question could ask their team lead and get an answer in two minutes, or ask the client and get an answer in three days. Every rational reviewer asked their team lead, the team lead answered from the shortcut sheet, and the client's guideline stopped being the thing that decided anything by about week three. The client's total visibility into all of this was an agreement dashboard that showed 0.86 and did not move.

The cheapest control that would have caught it is unglamorous: somebody on the client side reads 25 randomly drawn training items per round, with the label visible, and writes down anything that surprises them. Twenty-five items at two minutes each is 50 minutes per round, four times over three months, so about three and a half hours across the entire project against the $25,200 that was spent. On this corpus, where the defect touched 12% of items, a 25-item sample has a 96% chance of containing at least one instance, and the instances are not subtle — a ticket that plainly says "I want to send this back" carrying the label order_status reads wrong immediately to anyone who knows what the classes are for.

The contract was paying for exactly this#

The vendor contract pays per item delivered, with the quality bar defined as inter-reviewer agreement above 0.8. Read as an incentive, that pair of terms says: produce as many items as possible, subject to your reviewers matching each other. Nothing in it references the guideline, the downstream model, or whether a label is right.

A rational team optimises it in the obvious way. Simple deterministic rules go up, since a rule like "order number means order_status" is both faster than reading for intent and perfectly agreeable. Hard cases get resolved downwards to whichever label is quickest to justify, because time spent deliberating costs throughput and deliberation is the main source of disagreement. Flagging an item as unresolved becomes actively costly, since it does not count as a delivered item and, depending on how the metric is computed, may show up as a disagreement. Questions to the guideline owner drop toward zero, which the client experiences as a low-maintenance vendor. If the double-labelled subset is identifiable in advance — a separate batch, a different filename, a queue that appears on a schedule — effort concentrates there, and the measured agreement stops describing the rest of the delivery at all.

None of that requires anyone to act in bad faith. It is what the contract asks for, delivered competently. A contract that wanted correctness would have to pay for something that correctness can be measured against, which means a reference sample the vendor does not control, a defect rate defined against it, and a share of the fee attached to that rate rather than to the agreement floor. It would also have to fund the unresolved queue, so that flagging an item the guideline does not settle is a paid outcome instead of a personal loss.

Where this leaves the diagnosis#

The residual analysis and the training-sample read are strong circumstantial evidence and they cost an afternoon, which is the right trade when the alternative on the table is a quarter of model work. They do not settle the question, because both of them lean on the held-out set being right, and the held-out set was produced by three people at kickoff under time pressure, using a guideline that has since been revised four times.

Settling it requires a measurement that has authority over the convention rather than merely being older than the current one, and it requires the price list that turns the finding into a decision: which items get re-labelled, who re-labels them, whether the class survives in its current form, and what any of that costs.