Agreement as a routing signal

Once a service is sampling N times, the vote share is already being logged and it looks like a confidence score: five out of five means sure, three out of five means doubtful. Routing on it is then a sprint rather than a project — send the doubtful ones to a human, keep the confident ones automatic, and accuracy on the automatic path should rise because the hard cases have been taken out of it.

Whether that works is a property of the task, not of the technique, and it is measurable before any queue is built. Calibration is the property that a signal's reported level tracks the rate at which the answer is actually right, so that four-fifths agreement means something dependable about how often four-fifths agreement is correct. The test takes an afternoon on labelled data that already exists, and it answers a question that a routing rule cannot answer once it is running.

Bucket by agreement, then look at correctness#

Group held-out cases by how much the samples agreed and compute observed accuracy within each group. What you are looking for is the slope: does accuracy fall as agreement falls, and by how much? And then the column most reports leave out, which is where the errors actually sit.

def calibration(cases, samples, gold) -> list[dict]:
    """Observed accuracy and error mass by agreement level."""
    buckets: dict[int, list[bool]] = {}
    for case in cases:
        draws = samples[case.id]
        winner, votes = Counter(draws).most_common(1)[0]
        buckets.setdefault(votes, []).append(winner == gold[case.id])
    total_errors = sum(f.count(False) for f in buckets.values())
    rows = []
    for votes in sorted(buckets, reverse=True):
        flags = buckets[votes]
        rows.append({
            "agreement": f"{votes}/5",
            "share_of_traffic": len(flags) / len(cases),
            "accuracy": sum(flags) / len(flags),
            "share_of_all_errors": flags.count(False) / total_errors,
        })
    return rows

A claims triage service produced this.

Accuracy does decline as agreement declines, which is why the signal looks usable to anyone who glances at the middle column: the ordering is monotone and the bottom bucket is eleven points worse than the top. What decides the routing question is the last column. Unanimous cases carry 88.6% of all the errors in the system, because they are 92% of traffic and their error rate is only slightly below average. Escalating everything else removes 11.4% of the errors at a cost of reviewing 8% of traffic — barely better than escalating 8% of items chosen at random, which would remove 8% of the errors.

The share-of-all-errors column is not an extra measurement. It is the two columns beside it multiplied together and then normalised, and the multiplication is the entire argument. How many errors a bucket contributes is how much traffic lands in it times how often the answer there is wrong, because an error rate is a rate — it only becomes a count when something supplies the volume to apply it to. The top row supplies a great deal of volume: ninety-two items in every hundred, each wrong a shade over a tenth of the time, which is a little over ten wrong answers per hundred items of traffic. The bottom row supplies almost none: one item in every hundred, wrong a bit more than a fifth of the time, which is roughly two wrong answers per thousand items of traffic. Divide each of those products by the sum of all four and you have the last column, which is why the top row reads 88.6% and the bottom row reads 1.9%.

The stumble here is not an arithmetic slip, and calling it careless would be unfair. It is the belief that the bucket where the model is worst is the bucket worth reviewing. That is tempting because it is where the model is worst, and because every dashboard in the building is built to surface the lowest number on a chart. It is false because a review queue is spent per item and not per percentage point. What a reviewer's hour buys is a wrong answer turned right, so the bucket worth reviewing is the one holding the most wrong answers, which is a count. The 5/5 row has the highest accuracy in the table and the most errors in the table, and both of those statements are consequences of the same number in the second column.

The random baseline that 11.4% was compared against is worth being explicit about, because it is the thing the rule has to beat and it is easy to mistake for zero. A rule that picks 8% of items without looking at anything relevant catches, on average, 8% of the errors, since errors that the selection ignores are spread through the escalated sample in the same proportion as they are spread through traffic. So the informative part of this rule is not the 11.4%. It is the gap between 11.4% and 8%, which is a bit over three points of extra capture, bought at the price of putting one item in roughly every dozen in front of a person.

The capture curve, and what its shape tells you#

The right way to look at this is the capture curve: for an escalation budget of X% of traffic, what fraction of the system's errors does the routing rule catch? A rule with no information sits on the diagonal, catching X% of errors for X% of budget. The ratio between the two is the lift.

Every lift column there is one division. Claims triage caught 11.4% of errors for an 8% budget, and 11.4 divided by 8 is the 1.4× beside it; clause review caught 34% for the same budget, and 34 divided by 8 is its 4.3×. Dividing by the budget is what makes two services with different escalation rates comparable, and it puts the null result at one rather than at zero, because the uninformative rule catches exactly its own budget.

Reading the shape means comparing successive slices of budget rather than reading the column downwards, because the column downwards always rises and a rising column proves nothing. On clause review the first four points of budget buy 21% of the errors, which is a bit over five points of capture per point spent. The next four points buy thirteen more, which is around three per point. The eight after that buy fifteen more, under two per point. Each slice is worth less than the one before it, and that declining sequence is the bend: the signal put the genuinely suspect items at the very top of its ranking, so the budget that reaches them first is the budget that buys most. On claims triage the same three slices buy 5.6%, then a shade under six points more, then nine and a half points more, which works out at about one and a half points of capture per point of budget every time. Nothing is concentrated at the top of that ranking, because there is nothing in the ranking.

The two services behave differently in a way that is visible from the first row and does not require building anything. On the clause-review service the curve is steep at the left and bends over, which is what an informative signal looks like: the first slice of budget buys disproportionately, and lift falls as you dig further into the ranking. That routing paid for itself in a month. On claims triage the curve is a straight line through the origin at a slope of roughly 1.4, so every additional reviewer hour buys the same small fraction of errors as the one before it, and there is no budget at which the rule becomes a good deal. That shape is the tell, and it is worth plotting before anyone argues about where to put the threshold.

Why the top bucket holds the errors#

Two things are happening in that first row, and they pull in the same direction. The first is that when the error comes from a systematic misconception or an ambiguous instruction rather than from sampling, all five samples reproduce it and the vote is unanimous. Those cases are not merely uncaught by an agreement-based filter — they are the cases the filter most confidently waves through, because unanimity is exactly what a shared error produces.

The chain under that sentence is worth walking one link at a time, because it is the reason the signal cannot be fixed by tuning it. Each sample is a draw from the model's distribution over answers for that input. Counting how many draws agree therefore measures how spread out that distribution is, and nothing else — a unanimous vote says the distribution has almost all its mass on one answer. Where that mass sits is a separate fact, and a systematic misconception moves the mass rather than spreading it: if the instruction is ambiguous, or the model holds a wrong belief about how a policy category works, every draw is made under the same ambiguity and the same wrong belief, so every draw lands on the same wrong answer. Sharp and wrong is an ordinary state for a distribution to be in. Agreement reports sharpness, the routing rule needs wrongness, and no threshold on the first quantity can recover the second.

The second of the two things is a matter of where the volume is. Even if unanimous cases had a genuinely low error rate, they are 92% of traffic, so a modest rate over a large base outweighs a high rate over a small one. Getting the majority of your errors out of the auto path through this mechanism would require the non-unanimous bucket to be both large and dramatically worse, and the same sampling behaviour that makes the model good enough to deploy makes that bucket small.

The complementary half of the diagnosis is what disagreement actually selects. Pulling fifty non-unanimous claims and reading them showed the pattern clearly: they were long submissions with multiple injuries, claims that straddled two policy categories, and documents with poor scan quality — genuinely hard items where reasonable annotators also disagreed. The model got 83% of them right anyway. Disagreement is a measure of how difficult the model found the input, and difficulty correlates with error only weakly, because the model is frequently right about hard things and occasionally very wrong about easy ones.

Captured is not corrected#

Even the 11.4% figure flatters the rule, because capturing an error in the review queue is not the same as fixing it. What escalation returns is the captured share multiplied by how much better the reviewer is than the model on exactly the population being escalated, and that second factor is routinely assumed to be one.

Assuming it is one means assuming the reviewer is right by definition, and the arithmetic that follows from that assumption is a count of the errors sent to the queue. The arithmetic that follows from measuring it instead is a subtraction, and the subtraction is the part that gets skipped. A reviewer who is right on some share of escalated items and a model that is right on some other share both make mistakes on the same population, and the queue only changes an outcome where their verdicts differ. Where both are right, nothing moves. Where both are wrong, nothing moves either — the reviewer confirms the model's answer, or replaces it with a different wrong one. Where the reviewer is right and the model was wrong, an error is repaired. Where the model was right and the reviewer is wrong, an error is created, and that last case is real rather than a technicality. What survives is the difference between the two accuracies, on that population, and it is smaller than either of them.

The reviewer's lift is smallest exactly where an agreement-based rule sends work, and for the same reason the samples disagreed. Multi-injury submissions and scans that straddle two policy categories are hard for annotators too: inter-annotator agreement on the escalated population was 0.71 by Cohen's kappa — a score for how often two humans give the same label, corrected for the agreement they would reach by guessing, where 1.0 is perfect and 0 is no better than chance — against 0.93 on the auto path. Sizing a queue on error capture alone assumes a reviewer who is right by definition, so measuring reviewer accuracy on a sample of the escalated population, before the queue is staffed, belongs in the same afternoon as the calibration itself.

What a usable result looks like#

Not every task behaves this way, and the point of running the calibration is that you cannot tell in advance which one you have. A signal worth routing on shows four properties, and it is worth writing them down as acceptance criteria before the experiment, so that the result decides the project rather than the project deciding what counts as a good result.

Running the claims service through those four rows shows why writing them down in advance matters. Its slope is eleven points, which clears the "under 10 points" disqualification and comes nowhere near the 25 that would make it usable — it lands in the gap between the two columns, which is exactly the region where a team argues about whether the number is encouraging. Its concentration is 1.4×, comfortably inside the "under 1.5×" column, and that is the row that settles it. The reason concentration decides and slope does not is the one the error-mass column already made: slope is a statement about rates and concentration is a statement about counts, and a review queue is paid for in counts. The 1.5× floor is where it is because a lift below it means a reviewer hour buys less than half again what an hour spent on randomly chosen items would buy, and both hours cost the same.

The monotonicity row is where per-slice discipline returns with consequences. A signal can look monotone in aggregate and be inverted on a slice — high agreement, low accuracy — and routing on the aggregate then systematically under-escalates exactly the population that needed review. Checking the calibration per slice costs nothing beyond a group by and is the single most informative version of this measurement.

Sample size is the practical constraint#

The bottom bucket is 1% of traffic, so a held-out set of 2,000 labelled cases puts twenty items in it, and an accuracy estimate on twenty items has a standard error of about nine points. Establishing the slope credibly needs either a much larger labelled set or stratified sampling, which means spending the labelling budget deliberately across the buckets rather than uniformly across traffic, over-sampling the thin low-agreement ones and reweighting the results afterwards to match the real mix. Stratified sampling is the cheaper route and is worth setting up permanently, because the calibration has to be recomputed on every model change.

Both of those numbers come out of one line of arithmetic each, and the second is the one that decides how much of the table to believe. Twenty items is 1% of 2,000, which is the whole of the first step. The nine points is the standard error of a proportion: multiply the accuracy by one minus the accuracy, divide by the number of items in the bucket, and take the square root. With an accuracy near four fifths and twenty items in hand, that comes out near nine points. The count sits under a square root, so the estimate improves slowly with more labels and degrades fast with fewer — quartering a bucket's sample doubles its error bar, which is exactly what happens as the buckets thin out towards the bottom of the table.

Nine points either side of 78.0% is a band wide enough to contain 88.9%, and 88.9% is the top row. On a 2,000-case held-out set the bottom bucket is therefore not measurably worse than the bucket it is supposed to be worse than, and the eleven-point slope that made the middle column look promising is smaller than the noise on the number producing it. The four properties above were written as acceptance criteria for a reason, and this is the one most likely to be quietly failed: a table can satisfy slope, concentration and monotonicity on paper while every row below the first rests on a few dozen labels.

Setting the operating point once the signal passes#

When the capture curve does show real lift, the threshold is determined by capacity rather than by the shape of the curve. Reviewers can handle a fixed number of items per day, so the escalation rate is fixed by that, and the threshold is whatever produces it. Setting it the other way around — choosing "escalate anything below 4/5" because it sounds principled — produces a volume that the queue either cannot absorb or does not fill.

Two details keep this working in production. The threshold has to be enforced as a rate rather than a fixed cut-off, because the agreement distribution drifts with traffic mix and with model versions, and a fixed cut-off that escalates 8% in March will escalate 19% in June after an upgrade changes how sharply the model samples. Ranking by agreement and taking the worst K per hour is more stable than thresholding, at the cost of some latency for items that have to wait to be ranked. Alongside that, the escalated items need to keep being labelled, since they are the only continuing measurement of whether the signal still works; if reviewers correct items without their verdicts flowing back into the calibration, the routing rule becomes unauditable within a quarter.

The drift is not a quirk of one upgrade, and the mechanism behind it is the same one that made the signal weak in the first place. A cut-off on vote share is a cut-off on how sharp the model's answer distribution is. That sharpness is a property of the model, not of the item in front of it, so a change to the model moves every item's vote share at once. A version that samples more sharply pushes items that used to come back 4/5 up to 5/5 and shrinks the escalated population; a version that samples more loosely does the reverse and floods the queue. Neither change says anything about whether the items are harder or the answers are worse. Ranking by agreement and taking the worst K survives this because a ranking only depends on the order of the scores, and the order is far more stable under a model change than the absolute values are.

The dotted edge is the one that gets dropped first and matters most. Labels arrive for free on the escalated path because a human touched every item there, so a team that audits nothing else still knows the accuracy of the 8% it reviews. The auto path produces no labels at all unless someone deliberately samples it, and without those the top bucket's accuracy is unmeasurable — which is precisely the number that decides whether the rule is worth running. A few hundred audited auto-path items a week is enough to keep the first row of the calibration table alive, and it is the cheapest line item in the whole arrangement.

Vote share is not a probability#

The fraction of samples agreeing is not a probability that the answer is correct, and treating it as one causes trouble downstream when it gets exposed as a confidence score. A 5/5 vote on the claims service corresponded to 88.9% observed accuracy, not 100%, and a 3/5 vote corresponded to 83.5%, not 60%. The mapping from vote share to accuracy is task-specific, is compressed into a narrow band at the top, and has to be fit on labelled data if you want to publish a number. Handing agreement=1.0 to another team that will read it as certainty is a reliable way to have a wrong answer treated as authoritative.

Laying the two scales side by side shows how far apart they are. Read as a probability, the vote share runs from certainty at 5/5 down to three chances in five at 3/5, which is a drop of forty points. Read against what actually happened, the same two buckets are 88.9% and 83.5%, which is a drop of about five. The nominal scale therefore travels seven or eight times as far as the real one over the same range of votes, and almost all of the real variation is squeezed into the narrow band near the top. Even stretching to the bottom of the table, where no answer wins a plurality at all, only reaches 78.0% — so the entire observable range of this signal is the eleven points already noted, sitting under a label that promises a hundred.

It is also worth separating this from the model's own token-level confidence, which is a different signal with different failure modes and is sometimes better. Agreement measures the spread of outcomes under resampling; logprobs measure the sharpness of the distribution at each step. They can disagree, and where a task has short discrete answers, average logprob on the answer tokens is available at zero additional cost and is worth putting on the same capture curve before spending anything on N.

Two columns decide this, then, and only one of them is the one people look at. Accuracy by agreement level tells you the signal has a slope, which is necessary and means very little on its own. Share of errors by agreement level tells you whether the slope is anywhere near the volume, and that is the column that says whether a review queue will earn its money. Add the reviewer's accuracy on the escalated population and you have the whole answer before a queue exists: errors captured, times the reviewer's lift over the model on that population, against the cost of the reviews and the price of an error.

Ending a quarter with a routing rule that does not work, a review queue sized for it, and the knowledge that agreement is not going to be the thing that feeds it is a common enough outcome to be the normal one. It leaves a specific question: what do you build to get an uncertainty signal when the generator's own agreement will not supply one?