Six tenths of a point on 400 cases is about two tickets

The slide has two bars on it. Support tickets are classified into billing, technical and account, and on a 400-case labelled set the single-pass classifier scores 91.2% while a two-agent debate with a judge scores 91.8%. Underneath, in smaller type, are the other two numbers: roughly six times the cost, and eight seconds of added latency at the median. Product has read the top line and wants to ship the debate version this sprint, and the argument in the room is about whether eight seconds is tolerable in the ticket intake flow.

That argument is premature, because the top line has not established anything. Converting the percentages back into tickets is the first thing to do: 91.2% of 400 is 365 correct, and 91.8% is 367. The entire result under discussion is two support tickets.

What two tickets out of 400 can and cannot tell you#

Two is small in the specific sense that matters, which is that it is small relative to how much the number moves when nothing about the systems changes. The standard error of a proportion near 0.91 on 400 samples is about 1.4 percentage points, so a 95% interval around the single-pass score — the band the true accuracy is very likely to lie in, given a sample of that size — runs from roughly 88.4% to 94.0%. The debate score sits comfortably inside it, and the single-pass score sits inside the debate's interval too. Two independent reruns with a different seed could easily produce 91.5% and 91.0% and reverse the ordering on the slide.

The interval comparison is actually the conservative version of the objection, and the paired test is sharper because both systems ran on the same 400 tickets. What matters there is not the two scores but the cases where the two systems disagreed, and those are the only cases carrying any information about which is better.

Sixteen tickets are decided differently. Debate fixed nine that the single pass got wrong and broke seven that it had right, for a net of two. Under the hypothesis that the two systems are equally good, those sixteen discordant cases should split roughly evenly, and a 9–7 split out of 16 is what a fair coin does most of the time.

from scipy.stats import binomtest

# discordant pairs only: debate right & single wrong = 9, reverse = 7
result = binomtest(k=9, n=16, p=0.5, alternative="two-sided")
print(result.pvalue)      # 0.8036

# how many labelled cases would it take to see a real 0.6pp gap?
# net gain per case = 0.006; sd of the net ~ sqrt(discordance rate) = sqrt(0.04)
# solve 0.006*N = 1.96 * 0.2 * sqrt(N)
N = (1.96 * 0.2 / 0.006) ** 2
print(round(N))           # 4268

A p-value of 0.80, which is the probability of seeing a split at least this lopsided if the two systems were in fact equally good, is not a near miss. It says the observed split is entirely ordinary under the assumption that the two systems perform identically, and the second calculation says that demonstrating a genuine 0.6-point improvement on this task would take on the order of 4,300 labelled tickets rather than 400. Whether it is worth labelling 4,300 tickets to detect an effect of that size is a reasonable question, and it is a different question from the one on the slide.

The three bugs it caught last week#

The same decision arrives in a second form that is harder to argue with, because it comes with stories. A colleague reports that the debate ensemble caught three specification bugs last week that the single-pass reviewer had missed, names them, and proposes rolling it out across the review pipeline. The three cases are real, the bugs were real, and the debate genuinely found them.

The problem is where those three came from. They are the cases someone noticed, and someone notices a case when the debate surfaced something interesting, which selects on exactly the outcome being used as evidence. Nobody keeps a list of the weeks the debate argued its way to a wrong conclusion that the single pass had right, because a wrong conclusion arriving with plausible reasoning does not announce itself. In the 400-ticket table above, seven tickets moved the wrong way, and if you had spent that week watching the pipeline you would have collected the nine wins and not the seven losses.

Three anecdotes are also uncountable against a base rate. Three per week out of how many? If the pipeline reviews 200 specifications a week, three catches is 1.5%, and the question is whether the single pass would have caught two of them on a rerun and whether the debate lost four elsewhere. The only structure that answers this is running both approaches over the same labelled cases and comparing on all of them, which converts a collection of memorable events into a rate.

Four numbers on one labelled set#

Error rate on its own is too thin to make the decision even when it moves, so the comparison worth running produces four columns from a single replay over the same cases.

The calibration rows carry the most information here, and they run against the debate version. Expected calibration error is the average gap between the confidence a system reports and how often it is actually right at that confidence, so 0.041 against 0.088 says the debate's stated numbers mean less. The judge reports confidence above 0.9 on 310 of 400 tickets against the single pass's 240, and is right 92.3% of the time in that bucket against 95.0%. The debate is not merely failing to improve accuracy; it is producing more confident output at a lower accuracy within its confident band, which is the property chapter 3 predicted from a judge selecting on how cleanly an exchange resolved. Any downstream routing that trusts a 0.9 to mean 0.9 gets worse when you ship this, and ticket intake almost always has such a rule.

Latency belongs in the same table rather than in a follow-up conversation, because eight seconds at the median is fourteen seconds at p95 for a user waiting on an intake form, and cost belongs there because $128 per thousand tickets against $4.50 is the number the accuracy gain has to be worth.

Routing the hard cases in makes it affordable, not true#

The natural compromise at this point is to send only low-confidence cases into the debate and keep the cheap path everywhere else. If the single pass reports confidence below 0.8 on 12% of tickets, routing that 12% into a 6x path gives a blended multiplier of about 1.5x, and the latency cost lands only on cases that were going to be hard anyway. Selective routing is a good pattern and the arithmetic is right.

It answers the wrong objection. The cost figure was never the reason to hesitate; the reason was that no improvement has been demonstrated, and making an undemonstrated improvement cheaper does not demonstrate it. What the compromise produces is a permanent second classification path — a threshold to tune, two prompt sets to maintain, two latency profiles, two sets of incidents — carried on the strength of a two-ticket difference that a different random seed could reverse.

The routing also makes the measurement harder rather than easier, which is the part that tends to go unnoticed. Once only 12% of traffic goes through debate, any future evaluation of the debate arm is running on 48 of the 400 cases, and the discordant set on 48 cases will be three or four tickets. Whatever chance there was of ever settling the question on production data has been reduced by a factor of eight. If the effect is real, the routed subset is where it should be largest, and that is an argument for evaluating the debate arm on the low-confidence subset specifically before routing to it, with enough labelled low-confidence cases to see something.

Look at what both of them get wrong#

The 26 tickets both systems got wrong are more informative than the 16 they disagreed on, and they are the thing to ask for before any decision. On a three-way split with 91% accuracy, the residual is rarely a scatter of unrelated mistakes.

Reading those 26 in the support case gave the following. Twenty-one were tickets where the gold label itself was contested, the gold label being the answer the evaluation set treats as correct and marks everything else against: a customer reporting that they cannot log in because their subscription lapsed is technical to one annotator and billing to another, and the label in the set is whichever annotator happened to see it. Three were tickets containing two distinct issues, where the taxonomy has no way to express both and any single label is half right. Two were genuine model errors of the kind an ensemble might fix, and one of those two the debate did fix.

That distribution is the usual one, and it changes what to do next. Twenty-four of the 26 residual errors are properties of the label space rather than of the classifier, so no amount of inference — three samples, two debaters, ten rounds — moves them, because the systems are being marked wrong for producing a defensible answer. The available headroom above 91.2% is roughly 2 points wide, not 8.8, and the work that recovers it is a taxonomy change: adding a billing-access category, or allowing a second label, or writing the annotation guideline that decides the lapsed-subscription case one way and re-labelling accordingly. That is cheaper than 6x inference and it raises the ceiling for every future version of the classifier rather than for one of them.

What this is not an argument against#

Small effects are not automatically unreal or unimportant. A 0.6-point improvement on a pipeline handling 400,000 tickets a year is 2,400 tickets routed correctly, and that can be worth a great deal. The claim here is narrower: a 0.6-point difference measured on 400 cases has not been observed, in the sense that the measurement cannot tell it apart from zero. If the effect matters at scale, the correct response is to build an evaluation large enough to see it, which the arithmetic above sizes at a few thousand labelled cases.

Nor is it an argument that debate lost. The debate arm did not underperform; it produced a result indistinguishable from the cheap arm, which is a different finding from being worse and it points somewhere specific. When two designs are indistinguishable on accuracy and one costs 28x the tokens, the burden sits with the expensive one, and the calibration column is the only place in this comparison where a clear difference appeared at all.

All of this rests on having 400 labelled cases to argue over, and that is the thing the system most in need of this analysis does not have. The compliance pipeline from chapter 3 has run for six months on 9,000 documents with no labels at all, so none of these tables can be filled in for it. Producing the labelled sample is the next problem, and it is the part of the work that involves no model calls, which is why it gets skipped.