Missed escalations reach users, false ones only cost money

The measurement took an afternoon, because most of it already existed. The replay harness from CI — the one that pulls a stratified sample of production requests, meaning a sample drawn to a quota per kind of request rather than in proportion to how often each kind occurs, runs them against a pinned model version and scores the outputs with a judge rubric — needed one change: run each sampled request through both models and keep both scores, alongside the escalation decision the live signal would have made. Two thousand requests, two model calls each, one judge call per output. Four hours of wall clock and about $34 of inference.

The result was a 2×2 table, and it was the first time anyone could say what the signal was doing.

Turning a replay into labels#

A label here is an answer to one question per request: would escalating this request have helped? That is deliberately narrower than "was the cheap answer correct", because a request the strong model also gets wrong is not one escalation would have rescued, and counting it as a miss inflates the problem with cases no threshold can fix.

Operationally the definition is a pair of judge scores against an acceptance bar:

NEEDS_ESCALATION = lambda cheap_score, strong_score: (
    cheap_score < ACCEPT and strong_score >= ACCEPT
)
ACCEPT = 4.0  # on the same 1-5 rubric CI already uses

rows = []
for req in stratified_sample(n=2000):
    cheap = run(CHEAP_MODEL_PINNED, req)
    strong = run(STRONG_MODEL_PINNED, req)
    rows.append({
        "id": req.id,
        "task": req.task,          # for the slicing in the next chapter
        "escalated": signal(cheap) < 0.70,      # what production would do
        "needed": NEEDS_ESCALATION(
            judge(req, cheap.text), judge(req, strong.text)
        ),
    })

df = pd.DataFrame(rows)
matrix = pd.crosstab(df.escalated, df.needed, margins=True)

Two things about the judge are worth settling before the numbers mean anything. It has to score each answer without knowing which model produced it, since a judge shown "model A (small)" and "model B (large)" will find reasons to prefer B; that means scoring each output independently against the rubric rather than presenting pairs. And its agreement with human labels has to be established at least once on a stratified subsample — 150 requests double-labelled by two engineers is enough to see whether the judge's acceptance decisions match human ones at 85% or at 60%, and 60% would mean the whole exercise is measuring the judge.

Chapter 3.16 established that apparatus. The only new part is running both arms on the same request so that each row carries the counterfactual, which is what converts a quality harness into a measurement of the escalation decision rather than of the system's output.

Two errors, unequal in every way that matters#

Read down the first column: 8% of traffic would have been improved by escalation, and the signal caught half of it. Read across the first row: of the 240 requests it escalated, 80 needed it, so two out of three escalations bought nothing. In the standard vocabulary those two readings are recall, the share of the requests that needed escalation which the signal actually caught, and precision, the share of its escalations that turned out to be warranted — 50% and 33% respectively.

It is worth saying plainly that this is better than nothing. A signal that escalated 12% of requests at random would catch 12% of the 160 that needed it — about 19 requests — for a recall of 12% and a precision of 8%. The self-reported confidence is roughly four times better than a coin on both axes, which fits the picture from chapter 3: it does detect the visible, hedging kind of uncertainty, and it is blind to the confident kind. The signal is weak rather than absent, and weak signals are the ones that survive longest in production because they produce plausible aggregate numbers.

Now price the two error cells, because they are not denominated in the same unit.

A false escalation costs one strong-model call on work the cheap model had already done acceptably. On the pricing from chapter 1 — 900 input tokens at $3 per million and 180 output tokens at $15 per million — that is $0.0054 per request. At 8% of 1.2 million monthly requests, 96,000 requests, the false escalations cost $518 a month. It appears on the invoice, it is bounded above by the always-strong bill, and it can be discovered by anyone who divides spend by escalation count.

A missed escalation costs a wrong answer delivered to a user. At 4% of 1.2 million requests that is 48,000 wrong answers a month, each one returned with a 200 status in 412 milliseconds, well-formed, fluent, and carrying no marker of any kind. There is no exception in the trace, no retry, no fallback event, no elevated latency and no unusual token count; the successful path and the failure path are byte-for-byte indistinguishable in telemetry. Chapter 15.3's machinery cannot help here, because there is no indicator to write an objective against — the failure does not emit anything.

The comparison only becomes a decision when somebody names a price for the second cell. That is uncomfortable and unavoidable: choosing a threshold is choosing an exchange rate between money and wrong answers, and refusing to name one means the exchange rate gets chosen implicitly by whoever picked 0.70.

def expected_cost_per_request(false_rate, missed_rate,
                              strong_call=0.0054, bad_answer=2.00):
    return false_rate * strong_call + missed_rate * bad_answer

expected_cost_per_request(0.08, 0.04)
# 0.08 * 0.0054  = $0.00043   false escalations
# 0.04 * 2.00    = $0.08000   missed escalations
# total          = $0.08043 per request

At $2.00 for a wrong answer — roughly the loaded cost of one support contact, which is a conservative figure for a customer-facing assistant and wildly optimistic for anything touching billing or compliance — missed escalations dominate the objective by a factor of 186. Even at $0.10 per wrong answer, which values a bad answer at less than twenty strong-model calls, they dominate by nine to one. There is no plausible price at which the invoice line matters more than the answers.

The move that looks like the fix and is not#

The obvious response to "half the requests that needed escalation did not escalate" is to lower the threshold, and the labelled data says how much that buys. The 80 missed requests came back with reported confidences distributed the same way as everything else: 34 at 0.95, 22 at 0.90, 9 at 0.85, 6 at 0.80, 4 at 0.75, 3 at 0.70, and 2 at 0.60 — the last two already escalating. Dropping the threshold from 0.70 to 0.60 catches three more of the eighty. Dropping it to 0.86 catches fifteen, and drags an additional 13% of traffic into the strong model, most of which did not need it.

Sliding the threshold up trades money for missed escalations at a terrible exchange rate, because the confidence values are close to independent of correctness in the region that matters. Escalating 56% of traffic — nearly half the always-strong bill — still leaves 2.1% of requests wrong and unescalated. The curve is shallow because the signal is weak, and no threshold on a weak signal becomes a strong one.

The fix is to change the signal rather than its threshold, and the labelled set is now the instrument for evaluating candidates. Deterministic checks go first because they are free and one-sided: on this workload, schema validation plus a check that every cited policy ID appears in the retrieved context caught 31 of the 80 missed requests on its own, at a cost of under two milliseconds and zero false escalations, because a citation that is not in the context is not a judgment call. A small verifier model scored against the same 2,000 labels caught 61 of 80 at a precision of 58%, adding about 180 milliseconds and $0.0002 per request. Those two together, evaluated on the same table, are a different system from a threshold nudge, and the reason to prefer them is visible in the same four cells rather than in an argument.

The false escalations do not need ignoring, only sequencing. Once the missed rate is at a level the business will accept, precision is what makes the saving larger, and it is the safe direction to optimise in because its failure mode is a bigger invoice.

Keeping the labelled set worth something#

Two thousand rows costing $34 is cheap enough that the interesting question is not whether to build the set but how to keep it from going stale, because a labelled set is an asset that depreciates.

Sampling deserves the same treatment the next chapter gives to evaluation samples: draw a floor per task rather than mirroring the traffic mix, since the point of the exercise is to characterise the escalation decision and the decision behaves differently on different work. A proportional sample of 2,000 gives the extraction workflow 60 rows, which is enough to estimate a 4% missed rate to a precision of roughly plus or minus five points — an interval wide enough to contain both "fine" and "catastrophic". Two hundred rows per named task, reweighted to traffic when computing the headline rates, costs about three times as much and produces intervals somebody can act on.

The set also has to be re-cut rather than reused indefinitely. The labels encode a snapshot of the workload, of both model versions, and of the judge, and every one of those moves. A practical cadence is a full re-label each quarter, plus an immediate re-label whenever the escalation signal changes, the cheap model's prompt changes materially, or either model version moves. Between those points, the continuous sampled comparison from the next chapters covers drift at lower confidence.

There is one hygiene rule that matters more than it sounds. Requests used for threshold selection and requests used for reporting the resulting error rates have to be different rows, because choosing a threshold to minimise cost on a labelled set and then quoting that set's error rates as the achieved performance is fitting and reporting on the same data. On a set of 2,000 with a signal this weak, the optimism is worth roughly a point of missed rate, which is a quarter of the number under discussion. A 70/30 split of the labelled set, with the threshold chosen on the first part and the rates quoted from the second, removes the problem for the cost of some arithmetic.

It is also worth naming the standard vocabulary, since the four cells are a confusion matrix — the conventional layout of a yes/no decision against a yes/no truth, with the two agreements on one diagonal and the two error types on the other — and the literature on cascade routing — the FrugalGPT line and the TMLS report on routing and cascade mechanics both do this — reports them as precision and recall of the escalation decision. Reporting precision and recall alongside the raw percentages makes the numbers comparable across teams, and the raw percentages of total traffic are what convert into money and wrong answers, so both belong in the write-up.

What the four cells still do not tell you#

The 4% missed rate is a lower bound in at least three ways, and it is worth carrying that in mind before quoting it.

The judge is a model, and its acceptance decisions inherit that model's blind spots, which overlap with the cheap model's blind spots more than anyone would like. If the judge cannot tell that the Brazilian retention answer from chapter 3 is wrong — and a judge without the policy documents in front of it cannot — then that request is labelled "did not need escalation" and disappears from the missed count entirely. Grounding the judge with the same retrieved context and giving it a rubric item about unsupported claims recovers some of this; nothing recovers all of it.

The label is defined against one specific version of the strong model, so "needed escalation" means "this version of the strong model would have done better", and that definition expires when the version does.

And the sample came from replayed production traffic, which is by definition traffic the current system already handles — new intents, a new customer segment or a new document type will be under-represented in exactly the proportion that they are new. The measurement describes the workload as it was during the sampling window.

There is a further limitation that the 2×2 cannot express at all, which is what happens when this whole comparison is run end to end as a quality test of the cascade against always using the strong model. That test has an aggregation problem severe enough to report equivalence for a system with a badly broken workflow in it.