The router should cost a fraction of the path it picks

The spend breakdown for October put the router at $2,415 against $5,250 for every specialist call combined, on 210,000 requests. Dividing each figure by the request count turns those totals into per-request prices: $0.0115 for the classification and $0.025 for the answer, so the decision about which prompt to use costs almost half of what the prompt itself costs. The latency trace says the same thing in a different unit. Median time to first token — the interval between the customer's message arriving and the first word of the reply appearing — is 4.1 seconds for the whole turn, and 700ms of that elapses before the specialist has received anything at all, because the router is a full round trip to claude-opus-5 that must complete before the second call can start.

What the router actually does with those resources is emit one of five strings. Its input is a support message averaging 120 tokens plus a 2,200-token system prompt of class definitions and boundary examples; its output is about 20 tokens of JSON. It is a five-way classification of short text, which is a task that predates large language models by several decades and which smaller models are extremely good at. The premium is being paid for capability that the task does not consume.

Three routers, measured against the same file#

The labelled route set makes this a comparison rather than an argument. Each candidate router runs over the same 800 items, produces the same confusion matrix, and gets scored on the same two views — traffic-weighted accuracy, and the recall on api_integration, which is the class where the boundary is genuinely hard.

Two of the three candidates are prompted language models of different sizes. The third is not a language model at all. An embedding model converts a piece of text into a fixed-length list of numbers, a few hundred to a few thousand of them, arranged so that texts with similar meaning end up close together under ordinary distance arithmetic; a logistic regression fitted on the labelled route set then learns where to draw the dividing surfaces between classes in that space, and reports a probability for each class rather than a bare label. It runs on a CPU in tens of milliseconds and costs almost nothing per call, which is why it belongs in the comparison even though its accuracy is the lowest of the three. The latency figures below are given as p50 and p95: the median, and the value that 95% of requests come in under, which is the one a latency budget is actually written against.

The aggregate row is the one that tempts people into a bad decision. Two and a half points of accuracy for a 190× cost reduction and a 15× latency reduction looks like an obvious trade, and if the aggregate were the whole story it would be.

The api_integration row is where the trade actually lives: the embedding classifier gives up 11.6 points on the one boundary that matters. The reason it gives up so much there and almost nothing elsewhere is a property of what an embedding measures. An embedding places texts near each other when their wording and subject matter are similar, so a classifier drawing surfaces in that space can only separate two classes whose messages sit in different regions. "We were charged for something our integration did" sits close to billing language in every embedding space, because it is billing language, and the thing that resolves the class is a piece of reasoning about who caused the charge — a judgement about the situation the words describe rather than about the words. The classes that are easy stay easy at every price point, because their vocabulary already separates them. The class that is hard degrades fastest exactly where the system can least afford it.

That is also the shape of the belief this chapter exists to break. "Routing is easy, so accuracy will hold" recommends the correct change for the wrong reason, and the reason matters because it is the thing that will be reused on the next taxonomy. Its premise is a claim about routing in general when the property in question belongs to individual boundaries. On this taxonomy incident_status is trivially separable and survives a 190× cost cut untouched, while billing versus api_integration was hard for two humans who answer these tickets professionally and is hard for every classifier in the table. Anyone who reasons from "routing is easy" to "the cheap model will be fine" is right on four boundaries and wrong on the one that produces the complaints, and they will find out from the complaint volume rather than from a measurement.

The aggregate accuracy is the quantity that stays calm while the dangerous one moves. It falls by two and a half points across the whole table, which reads as a rounding error, while the recall on the boundary that generates the complaints falls by more than eleven. Any single number that averages a hard boundary together with four easy ones will behave this way, which is why the per-class view from the previous chapter is not an optional refinement here — it is the only view in which this decision is visible at all.

Escalate the cases the cheap router is unsure about#

The way to keep the cheap path's economics and the expensive path's judgement is to route the routing decision: let the cheap classifier decide, and send only the cases where it is not confident to the large model. That requires a confidence signal, and where it comes from depends on which cheap router you picked.

A logistic regression over embeddings hands you a probability vector directly, and the useful quantity there is the margin, the gap between the highest-scoring class and the runner-up, rather than the top probability on its own. The difference is easiest to see on two cases. A message the classifier scores at 0.52 billing / 0.48 integration is a coin flip on the hard boundary: the winner is barely ahead, so a small change in wording would flip the label. A message at 0.55 how_to / 0.11 billing has a similar top score and nothing near it, so it is a confident-enough decision on a diffuse class even though the winner is far from certain. Margin separates those two situations; the top-1 probability — the score of the winning class read on its own, with no reference to what came second — puts them within three hundredths of each other. What you want to escalate is contested decisions, not diffuse ones, and margin is the quantity that measures contest.

A prompted small model gives you a label and nothing else, so you have to ask for the extra signal — a second-choice class and a self-reported confidence — and then check whether the self-report means anything. The check is a bucketing exercise on the labelled set: group the items by the confidence the model reported and compute accuracy within each group. If items the model called high are 96% correct and items it called medium are 79% correct, the signal is usable as a coarse gate even though the words themselves are not calibrated probabilities, calibrated meaning that a reported 0.9 would turn out correct about 90% of the time. If both buckets come back near 93%, the model is reporting confidence it does not have, since a label that does not partition the set by accuracy carries no information about accuracy, and you should use the margin from an embedding model instead, or accept the cheap router's error rate as-is.

ESCALATE_BELOW = 0.25   # margin threshold, chosen from the labelled set

def classify(message: str) -> tuple[str, str]:
    scores = embedding_classifier.predict_proba(message)     # {route: p}
    ranked = sorted(scores.items(), key=lambda kv: -kv[1])
    (top, p1), (_, p2) = ranked[0], ranked[1]

    if p1 - p2 >= ESCALATE_BELOW:
        return top, "cheap"

    return classify_with_opus(message), "escalated"          # the 700ms path

The threshold is the part that has to come from data rather than from taste, and the labelled set already contains everything needed to choose it. Sweep the threshold from 0 to 1, and at each value record two quantities: what fraction of the set escalates, and what the overall accuracy would be if escalated items were decided by the large model. The curve is steep at the low end and flat afterwards, because the contested items are concentrated in a narrow margin band — which is another way of saying that most requests are not close calls, so raising the threshold past the band starts sweeping in decisions the cheap classifier was already getting right. On this set, a threshold of 0.25 escalates 12% of traffic and lifts overall accuracy from 91.4% to 93.8%, recovering the entire gap to the large-model router, while a threshold of 0.45 escalates 31% for another 0.3 points. The knee is the setting; everything past it buys almost nothing at triple the escalation rate.

The blended economics follow from one weighted average. With 88% of requests taking the 45ms, $0.06-per-thousand path and 12% taking that plus the full large-model call, latency averages 129ms added with a p50 of 45ms, since the mean is the 45ms cheap path weighted by 88% plus the 745ms escalated path weighted by 12%, and the same weighting applied to price gives a cost per thousand of about $1.44 against $11.50. The router's share of total spend falls from 31% to about 5%. Per-class accuracy is the check that this held — api_integration recall comes back to 77.5%, within noise of the large-model router, and the reason it comes back is not luck. The contested integration-versus-billing messages are precisely the ones with a small margin, which is precisely the set that escalates, so the escalation rule spends its budget on the boundary the cheap classifier is worst at without anybody having told it to.

The measurement that should be run before and after#

Four numbers, on the same inputs, before and after the change. Routing accuracy per class against the labelled route set, which is the direct measure of what you changed. Added latency at p50 and p95, because a router with an escalation path has a bimodal latency distribution and the mean hides it — the mean is 129ms only because it averages a large cheap population against a small expensive one, whereas the p95 for this design is 745ms, and if there is a latency budget it is p95 that consumes it. Cost per thousand requests, split into cheap-path and escalated-path so the escalation rate is visible as a line item that can drift. And end-to-end quality on the graded conversation set, because routing accuracy is a ceiling and not a promise, and the only way to know the ceiling translated is to measure the thing users receive.

A cheaper router is not the same decision made more cheaply#

The failure profile changes shape, not just magnitude. The large model's errors on this taxonomy are concentrated on the genuinely ambiguous messages, which is to say it is wrong roughly where humans are wrong. The embedding classifier's errors include those and add a different kind: messages whose vocabulary belongs to one class and whose substance belongs to another, which a human would never get wrong and which look bizarre in a review queue. That follows directly from what the classifier has access to, since a model that sees only positions in a vocabulary space has no way to be right about a case where the vocabulary points one way and the situation points the other. Anyone triaging routing errors after the swap should expect the error list to become less defensible even as it becomes shorter.

The maintenance cost moves too. A prompted router is edited: a new class is a paragraph and three examples, shipped in an afternoon, with the offline report as the check. A trained classifier needs labelled examples of the new class before it can predict it at all, because a fitted model has no output value for a class that was absent from its training data — which means the taxonomy change and the labelling round have to complete before the router can ship, and the thing that took an afternoon now takes a fortnight. On traffic that drifts quarterly, that difference compounds, and it is the strongest argument for keeping the prompted small model as tier one rather than the embedding classifier, even at four times the cost and four times the latency.

And a threshold is a parameter with a half-life. The margin distribution shifts when the taxonomy changes, when a new product surface appears in the traffic, and when the underlying embedding model is upgraded; a threshold chosen once and never revisited slowly turns into either an escalation rate of 3% with the accuracy of the cheap router, or 40% with the cost of the expensive one. Re-derive it from the labelled set whenever that set is re-sampled, and alert on the escalation rate directly, since it moves before the accuracy does — an escalation rate can be watched on live traffic with no labels at all, whereas an accuracy figure needs the file.

There is a proposal that always arrives at about this point in the design review, usually from whoever has just watched the router get cheaper: if the classification is this marginal, delete the dispatcher entirely and let each specialist hand the request off when it turns out to belong to someone else. It is a serious proposal, it removes a component, and it relocates the very decision this chapter spent its length measuring. The next chapter takes it apart.