Routing requests across model tiers

A mixed workload on one model is either overpaying for its easy traffic or underserving its hard traffic, and usually both. Splitting it needs two things: a way of deciding which class a request belongs to before the expensive call is made, and a defensible threshold per class that says what quality, cost and latency that class must get. The second is arithmetic. The first is where the design goes wrong, because the signal people reach for first does not carry the information they think it does.

The thresholds come from measured runs on a frozen task set rather than from preference, and the guards that keep a router honest live in the router rather than at the call sites.

Length tells you almost nothing#

The routing rule people reach for first is prompt length, on the theory that a long prompt implies a complicated request. Six real requests from one week, with token counts and the difficulty band a human labeller assigned after seeing what it took to answer them:

Measured across 2,000 labelled requests, the correlation between prompt tokens and difficulty band came out at 0.09, close enough to nothing that a length-based router is a random assignment with extra steps. The reason is structural: prompt length measures how much context the caller happened to paste, and difficulty measures how much inference is needed to get from that context to an answer. A 21,400-token contract classified into one of three buckets is a long prompt and a trivial decision. A nine-word question about a flaky test carries the entire state of a distributed system in the asker's head and none of it in the prompt, which makes it both short and the hardest request of the day.

Two things in that paragraph are worth separating, because one is a measurement and the other is the reason it came out the way it did. A correlation near zero means that knowing one quantity tells you almost nothing about the other, so a rule that reads length and assigns a class assigns it at close to the rate chance would. That is what "a random assignment with extra steps" means, and it is a description rather than a jibe. The reason the number came out near zero is that the two quantities count different things. Prompt length counts the tokens that arrived. Difficulty counts the dependent inference steps between what arrived and the answer. Nothing ties those counts together — a long context can support a single-step decision, and a short question can demand many steps against knowledge that is nowhere in the prompt. The flaky test is the second case and the vendor agreement is the first, and the table above holds both.

Whatever signal the router uses has to be available before the expensive call, since that is the call being decided. Three qualify, and they are layers rather than alternatives.

The first signal is the one teams overlook because it is not a machine-learning problem. The calling surface usually knows what it is asking for: the endpoint is /classify or /debug, the interface has a mode, the batch job has a job type. That information is free, exactly correct, and available before any model runs, and a router that reads it handles the large majority of traffic without any prediction at all. Building a difficulty classifier for requests whose type is already encoded in the URL is a common and expensive detour.

What makes that signal so much better than the two below it is that it is not a prediction. A prediction carries an error rate by construction, because it is inferring something it was never told. The calling surface was told: some piece of code chose an endpoint, and that choice is a record of intent rather than an estimate of it. There is no confusion matrix for a fact. So the routing decision for the bulk of traffic costs nothing to make and nothing to maintain, and the two signals that follow exist only to cover the traffic where no such record was ever created.

The second signal is a cheap classifier for the traffic where the surface genuinely does not know — a single free-text box, a chat interface, an inbound queue of mixed work. A small model asked to output one of three labels costs around $0.0002 and 80 milliseconds, and an embedding-plus-logistic-regression classifier trained on your labelled set costs less than that. Either is fine; what matters is that you measure its confusion matrix, the table of which class it predicted against which class was correct, because the two kinds of mistake have very different prices. A hard request misrouted to the cheap tier costs a wrong answer and, where a verifier exists, an escalation worth about $0.09. A routine request misrouted to the debugging tier costs $0.088 of pure waste, and since routine traffic is sixteen times more voluminous than debugging traffic, a classifier with a 6% false-promotion rate spends more on misrouted classification than the entire debugging line costs.

Working that out in words is what makes it stick, and it is the same base-rate effect that decided the verifier's false-reject question in the previous chapter. The two error rates are charged against populations of very different size. A false demotion is charged against debugging traffic, which is the small slice. A false promotion is charged against classification traffic, which is the large one, and each instance costs the whole difference between the cheapest tier and the most expensive. A small rate multiplied by a large volume and a large per-unit price outruns a large rate multiplied by a small volume. That is why a classifier in this arrangement should be tuned reluctant about promotion rather than balanced, and why "balanced" is the wrong default any time the two classes differ in both volume and price.

The third signal is the cascade from the previous chapter, and it is the strongest available because it is not a prediction at all. Running the cheap model first and escalating when the verifier rejects twice measures difficulty rather than guessing it, at the cost of the cheap attempt and its latency. Use it wherever a real verifier exists, fall back to the classifier where one does not, and note that the two compose: the classifier decides the starting tier, the cascade decides whether to stay there.

Composing them is worth stating concretely, because the two signals answer different questions and the asymmetry between their mistakes is what makes the composition safe. A request arriving with no declared class is classified, starts on whichever tier that produced, and escalates only when the verifier rejects it twice. So a classifier mistake in the cheap direction is recoverable, at the cost of one wasted attempt and its latency. A mistake in the expensive direction is not recoverable at all, because nothing demotes a request the expensive tier has already answered — the money is spent before there is anything left to check.

Thresholds you can defend in a review#

A routing rule is three numbers per class, and each has to come from a measured run rather than from a preference.

The floors come from what the downstream consumer needs. Classification feeds an automated queue assignment, and below 95% the misrouted tickets cost more support time than the automation saves, so 95% is a product fact rather than an aspiration. The debugging floor is 70% because the alternative is an engineer reading the code themselves, and a tool that helps seven times out of ten is worth opening. The latency ceilings come from where the result is consumed: classification is inline in a request path, planning is behind a spinner, and debugging is delivered asynchronously — which is what makes a 45-second ceiling and a 16,000-token budget compatible rather than the unreachable-budget trap from two chapters ago.

The cost ceilings are then derived from the budget and the mix rather than chosen. Multiply each class's ceiling by its volume — $6,400, $18,000 and $24,000 — and the total is $48,400. Against a $40,000 budget that does not fit, so at least one ceiling has to come down or the mix has to change. That tension is the useful part of writing them down, because it forces the conversation about which class's quality you are willing to trade before the month rather than after it.

Ceilings chosen independently, each defensible on its own terms, have no reason to add up to a budget, and nothing in the process of choosing them makes them do so. The moment they sit in one table the arithmetic forces a conversation that would otherwise happen in arrears: which class's quality floor comes down, or which class's volume changes, or what the budget becomes. A ceiling that has survived that conversation is a different object from one that was merely picked.

CLASS_CONFIG = {
    "classification": Tier(model=CHEAP, thinking=None,  budget_usd=0.002, deadline_s=1.5),
    "planning":       Tier(model=BIG,   thinking=4000,  budget_usd=0.030, deadline_s=12),
    "debugging":      Tier(model=BIG,   thinking=16000, budget_usd=0.120, deadline_s=45),
}

def route(req: Request, spend: MonthlySpend) -> Tier:
    cls = req.declared_class or classify(req.text)     # surface first, model second
    tier = CLASS_CONFIG[cls]
    if spend.fraction_used() > 0.85 and cls == "debugging":
        tier = tier.demote(thinking=4000, budget_usd=0.030)   # alert, do not fail
    return tier

Two guards belong in the router rather than in the call site. The per-request one is the token allowance, sized from the decode-rate arithmetic so a thinking budget cannot consume the whole deadline; when it is exceeded the router returns the degraded result type rather than a truncated answer, with status, reason and verified populated so downstream code can decide what to do. The monthly one is a spend guard: when the month's spend crosses 85% of budget, the router demotes the most expensive class and raises an alert, which converts a budget overrun into a documented quality reduction. Both need testing on purpose. Set the allowance to 500 tokens in staging and confirm that every consumer handles a degraded result without crashing and without treating it as success, because the path that only executes during an outage is the path nobody has run.

The conversion the spend guard performs is the reason it belongs in the router rather than in a dashboard alert, and the three places you could put the same number do three different things. A budget with no enforcement anywhere produces an overrun, discovered once the month has closed and impossible to undo. A budget enforced at the call site produces failures, discovered by users. A budget enforced in the router produces a demotion: the requests still get answers, the answers are known to have come from a cheaper tier, the source field records which tier that was, and somebody can decide inside the remaining days of the month whether the trade was acceptable.

Thresholds also need a review cadence, since a fixed routing table drifts as the traffic mix moves and as model versions change underneath it. Sample two hundred requests a week, grade them, and compare each class against its floor. A class below floor promotes to the next tier and the cost implication goes on the dashboard. A class running three points above floor and twenty percent under its cost ceiling is a candidate for demotion, which is the check that catches a classification tier still paying for medium effort a year after someone raised it during an outage.

Four findings that would justify one model#

Routing is machinery, and machinery is worth its maintenance only while the gap it exploits is real. Four findings would justify collapsing to a single model, and each is worth computing before building rather than after.

The first is an oracle comparison. Route a frozen task set perfectly — every item to the tier a human says it belongs in — and compare against the best single model on the same set. If perfect routing beats one model by two points of quality and 15% of cost, then achievable routing, with a classifier that is 88% accurate, beats it by close to nothing, and the whole apparatus is a rounding error with an on-call burden attached.

The step doing the work there is that an oracle bounds the achievable. Perfect routing is the best any router can do, since a router cannot beat knowing the right answer for every request in advance. So the oracle's margin over a single model is a ceiling on the value of the entire design, and a real classifier, being imperfect, collects some fraction of it. Compute the ceiling first and the question of whether to build stops depending on how good the classifier turns out to be.

The second is a per-class quality gap inside noise. If the debugging class scores 68% on the cheap tier and 70% on the reasoning tier, with a confidence interval of ±4 points on 200 items, there is no gap to route on for that class regardless of what the benchmarks say, and the honest response is to run the cheap tier everywhere and spend the difference on tools, retrieval, or the verifier.

The third is price movement. The design rests on a roughly eleven-fold cost ratio between tiers. That ratio has compressed repeatedly as smaller models absorb capabilities that used to require larger ones, and at a ratio of two the savings on this workload fall to a few thousand dollars a month, which does not pay for two prompt sets, two eval suites, two failure taxonomies and an extra branch in every investigation. Re-run the numbers whenever a new model version lands rather than treating the routing table as settled.

The fourth is operational. If the router itself becomes the most common cause of outages, if engineers cannot reproduce a production answer because they do not know which tier served it, or if a class's prompt has silently diverged from its counterpart in another tier, the complexity is being paid for in reliability. Recording the serving tier on every response — the source field from the degraded result type — is what makes that diagnosable, and its absence is usually why the complexity became unmanageable.

Routing is therefore a threshold problem wearing an intuition problem's clothes. The thresholds are computable from what the consumer needs and what the budget allows, the assignment is free for most traffic and cheap for the rest, and the one signal everybody reaches for first carries almost no information. What makes the arrangement defensible in a review is that every number in the table came out of a measured run on a frozen set, including the ones that argue against building it.

What carries out of this topic is a habit rather than a configuration. Instrument the call so hidden tokens are counted, sweep the effort dial per band rather than globally, check whether the bottleneck is deliberation before paying for it, distinguish an exhausted budget from a slow request, and prefer a cheap verifier over an expensive model wherever the check is executable. The first workload to apply it to is the one you already suspect is over-configured: take a week of its traffic, freeze two hundred requests, run the current setting and one setting below it, and look at the per-item deltas. That measurement takes an afternoon, and it is the one that decides whether any of the rest is worth building.