A router decides blind, a cascade decides after seeing

The ticket said: I was charged twice for order 88214 — the second charge is a subscription I cancelled in March. A 60-million-parameter classifier sitting in front of the support assistant scored it 0.11 on the "needs the big model" head, which was comfortably under the 0.4 threshold, so the request went to the 8B model. The 8B model recognised the shape of a duplicate-charge complaint, produced the standard reply about automatic reversals arriving in five to seven business days, and returned in 380 milliseconds. The customer waited seven business days for a reversal that was never going to happen, because the second charge was not a duplicate at all; it was a live subscription that the cancellation had failed to terminate, and answering it correctly required reading the billing history rather than pattern-matching the first sentence.

Nothing in the system registered a problem. The HTTP status was 200. The latency was excellent. The token counts were unremarkable. There was no exception, no retry, no fallback event, and no second answer anywhere to compare against, because at the moment the routing decision was made there was no answer of any kind in existence. The classifier had scored the question, and the question looked easy.

That last detail is the whole subject of this chapter. Two designs are commonly described with the same words — send the easy work to the cheap model, send the hard work to the expensive one — and they differ in exactly one respect, which is whether the decision is taken before or after the cheap model has done the work.

Both belong to the same family, and it is worth saying plainly what that family is before the difference between its members starts carrying any weight. Model routing is the practice of keeping more than one model behind a single endpoint — in the simplest arrangement a small cheap one and a large expensive one — and deciding, per request, which of them answers, rather than sending everything to the strongest model you have access to. It exists because two gaps fail to line up. A small model can cost twenty to fifty times less per token than a frontier one and answer in a third of the time, while most of what production traffic asks for is an order status, a policy paraphrase or a two-line summary, which the small model handles about as well as the large one. Routing is the machinery for spending frontier money only on the requests that need it. You meet it on high-volume surfaces — support assistants, document pipelines, code completion, search summarisation — and inside the "auto" model settings providers now ship, which are routers running on somebody else's judgment about your traffic.

The decision has two possible positions#

A router places the decision in front of everything. Some function of the request alone — a classifier, an embedding similarity, a set of regexes over the prompt, a learned scoring model of the kind shipped in RouteLLM — assigns the request to a model, that model runs once, and its output is returned. Total latency is one model call plus whatever the routing function costs, which for a small classifier is single-digit milliseconds. Exactly one model ever sees the request.

A cascade places the decision after the cheap model has produced a candidate. The small model runs first on everything, its output is examined by some escalation signal, and requests that fail that examination are re-run on the strong model. Easy requests cost one cheap call. Hard requests cost a cheap call and a strong call, in sequence. The design comes out of the FrugalGPT line of work, whose central result is that a chain of cheap-model attempts with a scored stopping rule can match a frontier model's quality on a benchmark at a fraction of the spend, precisely because most requests never need the expensive attempt.

Diagram

The two boxes look similar on a diagram and behave very differently under a wrong decision, because the evidence available at the decision point is not the same in the two cases.

The row that matters most is the second-to-last one. In a cascade there is a defined place to put a verifier, a schema check, an execution attempt, a judge, or a self-consistency vote, because by the time the decision is taken a concrete answer exists and can be inspected. In a router that place does not exist. You can inspect the request all you like; the request is not the thing that goes wrong.

The error that leaves no receipt#

Consider what the two systems produce when the escalation decision is wrong in the direction of "too cheap".

The cascade produced a candidate answer, applied its signal, and accepted. Both of those things are recordable. You can log the candidate, the signal's value, and the threshold, and later replay the same request against the strong model and compare the two answers under a judge. The decision was conditioned on something observable, so it can be audited after the fact against a different observation. That does not make the error harmless — a wrongly accepted answer still goes to a user, which is the whole subject of chapter 4 — but it makes it a thing with a measurable rate.

The router produced nothing except the answer it shipped. There is no candidate, no score on the output, no second opinion, and no signal whose calibration you could go and measure, because the classifier scored a question and questions do not have a ground truth about which model was needed unless you generate one deliberately. To find out how often the router is wrong you have to construct the missing evidence yourself: sample production traffic, run the strong model on requests the router sent to the cheap one, and judge the pairs. That is an offline experiment somebody has to fund and maintain. Absent that experiment, the router's error rate is not merely unmeasured; there is nothing in the running system that could measure it, which is a different and worse situation.

This is why the distinction is worth defending against the objection that it is academic. The objection usually runs: both designs escalate the hard requests, one with a classifier and one with a retry, so who cares. The answer is that a retry is conditioned on a fact about the answer while a classification is conditioned on a guess about the question, and a system built on a guess with no downstream check has no mechanism by which the guess can ever be corrected — not at request time, not at review time, and not by a customer complaint, because the customer receives a fluent, confident, well-formatted wrong answer that looks exactly like a right one.

"Classify, then send" is a router with a friendlier name#

The most common version of this confusion in a design document reads roughly as follows: we classify the incoming request; simple ones go to the small model, complex ones go to the large model; if the small model fails we escalate. The last clause is doing a lot of work that it cannot do, because "fails" in that sentence almost always means the call failed — a timeout, a 5xx, a malformed JSON body — and those are infrastructure failures rather than quality failures. A small model that confidently answers a hard question does not fail. It returns 200 with well-formed content in 380 milliseconds, which is precisely the case the design was supposed to catch.

So a system described that way is a router. It has one model call per request on the happy path, no examination of the produced answer, and an escalation clause that only fires on transport errors. Calling it a cascade in the design doc has a practical consequence: the reviewers assume there is a verification step, so nobody asks the question that would have exposed the gap, which is what looks at the cheap model's output before it goes out.

The checklist item that this design is meant to satisfy has two halves, and both need to hold. The default path is the cheap model, and it should meet the latency and quality target for routine traffic on its own, because if the small model is not actually good enough for the routine case then you do not have a cost optimisation, you have a quality cut with a cost saving attached. And harder requests need a bounded path to a stronger model — bounded meaning one escalation rather than a ladder that can loop, with an explicit deadline and a defined behaviour when the strong call also fails. An unbounded escalation path turns a cost optimisation into a tail-latency problem, which chapter 7 returns to in the specific case of a dependency that stops answering.

Where deciding blind is the right trade#

None of this makes routers a mistake. There are workloads where the router is the correct design and the cascade is not affordable.

The clearest is anything with a hard latency budget that a second sequential model call would blow. A voice agent aiming to start speaking within 300 milliseconds of the user finishing cannot run a 900-millisecond small model, examine the output, and then run a 1.5-second large one; the escalation path alone exceeds the budget by a factor of five, and the user hears silence. A router with a 4-millisecond classifier fits. The same argument applies to autocomplete, inline suggestions, and any interactive surface where the deadline is shorter than two model calls.

The second is when the class is genuinely known from metadata rather than inferred from content. If your pipeline processes both scanned PDFs and structured JSON exports, and the structured exports are handled correctly by the small model essentially by construction, then dispatching on document type is a routing decision that is right by definition and needs no verification. The failure mode being discussed here comes from inferring difficulty from a request, and there is nothing to infer when the field is already in the payload.

The third is when verification is genuinely more expensive than escalation. If checking a candidate answer costs nearly as much as producing a better one — long documents where the verifier must read the same context, or tasks where the only credible check is another frontier model — then the cascade's economics collapse, because you are paying a near-strong-model price on 100% of traffic to save a strong-model price on 85% of it.

What a router costs you in exchange for that latency is the recovery path and the running measurement, and both have to be bought back deliberately: a periodic sampled dual-run where some percentage of cheap-routed traffic is also sent to the strong model offline, judged, and tracked as a quality delta over time. That is a fixed ongoing cost in both money and maintenance, and if nobody is willing to fund it, the honest conclusion is that the router's accuracy will not be known.

The cascade avoids that by paying for its evidence at request time instead, and the bill for that arrives as latency: every escalated request runs two models back to back, one after the other, on the clock. The team in the opening story eventually replaced their classifier with a cascade, and the first thing that happened was that their p95 went up.