Model routing and cascades

Read these first

• Metrics and SLOs
• Evaluation in CI

What you will know

• Route to a cheap model by default and escalate on a signal

  The default path should meet the latency and quality target for routine traffic, while harder requests have a bounded path to a stronger model.

• Define the escalation signal and measure its accuracy

  A confidence score, verifier, or task classifier can be miscalibrated; measure false escalations and missed escalations against labeled outcomes.

• Report the cost saving and the quality cost together

  Include per-request spend, escalation rate, latency, and task-level quality so aggregate savings cannot hide regressions in important cases.

• Run an offline evaluation that compares routing with an always-strong baseline

  Replay representative traffic with fixed model versions and slice results by task difficulty, since average scores can conceal failures concentrated in rare workflows.

• Build a cascade that fails safe

  Timeouts, malformed outputs, and verifier outages need bounded retries and a known fallback rather than silently returning a weaker or unvalidated answer.

Chapters

1. 1

   A router decides blind, a cascade decides after seeing

   Both designs send easy work to a cheap model, and the difference is where the decision sits relative to the work. A router classifies before any answer exists, so a misclassification is permanent and leaves no trace; a cascade runs the cheap model first and reacts to what actually came back, which gives it something to check and somewhere to recover. Classify-then-send is a router with no verification step, whatever it is called.

   Read chapter
2. 2

   Escalated requests pay for both calls in sequence

   With the cheap model at 400ms, the strong one at 1.2 seconds and 15% escalation, 85% of traffic genuinely did get faster while p95 landed on the cascade path near 1.6 seconds. In practice it is worse, because the requests that escalate are the long, hard ones both models are slow on. Latency objectives for a cascade have to be set on the escalation path, not on the population that stayed cheap.

   Read chapter
3. 3

   A model's confidence in itself measures fluency

   Asking the cheap model to score its own answer and escalating below 0.7 produces a steady 12% escalation rate and 38% savings while measuring nothing about correctness. A self-report tracks fluency, so the answers it is most confidently wrong about are precisely the ones that never escalate. The rate is stable because the self-assessment is stable, and two quiet months prove little about a failure mode whose output is plausible prose.

   Read chapter
4. 4

   Missed escalations reach users, false ones only cost money

   Labelled outcomes split the signal into two errors with unequal consequences. Eight percent false escalations buy the strong model for work the cheap one had already done well, which shows up on the invoice. Four percent missed escalations arrive as confident wrong answers with no error, no retry and nothing in the trace to flag them, so they are the ones to fix first. Draw the labels from the replay and judging apparatus already running in CI.

   Read chapter
5. 5

   Most of your replay sample cannot show a difference

   A cascade scoring 4.31 against 4.34 for always-strong over 5,000 replayed requests reads as equivalence and is close to incapable of reading anything else: the router sent most of that sample to the same model in both arms, so those rows contribute identical scores and dilute the rest. A 3% multi-step extraction workflow that is catastrophically worse moves the mean by 0.03. Pin model versions and slice by task and difficulty.

   Read chapter
6. 6

   A saving without a quality delta is not a result

   Forty-one percent off the bill means nothing beside an aggregate quality score dominated by easy requests that routing could never have affected. Publish per-request spend, escalation rate, latency on both paths, and task-level quality sliced by difficulty, so a regression concentrated in one workflow has somewhere to become visible. Savings and quality have to travel in the same report or neither number can be acted on.

   Read chapter
7. 7

   A verifier that times out must escalate, not wave through

   When the verifier's provider fails a third of calls, the cascade escalates on deadline rather than returning an unverified answer with a flag — a flag is a log entry, and the requests the verifier existed to catch would go to users at a third of your volume. Bound that escalation and alert on it so an outage surfaces as a cost spike. A deprecated version on the escalation path invalidates the comparison the design rests on.

   Read chapter