Two points spread thin usually means an under-tuned baseline

The 87% that the routed design has been compared against for two quarters was measured on a prompt that has not been touched since the day the router shipped. In the meantime the specialists absorbed a better retrieval setup — retrieval being the step that searches the documentation for passages relevant to the message and pastes them into the prompt before the model runs — a rewritten tool-result format, twenty new boundary examples, and the escalation and fallback machinery. The general prompt, sitting in the deployment as the fallback path, received none of it.

That asymmetry is what makes the comparison unsafe, and it is worth naming as a shape rather than as an incident. The baseline figure is a quantity that stays perfectly constant while the thing it is supposed to represent — how good a single well-maintained prompt can be on this traffic — moves every sprint. Nothing alerts on it, because nothing changed. Every quarter's review then reads the same number as though it were a current measurement, and the gap it appears to describe is partly a gap between the routed system and its own past.

Unfreeze the baseline before drawing any conclusion#

Re-running the general prompt today with the current model, the current retrieval, and the accumulated prompt fixes applied gives 89%, not 87%. The routed system is at 91%. The margin under discussion is two points, and one of the four points it was thought to be had been sitting in a frozen baseline.

That correction is cheap when the general prompt is still running as the fallback destination, because the prompt exists, its tool schemas are current, and the graded conversation set is the same one the routed system is scored on. Where the general prompt has been deleted, the comparison usually cannot be made honestly at all. Reconstructing it is a week of work whose only product is an argument against the current architecture, which is not a task anybody volunteers for, so the frozen number survives indefinitely and every quarter's review repeats it. Keeping the single-path prompt alive and maintained is the cost of being able to answer this question at all, which is a reason to keep it independent of its usefulness as a fallback.

Where the two points live decides what to do about them#

Two points is an aggregate, and the aggregate is not the decision-relevant quantity. Split it by class: for each route, compare the re-tuned general prompt's score on that slice of traffic against what the routed system actually delivers on the same slice — end to end, misroutes included, since that is what a user of that class experiences.

The last column is the second multiplied by the fifth, and that multiplication is the whole point of the table. A class contributes to an aggregate in proportion to how much traffic it carries, so api_integration at 19% of traffic and seven points of gain contributes 1.33, while how_to at 38% and one point of loss contributes −0.38. Adding the five contributions gives the weighted delta at the bottom, which is the same two points the aggregate reported — the aggregate is not a different measurement, it is this column summed, and summing is where the information goes.

Two rows produce the entire gain and three produce none. api_integration is worth 1.33 of the two points, because the general prompt genuinely is bad at it — it holds the retry semantics somewhere among four other knowledge bases and reaches for search_delivery_log inconsistently when nine tools are in scope. incident_status is worth 0.75, and it is not a specialist at all: it is a status-page lookup and a template, which beats any prompt because the correct answer is a fact in a system rather than a generation. The billing and how-to deltas are ±1 on slices of roughly 80 graded conversations each, which is inside the rubric's noise; treating either as a signal would be reading the measurement error. account_access is flat, and its specialist has been maintained for two quarters for zero measured benefit.

That shape — concentrated in one or two classes, absent or negative in the rest — points at a specific redesign. Keep the paths that earn their split and put everything else on the general prompt as the default.

The alternative shape is the one to be suspicious of. If the same table had come back with every class up by 1.5 to 2.5 points, the honest reading would be that specialisation is not doing anything class-specific. A gain that appears equally in five unrelated classes cannot be coming from any of the five prompts' domain material, since that material differs completely between them; what it can be coming from is something all five have in common, which is that they were written more recently and more carefully than the prompt they are being compared against. A diffuse gain of that size is what you get from any competent prompt rewrite, and the routed design would be charging permanent complexity for something a day of work on the single path would have delivered. Aggregates cannot distinguish those two worlds, and they are opposite recommendations.

Collapsing to the classes that earn their split#

The redesign that follows from the table is three destinations: incident_status to the deterministic lookup, api_integration to its specialist, and everything else to the re-tuned general prompt as the default. That deletes the billing, access and how-to specialists and drops the taxonomy from five classes to three.

It also drops the router's job from a five-way decision with ten boundaries to a three-way decision with three, and that count is worth deriving rather than accepting, because it does not shrink the way the class count does. A boundary is an unordered pair of classes — a place where the router can put a request from one into the other — so the number of boundaries is the number of ways to choose two classes from n, which is n · (n − 1) / 2. That expression grows with the square of n while the class count grows with n, so removing two classes from five removes rather more than two-fifths of the boundaries. What that licenses is a general claim: the error surface a router presents is not proportional to the number of prompts you maintain, it grows faster, so merging classes buys more than it appears to. The residual is that the formula counts places where an error can live and says nothing about how much traffic crosses each one, and on this taxonomy nearly all of the error sat on a single pair.

The complication is that the class worth keeping is the class the router is worst at, so this does not eliminate the hard boundary — it keeps exactly the one that produced every misrouting problem in this topic. What changes is the price of getting it wrong. In the five-class design, an integration question misrouted to billing was answered by a prompt with ledger tools and no way to see a delivery, scoring around 71 on that slice. In the three-class design, the same miss lands on the general prompt, which has all nine tools and scores 82 on integration traffic. The misroute penalty falls from 22 points to 11.

Read that through the condition from the first chapter and the consequence is direct. Expected quality is the specialist's score weighted by the routing accuracy plus the misroute score weighted by the rest, so the damage a wrong label does is the gap between those two scores, times how often it happens. Halving the gap halves the damage at any accuracy, which is what "the same router accuracy buys roughly twice as much" means.

Re-measuring bears that out. Routing accuracy on the api_integration-versus-default boundary comes in at 81%, a little better than the 78% in the five-class taxonomy because the how-to confusions no longer count as errors — those requests now land on the default, which is where they were going to be answered anyway. Applying the per-class form of the condition, 0.81 × 93 + 0.19 × 82 ≈ 90.9 for the integration slice, against 89 for the same slice in the five-class design and 82 for the general prompt alone. The class improves by dropping the classes around it.

The cost row is the one that usually surprises people, because it runs against the intuition that adding a model call adds spend. Work out where the money goes and the direction reverses. The general prompt carries around 5,000 input tokens of combined knowledge on every request, whether or not the question is about billing, and an input token is billed whether or not it was relevant. A specialist call carries 3,000, and a status lookup carries none at all because there is no model call. So the routed design pays for one small classification and one narrow prompt where the single-path design pays for one wide prompt, and the wide prompt is the larger of the two. Routing is cheaper per request in proportion to how much of the traffic it can send somewhere narrow, which is why the five-way design is the cheapest of the three and why the three-way design gives some of that saving back. Cost is a reason to route; it is rarely the reason anyone builds one, and it is worth knowing which direction it actually points before quoting it in a review.

Name the result that would have you delete the router#

Three measured outcomes should end the routed design, and it is worth committing to them in writing before running the measurement, because afterwards there is always a reason the numbers are unrepresentative.

The first is the per-class condition failing everywhere. For each class, keep the split only if p·S + (1 − p)·M > G on that class's own slice — its routing recall, its specialist's score on correctly routed inputs, the score its misroutes get wherever they land, against the re-tuned general prompt on the same traffic. how_to fails this test in the table above: at 98% recall and a specialist scoring 93, with misroutes landing around 60, the routed path predicts 92.3 against the general prompt's 92, which is a tie bought with a prompt to maintain. If no class passes, there is nothing for a router to do and the correct system is the single path.

The second is the baseline closing the gap. Give the general prompt the attention the specialists have had — a day of work, the same retrieval, the same boundary examples turned into instructions — and re-run. If the routed advantage falls below about a point, the specialisation was standing in for prompt maintenance, and one prompt at 90.5 is a better system than five prompts and a classifier at 91.4 by every operational measure that is not on the quality rubric.

The third is the gap being smaller than the measurement, and this is the one that needs its arithmetic stated rather than asserted. Two points on 300 graded conversations, with a rubric that has any grader disagreement at all, has a 95% confidence interval — the range that would contain the true value on 95% of repeats of the same measurement — of roughly ±3.5 points on the difference between two systems. An interval that wide straddles zero, which means a measured advantage of two points and a true advantage of nothing produce the same reading, so "2 points better" and "no different" are not distinguishable with that sample. This is the most common way the decision goes wrong, and it goes wrong in whichever direction the team was already leaning. Either widen the graded set until the interval is smaller than the effect you are trying to detect, or state plainly that the comparison cannot resolve it and decide on latency, cost and maintenance instead — which are all measured to far tighter precision.

Two things do not belong on that list. Latency inside the budget is not an argument for keeping a router; it is the absence of an argument against one, and the two get confused whenever a dashboard is green. And "five prompts is a team problem, not an architecture problem" gets the causation backwards: the number of prompts is a direct consequence of the number of classes, the number of classes determines how many boundaries the router can be wrong on by the formula above, and every one of those boundaries is a permanent source of the failure the first chapter opened with. Maintenance burden and error surface are the same quantity measured in different units.

The condition for building a router at all, in the form the per-class table gives it, is that specialist paths must beat one general prompt on their own slices by more than the router's errors on those slices take away, and that has to be true of at least one class by a margin larger than your measurement noise. Everything else in this topic is instrumentation for evaluating that condition — the taxonomy derived from traffic so the classes are separable and consequential, the labelled route set so accuracy is a computation rather than a claim, the per-class confusion matrix so a starved specialist is not mistaken for a bad one, the cheap classifier so the decision costs a fraction of the path it picks, the capped escalation so a specialist can correct the router without the two of them bouncing a request between them, and the fallback so the request that fits nothing has somewhere to be.

To use it on a system you already have, four things run in order. Unfreeze the baseline and re-tune it, because the comparison is worthless otherwise. Score the routing decision on its own against labelled routes and keep the matrix, per class, in both directions. Build the per-class slice table and read where the gain lives rather than what it sums to. Then decide per class — not globally — which splits survive, and delete the rest.

And keep two artifacts alive permanently, because they are what makes the decision repeatable rather than a one-off argument: the labelled route set, re-sampled quarterly as traffic drifts, and the general prompt, maintained at parity with the specialists. The first tells you whether the router still works. The second tells you whether it still needs to exist.