Every dispatcher needs a path for the input that fits nothing

Three messages from one Tuesday, all of which received an answer nobody would defend. The first: "Before we can renew we need your DPA countersigned and the sub-processor list for the EU region." The second: "We've decided to move off the platform at the end of the term — what's the process for exporting three years of invoice data and closing the account?" The third, in Portuguese, asking whether the platform supports Brazilian CNPJ tax identifiers on invoices issued to local subsidiaries. A DPA is the data processing agreement a customer's legal team requires before it will renew, and the sub-processor list names the outside companies that touch their data; a CNPJ is the registration number Brazilian companies put on invoices. None of the three is a billing question, an integration question, an access question, a how-to question, or a status check.

The router assigned all three to how_to, because how_to is the widest class and something has to win an argmax — the operation that returns whichever class scored highest, whatever the scores happen to be — and the how-to specialist searched the documentation, found nothing exactly on point, found something adjacent, and produced three fluent, confident, wrong answers.

Why no amount of classifier work fixes this#

The mechanism is worth stating precisely, because it is not a bug in the classifier and it will not be repaired by training or prompting one.

A classifier is a function, and a function's output is drawn from a range fixed in advance. Whatever the router is — a prompt returning an enum, a logistic regression returning a probability vector — its output space is the set of classes it was given. The final step in both cases is an argmax over that set, and an argmax over a finite set always returns a member of it, because it is defined as "the largest of these" and there is always a largest. So the router will return the best member of that set for any input whatsoever. Feed it a haiku and it returns a route. There is no configuration of the classifier that makes it say "none of these", because "none of these" is not in the range of the function.

What follows is a design requirement rather than a tuning problem. The system needs a destination that is selected by something other than the argmax, which means the selector has to read the scores rather than the winner, or has to sit outside the classifier altogether. Everything in this chapter follows from that one step.

The gate is a confidence floor, not another class#

Two mechanisms are available and they behave very differently, which is the part that catches people out.

The first is a score floor on the existing classifier. The embedding classifier already produces a probability per class; take the top one and, if it is below a threshold, send the request to a fallback path regardless of which class won. This does not touch the decision boundary between real classes — the classifier's opinion about billing versus integration is unchanged, and every request that clears the floor is routed exactly as before. It is a filter applied after the decision, not a modification of it, which is what makes its effect on the rest of the taxonomy exactly zero.

The second is giving the router an explicit none_of_these output. For a prompted router this is the obvious move, since there are no probabilities to threshold, and it is genuinely useful. It is also not free, and the reason is the argmax again. A new output value is a new member of the set the argmax runs over, so it competes with the real classes for every input rather than sitting outside them. On this taxonomy, adding none_of_these to the enum with a paragraph describing when to use it raised the fallback rate from 3.1% to 7.8% and dropped api_integration recall by 3.1 points.

Follow that drop to its cause, because it is the opposite of what the change was meant to do. The contested billing-versus-integration messages are hard precisely because they have features of both classes, which means neither class is a comfortable fit for them. A label meaning "neither" is a comfortable fit for exactly that description, so those messages started resolving to it and taking a general path when one of the two specialists would have served them well. The option becomes an attractor for exactly the traffic you most want routed.

So: use the score floor where scores exist, and where the router is a prompt and an explicit label is the only mechanism available, treat it as a change to the classifier and measure it as one — re-run the offline report and check what happened to every other class's recall, not just the fallback rate.

FALLBACK_FLOOR = 0.40      # top-1 probability below this: nothing fits well

def route_request(message: str) -> Decision:
    scores = embedding_classifier.predict_proba(message)
    ranked = sorted(scores.items(), key=lambda kv: -kv[1])
    (top, p1), (second, p2) = ranked[0], ranked[1]

    if p1 < FALLBACK_FLOOR:
        return Decision(route="fallback", reason="no_class_above_floor",
                        top=(top, p1), second=(second, p2))

    if p1 - p2 < ESCALATE_BELOW:
        return Decision(route=classify_with_opus(message), reason="contested",
                        top=(top, p1), second=(second, p2))

    return Decision(route=top, reason="confident",
                    top=(top, p1), second=(second, p2))

The order of those two tests is deliberate, and so is the fact that there are two of them. The floor reads p1 alone and asks whether anything fits; the escalation reads the gap between p1 and p2 and asks whether two things fit about equally. A contested request has two plausible homes and wants a better classifier; an unmatched request has none and wants a different kind of destination. A single threshold that tries to cover both either escalates the whole unmatched pile to an expensive model that will also return one of five wrong classes, or sends every hard boundary case to a general prompt and gives away the specialists on the traffic they were built for.

Choosing the floor needs a second file#

Choosing the floor value is a different exercise from choosing the escalation threshold, and the difference is easy to miss because both are numbers swept against data.

The escalation threshold could be swept against the labelled route set, because every item in that set has a correct class and the sweep asks how often the cheap classifier gets it wrong at each margin. The floor cannot be swept that way at all. The labelled set contains only requests that belong to a class, by construction, since a class label is what makes an item eligible for the file. So every fallback the floor triggers on that set is a diversion of a request that did have a proper home — a false positive with no true positives available to weigh against it — and a sweep that only counts errors of one kind drives the floor to zero.

What the floor needs is a second, smaller set: sixty or eighty requests that genuinely fit nothing, which the support team can assemble from memory in an afternoon before any fallback exists. Score the floor against both files at once, and read two quantities at each value — what fraction of the unmatched file clears the gate and gets routed anyway, and what fraction of the labelled file is unnecessarily diverted. On this taxonomy, 0.40 caught 79% of the unmatched file while diverting 2.1% of the labelled one, and 0.50 caught 91% while diverting 6.4%.

The second of those is the wrong trade, and seeing why requires converting both percentages into the same units, because they are measured over populations of very different sizes. The number of requests caught is the catch rate times the share of traffic that is genuinely unmatched. The number diverted is the divert rate times the share of traffic that has a proper home. The second share is very much the larger of the two, so a small divert rate applied to it can outweigh a large catch rate applied to the small one, and at 0.50 it does: the diverted requests outnumber the caught ones by a factor of three at production traffic ratios. Two percentages that look comparable on the page are not comparable at all, and the only way to see it is to multiply each by the population it was measured over.

Where a fallback request actually goes#

Two destinations are worth having, and the choice between them is a product decision rather than a technical one.

A general prompt — specifically, the single-path prompt the routed design replaced — handles the largest share. It has all four knowledge bases and all nine tools, it was already written, and its whole character is that it degrades gracefully on inputs it was not expecting, which is precisely the requirement. Keeping it alive costs something: it has to stay in the deployment, stay compatible with the current tool schemas, and get re-tuned when the specialists do. That maintenance turns out to be worth paying for a second reason, which the next chapter depends on.

A human queue handles the rest, and some requests belong there by policy rather than by capability. The cancellation message above is one: the model could produce a competent export-and-close walkthrough, and the business does not want it to, because a customer announcing an intent to leave is a conversation someone should have. Routing that to a human is not an admission that the fallback prompt is weak; it is the correct destination, and encoding it means the fallback path needs its own small decision about which of the two it takes.

The logging is what turns complaints into a workstream#

An unmatched request that is answered badly and then forgotten looks like a one-off. Three hundred of them look like a category. The difference is entirely whether the fallback event was recorded with enough structure to group by.

{"ts": "2026-03-17T09:22:41Z", "request_id": "r_8841c2",
 "reason": "no_class_above_floor",
 "top": ["how_to", 0.31], "second": ["billing", 0.22],
 "destination": "general_prompt", "router_tier": "cheap",
 "text_hash": "9f2c…", "excerpt": "Before we can renew we need your DPA…",
 "resolution": null, "resolved_class": null}

The last two fields are filled in later, by whoever reviews the pile, and they are the ones that make the log a dataset rather than a debug trace. Thirty minutes a week is enough: sort by the top-1 class, read the excerpts, and assign each one either an existing class — meaning the router was uncertain about something it should have got — or a new cluster name.

After two weeks, at a 3.1% fallback rate on 210,000 monthly requests, the pile looked like this.

The third column is the second multiplied by the fallback rate, which is the conversion that makes the rows comparable with anything else in the system. A share of the fallback pile is a share of 3.1% of traffic, so the 41% in the first row is 1.27% of everything, and reading the second column as though it were a share of traffic would overstate every row by more than thirtyfold.

What the fallback costs, stated as a number#

Adding a fallback is not free quality, and the honest way to see that is to price both sides of it in the same unit — points of end-to-end score.

Take the losing side first. Of the 3.1% that goes to fallback, 41% would have been answered acceptably by the class the argmax picked, which is 1.27% of traffic. Those requests now take a path that scores about 79% on the graded rubric against the 91–93% a specialist manages on its own traffic, so each of them gives up roughly 13 points. A share of traffic multiplied by the points it gives up is the contribution to the aggregate, and 1.27% of traffic losing 13 points is 0.17 points of end-to-end quality handed over.

The other side uses the same multiplication. The 1.8% that genuinely fit no class — the rest of the 3.1% — were scoring about 34% before, because a confident answer from the wrong playbook fails the rubric badly and occasionally generates a complaint that costs far more than a graded point. On the fallback path they score about 79%. That is a gain of 45 points on 1.8% of traffic, which is 0.81 points recovered. Net, the fallback path is worth about 0.64 points end to end, and it converts the worst failure mode in the system — fluent, specific, wrong — into a slower and vaguer but honest one.

The rest of the bill is operational rather than statistical. The floor is a parameter that drifts with the taxonomy and the embedding model, so it gets re-derived whenever the labelled set is re-sampled, and the fallback rate is worth alerting on directly because it moves first when something upstream changes. The human queue is about 1,000 messages a month, which is a staffing line someone has to own. And the review ritual only works if it actually happens; a fallback log nobody reads is a slightly more expensive way to lose the same information.

Falling back is not failing closed, and the fallback class is not a class#

The tempting shortcut is to return an error or a refusal for low-confidence inputs — "I'm not able to help with that, let me get a human" — on the grounds that a non-answer is safer than a wrong one. That is a belief that a defence works, and it holds for exactly one part of the population. For the 1.8% that fit nothing, refusing is defensible. For the 41% of the fallback pile that turned out to belong to a real class, it is a self-inflicted failure: the system had a perfectly good path for those requests, declined to use it because a probability came in at 0.37, and told the customer it could not help with a question about their invoice. Failing closed converts a routing uncertainty into a refusal, and refusals are visible to customers in a way that a slightly generic answer is not.

The reason the shortcut is tempting is that it treats a low score as evidence of a hard request, when a low score is evidence of a low score. A top-1 probability below the floor can mean the request fits nothing, and it can equally mean the classifier is diffuse about a request with an obvious home — the two are indistinguishable from the number alone, which is why the destination has to be something that can handle both.

It is equally not a sixth entry in the taxonomy. The classes in the labelled route set are things the router is asked to recognise; the fallback is a destination selected when it recognises nothing well. Blurring the two — labelling a few hundred junk messages as other and training the classifier to predict it — reintroduces the attractor problem from the top of this chapter, with the added twist that other has no coherent definition and so the labellers will not agree on it either, which fails the first of the two boundary tests before any classifier is involved.

The system is now complete in the sense that every request has a defined destination, including the ones the taxonomy never anticipated. It also, because the fallback path is the old general prompt kept alive and maintained, still contains a working copy of the design it replaced. That turns out to be the most useful asset available for the question the next chapter asks, which is whether any of this was worth building.