Your route classes are a design you derive from real traffic

The five classes a support router chooses among have to come from somewhere, and in the assistant this topic follows they were not designed at all. They were copied from the names on the support team's queues — Billing, Technical, Account, General — plus a fifth that got added when someone noticed how many messages just asked whether the API was down. That origin is worth stating because it is the default, and because it produces a specific and measurable failure. When two support engineers were given 600 sampled conversations and asked to assign each one a route independently, they agreed on 71% of them. Nobody was being careless. Of the 174 disagreements, 66 sat on the line between Billing and Technical, and almost all of those were one of two shapes: "we were charged for something our integration did" and "the invoices API returns 404 for an invoice I can see in the dashboard."

The taxonomy — the fixed set of classes the router chooses among, together with the written definition of each one — is the artifact under discussion here, and a router trained, prompted, or measured against a taxonomy where two experienced humans disagree a quarter of the time cannot be more than about 71% accurate in any meaningful sense against either labeller, because there is no fact of the matter for a large slice of the traffic. This is where the phrase "routing is easy" starts to come apart. Separating incident_status from everything else is genuinely easy — the two labellers agreed on 99% of those. Separating billing from integration on this traffic was hard for people who answer these tickets for a living. Difficulty is a property of a boundary, not of routing, and a single sentence about routing in general is a sentence about an average of boundaries that are nothing like each other.

Traffic decides the classes, not the org chart#

Support queues and route classes exist for different reasons, and the reasons determine what a good boundary looks like in each case. A queue exists to distribute work across humans with different training and different shift patterns, so a queue boundary is good when it matches how people are trained and rostered. A route class exists to bind a request to a prompt, a tool set, and a context budget, so a class boundary is good when the two sides need different prompts, different tools, or different amounts of window. Those two purposes coincide sometimes and diverge often. The org chart has a Billing queue because there is a billing team; the routing taxonomy needs a billing class only if there is a body of billing knowledge and a set of billing tools that a prompt is meaningfully better for having exclusively.

The derivation runs the other way round. Sample a few hundred real requests — 400 is enough to start, drawn across the whole week rather than one Monday morning, since weekend traffic skews toward incidents and month-end skews toward invoices. Read them and write down, for each one, what would actually have to happen to answer it: which document, which system, which query. Then group by that answer path rather than by topic.

Grouping that way works because the answer path is the thing a class binds. A class is a prompt plus tools plus a budget, so two requests belong in the same class exactly when the same prompt, tools and budget serve both, and "what would have to happen to answer this" is a direct reading of that condition off a real message. Grouping by topic asks a different question, one about what the message is about, and nothing in the routed design depends on the answer to it. A message about a failed payment and a message about a failed webhook are the same topic to a taxonomist and completely different work to whoever answers them; a message about "how do I change my billing email" and "how do I add a webhook endpoint" are different topics and the identical work, which is a documentation lookup.

Grouping by answer path collapses distinctions that feel important and creates ones that feel arbitrary. That is the signal it is working, because the two groupings would only agree if topic and answer path were the same thing, and the whole reason to do the exercise is that they are not. What comes out of it is a set of candidate classes, and the candidates then get tested.

Test every boundary twice#

Each candidate boundary faces two questions, and it has to pass both.

The first is whether anything can separate the two classes from wording alone. The router sees the message text and nothing else — no account history, no prior tickets, no knowledge of whether this customer has an integration at all. If a human holding only that text cannot reliably assign the class, the router will not either, and no prompt engineering closes that gap because the information is not present. That last step is the one worth slowing down on: a prompt can tell a model how to weigh evidence, and it cannot supply evidence that is absent from the input, so a boundary that needs the account history is unreachable from a component that never sees the account history. This is the question the inter-labeller agreement — the share of sampled items on which two people labelling independently chose the same class — answers directly, per boundary.

The second is whether the two classes genuinely need different knowledge or tools. If both sides of a boundary would be answered by the same prompt with the same tools against the same documents, the split buys nothing and costs router errors. Every boundary you add is another surface for the classifier to be wrong on, and a boundary with no downstream consequence is a pure liability.

The refunds/invoices row is the one that catches people, and it is worth being explicit about why, because the mistake it invites is a belief that a defence works. High agreement feels like a licence to split: if two people can tell refund questions from invoice questions 96% of the time, the boundary looks solid, and the router will find it easily. Both of those observations are true. Neither of them is the test that matters. Agreement measures whether a boundary is findable; it says nothing about whether finding it changes anything. This split was proposed because the refund policy is long and the person writing the prompt wanted it isolated, and it fails the second test outright — same prompt, same three tools, same policy doc — so splitting it would have added a boundary with a 4% error rate in exchange for nothing on the other side. Separability is the quantity that stays reassuringly high while the property you actually care about, the consequence of the split, varies from decisive to zero.

The account_access/how_to row fails the first test in a way that turned out to be fixable, and that repair is the general move. When two classes are not separable from wording, the options are to merge them, to redraw the line somewhere the wording actually changes, or to accept the boundary as permanently noisy and plan for it — which in practice means the fallback path and the escalation path that later chapters build. What is not an option is leaving it as it is and expecting the router to resolve an ambiguity the taxonomy created.

What an agreement number licenses, and what it does not#

Two agreement figures on the same set can mean different things, because agreement includes whatever two labellers would have reached by chance. Cohen's kappa discounts for that: it is the agreement rate discounted for how much of it two people guessing at the class proportions would have produced. Write the observed agreement as po and the chance agreement as pe. Of the agreement available above chance, which is 1 − pe, the labellers captured po − pe, so kappa is the ratio

κ = (po − pe) / (1 − pe)

and that formula is worth running in both directions on this taxonomy's two measurements. Before the re-cut, agreement was 71% at a kappa of 0.63; after it, agreement was 89% at a kappa of 0.86. Rearranging the identity for pe gives pe = (po − κ) / (1 − κ), and substituting each pair returns very nearly the same chance-agreement rate for both. That agreement between two independent measurements is the useful part. It licenses one specific claim: the jump from 71% to 89% is a real improvement in how separable the classes are, rather than an artefact of the class proportions shifting so that guessing got easier. The residual is that kappa is still computed from the labellers' own marginal frequencies, so it says nothing about which boundary improved, and a taxonomy can raise its kappa by merging one hopeless boundary while leaving four others exactly as bad.

The ceiling this implies is also worth stating with its conditions attached, because the unconditioned version is stronger than the evidence supports. What 71% agreement bounds directly is how well a router can match the judgement of any one labeller, since for a quarter of the traffic there is no answer both labellers would sign. It does not bound how well a router can match an adjudicated file, where a third person has broken every tie and the ambiguity has been decided by fiat rather than removed. A router can score well against such a file and still be guessing on the same messages the humans were, which is why the adjudicated label and the record of the disagreement have to travel together.

The other number a labelled set has to survive is per-class volume, and the arithmetic is unforgiving in a way percentages disguise. The 600 sampled items break down roughly as how_to 38%, billing 27%, api_integration 19%, account_access 11%, incident_status 5%. That last share is a comfortable-looking figure and it contributes 30 items, which is too few to say anything about the class. The resolution of a proportion measured on 30 items is one item in 30, and one in 30 is 3.3 points, so a single relabelled message moves that class's measured accuracy by more than the difference most router changes are trying to detect.

The fix is to stratify the set, which means drawing it in deliberate per-class quantities rather than uniformly at random: sample proportionally for the aggregate figure, then top up the small classes to at least 80 items each, and keep the traffic weights alongside the labels so both a per-class number and a traffic-weighted number can be computed from the same file. Keeping the weights is what makes stratification safe. A stratified set no longer matches production, so any aggregate read straight off it is wrong by construction, and the weights are the thing that lets the aggregate be reconstructed anyway.

Building the labelled route set#

The relabelled set is the artifact everything downstream consumes, and it is worth being precise about how it gets built, because the shortcuts are all tempting and all corrosive.

Who labels: people who answer these requests, not the engineer building the router. The engineer knows what the router is likely to find easy, and that knowledge leaks into the labels in exactly the direction that inflates the eventual accuracy number. Two labellers per item, independently, with no visibility of each other's choice.

What happens to disagreements matters most. The instinct is to send them to a third person, take the majority, and move on with a clean file. Do the adjudication, but record the disagreement rather than erasing it, because a disagreement is a measurement of the boundary and it is the most useful measurement in the set.

{"id": "48211", "text": "Our finance system shows two payments against the same invoice...",
 "labels": ["billing", "api_integration"], "route": "api_integration",
 "agreed": false, "boundary": "billing|api_integration", "weight": 0.27,
 "note": "customer's own retry handling; ledger is correct"}
{"id": "48219", "text": "Is the payments API down? We are seeing 503s since 08:40.",
 "labels": ["incident_status", "incident_status"], "route": "incident_status",
 "agreed": true, "boundary": null, "weight": 0.05}
{"id": "48244", "text": "How do I add a second admin to our workspace?",
 "labels": ["account_access", "how_to"], "route": "how_to",
 "agreed": false, "boundary": "account_access|how_to", "weight": 0.38,
 "note": "post re-cut: no broken access, no change needed on our side"}

Each record keeps both raw labels as well as the adjudicated route, which is what makes the agreed flag recoverable rather than a claim somebody has to remember. The boundary field turns the file into something you can query. Counting disagreements per boundary gives the difficulty ranking directly — billing|api_integration at 66 of 174 before the re-cut, account_access|how_to at 41, everything else in single digits — and that ranking predicts, with unnerving accuracy, where the deployed router's errors will land.

That prediction is not a coincidence, and the reason is the first test above. A boundary where two humans disagree is a boundary where the wording does not settle the class. The router sees only the wording. So the information that would decide the case is absent from the router's input for exactly the messages the humans could not decide either, and the classifier's errors have to concentrate there. When the confusion matrix arrives in the next chapter — the table counting, for every class, how much of its traffic the router sent to each of the others — its worst cells are the boundaries with the most human disagreement in this file. Building the file with the disagreements preserved means the prediction is available before the router exists.

This set is not the set you score specialists with#

The specialist fixtures — the 40 billing questions with expected answers, the 35 integration traces — measure whether a prompt answers its own traffic well. The labelled route set measures whether traffic reaches the prompt it belongs to. They contain different data, they are used at different points, and conflating them is the mistake that makes an entire class of failure invisible.

The clearest way to see the difference is to notice what each one assumes. Every specialist fixture is, by construction, a correctly routed input: someone wrote a billing question and put it in the billing fixture file. Running that file tells you nothing about how much billing traffic actually arrives at the billing specialist, and a specialist can score 94% on its fixtures while receiving a quarter of its real traffic from a class it was not built for and losing another quarter to a neighbour. The route set has the opposite blind spot: it says where requests go and nothing about whether the answer at the destination was any good. Both numbers are necessary, and the reason to insist on both is that the aggregate end-to-end score conceals which one is failing.

Two further things this taxonomy is not. It is not an intent ontology for analytics, which is the other artifact that looks like this and behaves nothing like it. An analytics taxonomy can afford thirty classes with fuzzy edges, because nothing dispatches on it and a misclassified row moves a dashboard by a fraction of a percent. A routing taxonomy pays for every class in router error, so the pressure runs toward fewer, sharper classes than an analyst would want, and a taxonomy inherited from the analytics team arrives pre-loaded with boundaries that fail the second test.

And it is not permanent. Traffic drifts: a new product surface shows up in support two weeks after it ships, and the class that absorbed 5% of traffic last quarter absorbs 14% this one. Both of those change the answers to both tests — a new surface can make a previously separable boundary ambiguous, and a shifting mix changes which boundaries carry enough traffic to be worth the error they cost. Re-sampling and relabelling a few hundred conversations quarterly is the maintenance cost of having a router at all, and it is the line item most often left out when someone estimates what this design costs to own.

With the taxonomy fixed and the labelled set built, the router's accuracy stops being a claim and becomes an arithmetic operation on a file: count the items whose predicted route matches their recorded one. The next chapter performs it, and then does the more useful thing. A single accuracy figure averages over the boundaries this chapter has just shown to be nothing like each other, so the number gets broken apart per class, in both directions, and a specialist which rarely receives its own traffic stops looking like a specialist that answers badly.