Specialist handoff hides the decision nobody is scoring

The counter-proposal in the design review was two paragraphs long and it read well. Delete the dispatcher. Give every specialist a hand_off tool, meaning a function the model can choose to call mid-answer, declared to it with a name and an argument schema in the same way as lookup_invoice or search_delivery_log. When the billing specialist opens a message and realises it is about webhook deliveries, it calls hand_off(route="api_integration", reason="customer's own retry produced the second row") and the request continues on the integration path, carrying a note that the general router could never have written. No extra model call on the happy path, no class taxonomy fossilised in a prompt somewhere, and the entity making the decision is the one with the domain knowledge — the billing specialist knows what a billing question looks like better than any classifier does.

The two-week trial produced a trace worth reading in full, because it is the design working exactly as intended rather than failing. Ticket 48211 went to billing first, the billing specialist called list_charges, saw one settled payment, called hand_off with a good reason, and the integration specialist found the duplicate delivery in one call and answered correctly. Total elapsed time 5.4 seconds against the 2.9 seconds the same ticket takes on a routed path, and two full specialist invocations billed instead of one.

The handoff worked by paying for both specialists to think about the request from scratch, and that is not an implementation detail that a better prompt removes. It is what the design is: the second specialist receives the customer's message and reasons about it with its own prompt and its own tools, because that is the only thing a specialist knows how to do.

The classification did not go away#

Nothing in that design removed a decision. Somebody still determines that this request belongs to api_integration rather than billing; the change is where that determination happens and what it is embedded in. It now happens inside a prompt whose primary job is answering billing questions, as a side effect of that prompt's reasoning, expressed as a tool call rather than as a routed label.

Count the classifiers afterwards and the cost consequence follows without any measurement. There are now four instead of one, each of them a full specialist prompt with a full tool set, which is to say the most expensive possible way to perform a five-way classification of short text. It is also the one place in the system where the cost principle from the previous chapter cannot be applied. You cannot move this decision down to a small model or an embedding classifier, because it is not a separable step with an input and an output — it is fused into the prompt that answers the question, and extracting it would be reinventing the dispatcher. The classification therefore costs strictly more than the path it selects, which inverts the property a router is supposed to have.

The argument for putting the decision there is not silly, and it is worth stating at its strongest before taking it apart. The billing specialist has read the message with the billing policy in context and, in the trace above, with the actual charge history in front of it; the router had 46 words and a class definition. On the boundary that has caused every problem in this topic, "is this charge real" is a question the ledger answers and the wording does not, so the specialist genuinely is better informed. That advantage is real and it survives everything below. What does not survive is the claim that it should replace the dispatcher rather than sit behind one.

The measurement is what disappears#

The measurement consequence is worse than the cost consequence, and it follows from what a tool call is.

A dispatcher emits a label as a discrete event on every request, whatever the label turns out to be. That means the decision can be scored offline against the labelled route set for about five cents per run, with no specialist ever invoked, and the result is a confusion matrix that attributes every error to a boundary. Handoff emits an event only when the tool is called. A tool that is not called produces nothing — no field, no null, no marker — so the absence of a handoff is not recorded anywhere as a decision that was made.

Follow that through the two cases. A specialist that receives a request outside its class and hands it off is visible in the logs, with an origin, a destination and a reason. A specialist that receives the same request and answers it anyway — confidently, from the wrong playbook, exactly as in the first chapter's ticket — produces a log line indistinguishable from a correct keep, because in both cases the log line is "specialist answered". The observable is identical while the property you care about differs, and that is precisely the situation in which a log tells you nothing. Those silent cases are the false negatives of the handoff mechanism — the requests that should have triggered it and did not — and they are the entire failure mode routing exists to control, unobservable by construction.

You can recover a version of the measurement by running the labelled route set through the whole assembled system and recording which specialist ultimately answered. That works, and it costs roughly $20 and forty minutes per run instead of five cents and two minutes, because every item now invokes at least one specialist and some invoke two. It also entangles the two things you were trying to separate: when an item ends up on the wrong specialist, you cannot tell from the transcript whether the classification was wrong or whether the classification was right and the specialist simply chose not to act on it, and those call for different fixes. The dispatcher's real product is not the label — it is that the label exists as a scoreable artifact independent of any answer.

Two specialists can bounce the same request between them#

The failure that shows up in week two of a handoff trial is circular, and it is a direct consequence of removing the single point of decision. A seat-count change that alters the subscription price is a legitimate member of both account_access and billing. The access specialist reads it, decides the substance is a price question, and hands off. The billing specialist reads it, decides the substance is a seat provisioning question, and hands back. Neither is wrong, and neither has any way to know the other has already looked, because a specialist sees the customer's message and its own prompt and nothing about the request's history. Each hop is a full specialist invocation with tool calls. In the trial this fired on eleven requests over two weeks, and the longest chain ran four hops and 14 seconds before a timeout returned an error to the customer.

The guard is straightforward and it is not optional: cap the handoff depth at one, and when a specialist attempts a second handoff, stop and take an explicit alternative path rather than honouring it. That cap has a consequence worth naming, because it is easy to read the cap as a fix rather than as a boundary. After one hop the system has no mechanism left, so the request is answered by whichever specialist happens to be holding it, which is the same silent wrong-prompt failure with an extra 2.5 seconds and an extra specialist call attached to it.

There is also an entry problem the proposal does not address. Something has to decide which specialist sees the request first. Sending everything to a default specialist is a router with one hard-coded rule and the worst possible accuracy, since every request outside the default's class pays a handoff. Sending it round-robin makes answer quality depend on a coin flip. Keyword rules are a router written in if statements, with all of the dispatcher's maintenance burden and none of its measurability. The entry decision is a routing decision; removing the router does not remove it, it just leaves it unspecified.

What the numbers say across the whole traffic#

Over the two-week trial the handoff fired on 9.2% of requests — higher than the 6.2% misroute rate of the router it replaced, because specialists also hand off requests that were theirs, particularly on the how_to boundary where "the answer is in the docs" and "this is an integration problem" are both defensible readings.

Read the latency rows together, and read them at the same percentile. The handoff design was proposed partly to avoid the router's 700ms, and by the time it shipped the router cost 45ms on the median path. So it eliminates 45ms from 90.8% of requests and adds a full second specialist invocation to the other 9.2%. The mean added latency is that handoff rate multiplied by the duration of the second invocation, which is the 239ms in the table, roughly twice the dispatcher's 129ms. At p95 the comparison is 2,900ms against 745ms, which is roughly four times.

The two-versus-four difference between those ratios is not noise; it is what a bimodal distribution does. The mean averages the fast majority against the slow minority, so it stays low. The p95 sits inside the slow minority for the handoff design, where the whole of a second specialist call is being paid. Comparing a p95 against a mean would make the gap look far larger than it is, and comparing a mean against a mean would make it look smaller than a latency budget will find it — which is why the two rows exist separately.

The cost rows tell the same story. The decision's price under handoff is the handoff rate multiplied by the price of a full specialist call, and a specialist call is the most expensive thing in the system. Handoff is not a cheaper design that trades measurability for speed. It is slower, marginally more expensive, and unmeasurable, and it looks attractive because the component it deletes is visible in the architecture diagram while the second specialist call is not.

Handoff earns its place as a complement, not a replacement#

None of this argues that a specialist noticing it has the wrong request is a bad idea. It is the only mechanism in the system that has read the message with domain knowledge, and on the hard boundary it is genuinely better informed than any classifier operating on wording alone. The design that works keeps both: the router picks, and a specialist may escalate exactly once, with a target class and a written reason, after which the request is answered wherever it lands.

ESCALATE = {
    "name": "reassign",
    "description": (
        "Call this only when the request cannot be answered with your tools "
        "and clearly belongs to another path. State the evidence in `reason`."
    ),
    "input_schema": {
        "type": "object",
        "properties": {
            "route": {"type": "string", "enum": ROUTES},
            "reason": {"type": "string"},
        },
        "required": ["route", "reason"],
        "additionalProperties": False,
    },
}

def handle(message: str) -> str:
    route, tier = classify(message)
    answer, reassignment = SPECIALISTS[route].run(message, tools_plus=[ESCALATE])
    if reassignment is None:
        return answer

    log_reassignment(message, frm=route, to=reassignment.route,
                     reason=reassignment.reason, router_tier=tier)
    # depth capped at one: the second specialist runs without ESCALATE
    return SPECIALISTS[reassignment.route].run(message)

Two properties make this version defensible where the replacement version was not. The escalation is bounded at one hop by the code rather than by a prompt instruction — the second specialist is invoked without ESCALATE in its tool list, so a second reassignment is not something it declines to do, it is something it cannot express — and the circular case therefore terminates deterministically rather than probably. And every escalation is logged with its origin, destination and reason, which turns a failure mode into a dataset. The escalation log, grouped by boundary, is a continuously refreshed list of the router's worst decisions, written by the component best placed to recognise them.

The measured trade is 5.4% of traffic taking a second specialist call, which costs 289ms of added latency on average across all requests and about $1.35 per thousand, in exchange for effective routing accuracy rising from 93.8% to 97.0%. Whether that is worth it depends on the latency budget, and the p95 row is the one to check, because for the escalated requests the added latency is not 289ms but the full duration of an extra specialist invocation.

Handoff-only is not the general prompt, and it is not orchestration#

It is worth being clear about which neighbouring designs this is and is not, because the arguments for and against handoff get borrowed from designs it only resembles. Handoff-only is not the single general prompt from the first chapter. That design performs no classification at all and answers everything with one prompt and every tool, which is a coherent position with its own trade-offs, and in particular it has no misroute failure mode because it has no route. Handoff-only performs classification four times, expensively, and records almost none of it. It is not the absence of routing but unmeasured routing, and the two get conflated because neither appears in the architecture diagram.

Nor is it orchestration in the sense of a coordinator decomposing work across agents. There is no decomposition here, only a request being passed sideways until somebody keeps it. That distinction matters for what you can expect from it: an orchestrator can produce an answer that no single agent could have produced, whereas a handoff chain can only ever produce the answer of whichever specialist stops the chain. Spending two specialist calls buys the second specialist's answer, not a better one.

What remains unaddressed by every design in this chapter is the request that belongs to none of the five classes. The dispatcher assigns it the nearest label and the specialist answers confidently from the wrong playbook; the handoff design is worse, since each specialist passes it along until the depth cap runs out and someone answers it anyway. Neither design has a destination for "this is not one of the things we route", and both fail on it silently, for the same reason the false negatives above are silent: nothing in either mechanism has a way to emit "none of these".

The next chapter builds that destination, along with the logging that turns those requests into a group you can review rather than a scatter of unrelated complaints.