Peer routing removes the bottleneck and pays in hops

Three incidents landed in the same week on a five-agent swarm. The billing agent answered a question about shipping. Two agents each emailed the same supplier about the same stock query, four minutes apart, and the supplier replied to both asking which one to answer. A conversation ended with an answer that read as complete and was missing half of what the customer had asked, and nobody noticed until the customer wrote back. The director's question at Thursday's review was whether the team should move to a supervisor, meaning the orchestrator shape of topic 11.9, where one component holds the goal and every result comes back to it.

The obvious response is that these are three bugs of three kinds — a routing bug, a duplication bug and a termination bug — and each has a plausible fix: tighten the routing prompts, add a shared claims list, require a confidence threshold before an agent may finish. Every one of those fixes is a guess, because none of the three incidents can currently be reconstructed. Tightening a routing prompt without knowing which agent routed wrongly and what reason it gave is editing the prompt you happen to suspect. A confidence threshold is worse than a guess, since chapter 4's carry-rate table found confidence propagated on zero of 1,043 transfers, so the threshold would gate on a self-reported number that no receiver has ever seen and that nobody has calibrated. What the three incidents share is that they are all composition failures — properties of a sequence of transfers rather than of any agent — and per-agent logs cannot express a sequence of transfers at all.

One build, and the things deliberately not built#

The build for the quarter is the run-scoped trace from chapter 5, emitted from the transfer wrapper, with the payload contract from chapter 2 as the thing it records. That single artefact makes all three incidents investigable: the routing incident becomes a sequence of reason strings, the duplication becomes two tool calls with the same side-effect identity under one run id, and the partial answer becomes a run whose stop_conditions were never met and whose last hop said so.

What that means not building is the three point fixes, in that order and for that quarter. Prompt-tightening is deferred because it is measurable afterwards and unmeasurable before. The claims ledger is genuinely needed, and it is second rather than first, because the trace will say whether the duplicate supplier email came from two agents both deciding to act or from one agent transferring after a failed attempt, and those want different remedies. The confidence threshold is not built at all, and the field it depends on gets added to the payload instead, so that a future version of the idea has an input.

Also not built this quarter: the supervisor migration. The reason is the subject of the rest of this chapter.

The trade, held to the same workload#

Both designs were run against the same 200-conversation labelled set, the same models, the same budget ceiling of $4.00 and 180 seconds per run, and the same five capabilities. The supervisor column comes from an offline replay described further down rather than from production, and it is marked as an estimate where that matters.

The supervisor wins most of the rows and the ones it wins are the ones that show up on an invoice. Its structural advantage is that routing state is compact — 1.4k tokens rather than a growing transcript — so each routing decision is cheap, consistent, and made by the same prompt every time, which is also why it is easier to evaluate: there is one classifier — a single component whose whole job is to pick a destination out of a fixed list — to build a labelled set for.

The row where the swarm wins is worth understanding rather than dismissing, because it is the honest case for the design. On multi-issue conversations the swarm routed correctly 91% of the time against the supervisor's 82%, and the gap comes from information that only the specialist holds. When the refunds agent has just read a policy page and seen that the order shipped from a third-party seller, it knows the next hop is orders; the supervisor sees a summarised result that says "refund declined, marketplace order" and routes on that summary. Every compression of the state loses something, and the specialist is the component that has not compressed. That advantage grows with the depth of each specialist's tooling and shrinks when the router's state schema is rich, which means it is a real advantage and a purchasable one.

The evidence a decision needs#

A director asking for a recommendation is asking for a number, and there are six worth having before answering. Destination accuracy against a labelled set of correct next agents, split by single-issue and multi-issue, because the aggregate hides the row where the designs differ. Hops per resolved conversation at p50 and p99, because the tail is where money goes. Cost per resolved conversation rather than per conversation, since a cheap design that resolves less is not cheap. Duplicated external side effects per thousand runs, counted from side-effect identity. Time to explain a routing incident, which was 3.5 hours before the trace and 11 minutes after. And the proportion of runs ending partial with the gaps named, which measures honesty rather than capability.

The strongest evidence is the one that does not require shipping anything, and it exists only because the trace records state_before and state_after on every hop. Each of those digests is what a supervisor would have had at that moment, so the routing decisions of a supervisor can be replayed against real runs, offline, and scored against the same labels.

def replay_as_supervisor(runs: list[RunTrace], router: Agent) -> Report:
    correct = hops = cost = 0.0
    for run in runs:
        for hop in run.hops:
            state = compact(hop.state_before)          # what a router would see
            decision = router.decide(run.objective, state)   # 1.4k in, ~60 out
            cost += price(decision.tokens_in, decision.tokens_out)
            hops += 1
            correct += (decision.to_agent == hop.label_correct_agent)
    return Report(accuracy=correct / hops, est_cost_per_conv=cost / len(runs), ...)

That replay cost $41 to run across 200 conversations and produced the supervisor column above, which is a materially better basis for an eight-week migration than an argument about architecture. Its limits should be stated when the number is presented: the replay scores the router's decision at states the swarm produced, so it cannot show what states a supervisor would have reached instead, and it flatters the supervisor slightly on the long runs, where a real orchestrator would have re-planned rather than continued.

Saying no while losing#

At 2,400 conversations a day, the $0.33 per-conversation gap is about $790 a day, and a difference of $24,000 a month usually ends the discussion. Three things argue for declining anyway, and they are all quantities rather than preferences.

The first is where the gap comes from. Of the excess hops in the swarm, 68% are on one edge, refunds ↔ notifications, and chapter 4 already identified why: constraints and stop conditions were being dropped there, so notifications transferred back for clarification it should have received the first time. Fixing the payload composition on that one edge is a two-week change with a measurable target, and if it recovers most of the excess hops it closes most of the $0.33. Deciding an eight-week migration before running a two-week experiment that might make it unnecessary is a bad trade even when the eight weeks would work.

The second is the direction the workload is moving. The roadmap adds twelve more specialist capabilities over the next year, and the supervisor's compact state is the thing that has to represent all of them; every new capability means a schema field, a router prompt revision, and a re-evaluation of a classifier that now chooses between seventeen destinations. The swarm's cost grows in hops, which budgets bound, and the supervisor's cost grows in a prompt that one team owns and everyone edits. Neither is obviously better, and the measured multi-issue accuracy gap of nine points suggests the compression problem gets worse rather than better as specialists get deeper.

The third is that the swarm's numbers were taken from an instrumented system that has existed for six weeks. The trace, budgets and payload contract were built precisely because the design was failing, and the team has not yet had a quarter to act on anything they showed. Comparing a supervisor design's estimate against a swarm's first six weeks of measured operation, and migrating on the difference, discards the instrumentation along with the design.

The answer to give the director is therefore a conditional with a date on it: run the refunds ↔ notifications payload fix, re-measure cost per resolved conversation and hops at p99 in six weeks, and if the gap is still above $0.20 per conversation, migrate. The same conditional also names what would flip it immediately, which is a duplicated side effect that costs real money, since the swarm's 3.1 per thousand against the supervisor's 0.4 is the one row where the difference is an external consequence rather than an invoice line.

For a swarm already in production, the order of work is the order of these chapters, and each step is small enough to ship on its own. Type the payload so the receiver is handed a contract instead of a transcript. Write the allowed edges down and enforce them in the transfer wrapper, with real handoff descriptions that name the exclusions. Emit the routing event from that same wrapper, with the model's own reason on it. Attach a budget to the run and make exhaustion a terminal answer with its gaps named. Then score the carry rates, watch the edges, and ask the supervisor question again with a replay rather than an opinion.