Traces tell you when the shape you assumed was wrong

The ticket-resolution agent has been running on a planner–executor split for six weeks. The dashboards are calm: error rate 0.4%, p95 within budget, evaluation score 88.4% against the 88.1% the fused loop scored before the migration. Nobody has any reason to look at it, which is why the finding arrived by accident, when an engineer investigating a billing spike ran a query over the trace store and noticed that the planner's output was almost always one line long.

Those three green signals are the reason this class of mistake survives for months, and the reason is structural. An error rate counts runs that failed, and a planner that produces a useless plan has not failed. A latency budget counts seconds, and one extra call is a few hundred milliseconds. An aggregate score counts answers, and the answers are fine. Each of those instruments holds a real quantity constant, and the quantity that has gone wrong — whether the machinery the design exists for is ever exercised — is not measured by any of them. A shape that never activates is invisible to every dashboard a team normally builds.

What a plan-length histogram says#

SELECT plan_steps, COUNT(*) AS runs,
       ROUND(100.0 * COUNT(*) / SUM(COUNT(*)) OVER (), 1) AS pct
FROM spans
WHERE span_type = 'plan' AND ts > now() - interval '14 days'
GROUP BY plan_steps ORDER BY plan_steps;

 plan_steps | runs  | pct
------------+-------+------
          1 | 41220 | 70.2
          2 | 10580 | 18.0
          3 |  4890 |  8.3
          4 |  1560 |  2.7
          5+|   470 |  0.8

The query is the whole instrument, and it is worth reading rather than admiring, because a version of it exists for every topology. It groups two weeks of planning spans by the length of the plan they produced and reports each length as a share of the total. The equivalent for a fan-out groups by the number of workers dispatched; the equivalent for a chain with branches groups by which branch ran. In each case the question is the same one: how often did the design do the thing that distinguishes it from the design it replaced?

Seventy per cent of production requests produce a plan containing exactly one step, which means that for seven requests in ten the planner reads the ticket, decides that the work is "resolve this ticket", and hands it to a single executor that does what the fused loop used to do. The split is doing nothing for those runs except adding a model call, about 2,100 input tokens and 180 output tokens, and one more place for the run to go wrong. The evaluation score being unchanged is consistent with exactly that: a design whose distinguishing machinery is inert on most of the distribution will score the same as the design it replaced.

The conclusion is not that planners are useless, and it is not that the team was foolish. It is narrower and more actionable: the shape hypothesis from chapter 1 was correct for a minority of the traffic and wrong for the majority, and the design applies the minority's topology to everything. That minority is the last three rows of the histogram added together — eight point three, two point seven and nought point eight per cent — which is the 11.8% of requests with three or more steps. Those may well be getting real value; the query cannot tell you that on its own, and the next query can.

One residual travels with every figure in this section and it is easy to miss. The bucketing is by the planner's own output, so a ticket that genuinely decomposes and that the planner wrongly summarised as one step is counted in the seventy per cent. The histogram is therefore a measurement of what the design did, which is exactly the right instrument for asking whether the machinery activated, and it is not a measurement of what the work needed. Those two readings come apart precisely when the planner is bad, which is a hypothesis the histogram cannot rule out on its own.

Splitting the metric before splitting the traffic#

Before routing anything, score the two classes separately, because an aggregate that mixes them will hide both effects. Grading the same evaluation suite with the cases partitioned by the plan length the planner produced gives a much sharper picture:

The last row is a weighted average and not a fourth measurement, which is worth confirming on the column where it is cleanest. Seven tenths of the traffic at 3.0 calls, not quite a fifth at 3.4, and the remaining ninth at 4.1, each multiplied by its share and added, comes back to the 3.2 in the overall row for the fused loop. That is what an aggregate is: the class values weighted by how much traffic sits in each class. Two things follow. A class can move a long way without moving the aggregate, if its share is small. And the aggregate can move without any class moving, if the shares change — which is the trap in the next paragraph.

The 6.7-point gain on the decomposing class is a real effect and it is being averaged into invisibility by the 70% of traffic where the planner costs a call and returns nothing. That is the practical reason to keep per-class metrics: the aggregate number that justified the migration and the aggregate number that would justify reverting it are both averages over a mix that the design itself is not uniform across.

There is a trap in the other direction as well. Once you route, the aggregate score will move even if neither path changes at all, because you have changed which cases go where. Report per-class scores and the routing distribution alongside any overall figure, and treat a change in the aggregate as a question about the mix until you have ruled that out.

Route by something you know before you pay for the plan#

The obvious fix — run the planner and skip the executor split when the plan has one step — saves nothing, because the planner call is the cost you are trying to avoid. That is worth stating as a causal chain rather than as a rule, since the same reasoning applies to every routing decision in this book. The expense being removed is one model call. A decision that reads the plan can only be made after the plan exists. A plan exists only once the call has been made and paid for. So a router that consults the plan has spent the money it was built to save, and the only thing it can still avoid is the executor's share of the work.

The routing decision has to be made from information available before any model call, or from a cheap signal, and there are three practical sources.

Request metadata is the cheapest and covers more than people expect. On this system, tickets arriving from the in-app widget with no attachments and a single message were 91% of the one-step class, and that is a WHERE clause rather than a model. A small classifier is the next option: a fast, cheap model given the ticket and asked only whether the work decomposes, at roughly 300 input tokens and a tenth of the planner's cost. The third is escalation, where the fused loop runs first and hands off to the planner when it hits a defined trigger — eight iterations without a resolution, or an explicit needs_decomposition tool the model can call — which has the advantage of being driven by what actually happened rather than by a prediction, and the disadvantage of paying for a partial run before escalating.

The escalation edge from the loop back into the planner is what keeps the router honest, because a router that mistakenly sends a decomposing ticket to the loop does not lose the ticket; it costs an extra few seconds and lands in the right place. The three sources also differ in when they are wrong, which is the property to choose on. Metadata and a classifier are predictions made before the work starts, so their errors are silent by default. Escalation is a reaction to something that has already happened, so its errors announce themselves in the trace. A design that uses a prediction for speed and an escalation edge for safety gets the cheap path most of the time and a recorded correction when the cheap path was wrong.

The case for one shape everywhere, and where it fails#

A staff engineer will make the opposite argument, and the strong version of it is not about elegance. Standardise every agentic feature on an orchestrator with workers and you get one span schema, so a single dashboard and a single alerting rule cover every feature. You get one retry and idempotency story, written once and reviewed once, rather than six variations with six sets of edge cases. You get one on-call runbook, so the engineer paged at 3am does not have to learn which of six architectures this feature uses before they can read the trace. And you stop having the architecture debate per feature, which is not a hypothetical cost: across six features in one quarter, the design discussions, prototypes and review cycles for this question consumed something like thirty engineer-days. The clinching move is that the pattern degenerates gracefully, since an orchestrator with a single worker is a loop with extra ceremony, so nothing is lost by standardising.

That last move is the one to examine, because it is a claim that a defence exists and the defence is never exercised. It is true that the degenerate case is expressible: an orchestrator with one worker really does behave like a loop. What is held constant by that argument is the capability of the pattern, which genuinely covers the simple case. Where it breaks is that the degenerate case is never what gets built, so what varies is what teams actually produce when the machinery is in front of them. Once the fan-out machinery is present in the framework everyone imports, features grow into it: a job that needed three sequential prompts acquires a planner because the planner is what the template has, and a second worker because there is a slot for one. The standardisation argument quietly becomes a floor on complexity for tasks whose shape never needed coordination.

The production symptoms are specific enough to look for. You expect the plan-length histogram above, but org-wide, with most features showing a majority of one-step plans. You expect p99 latency across the estate to be dominated by slowest-worker effects, since a fan-out's tail is a maximum rather than a sum. You expect the top cause in incident reviews to be partial-result handling — a worker returning empty and an aggregation step interpreting the silence — which is chapter 5's first line item showing up as a quarterly trend. And you expect the delivery symptom that is hardest to attribute: a feature that is genuinely a three-prompt chain taking two weeks to ship instead of three days, because it has to be expressed in the vocabulary of orchestrators and inherits their state, retry and resume obligations.

The proposal that keeps most of what the staff engineer wants puts the uniformity in the tooling rather than in the topology.

Read the two columns as a gap rather than as a scorecard. On the first two rows there is no gap at all, which is the finding: those benefits were never properties of the topology, they were properties of shared instrumentation, and shared instrumentation can be had without mandating a shape. The gap that remains is on the last two rows, and it runs in opposite directions — a small residue of debate on one side, a complexity floor on every feature on the other. A three-item menu with a required decision record removes most of the thirty engineer-days without making fan-out the price of entry. The debate that remains is the one worth having, and it now terminates in a document with a flip_if block instead of in whoever is most senior in the room.

What the six weeks of traces really bought was a habit rather than a routing rule. The topology you picked was a hypothesis about the distribution of work, formed before you had any production traffic to look at, and production is the only thing that can tell you whether the distribution matched. Put the plan-length histogram, the per-class scores and the calls-per-request figure on the same dashboard as your error rate, review them when you review anything else, and treat a shape that never activates as a finding rather than as a harmless default. If you want the shapes themselves as runnable code to A/B against your own traffic, the LangGraph workflows and agents documentation implements each of them small enough to read in a sitting, which is the right next thing to have open, because everything from here is about building the shape you argued for and instrumenting it well enough that the next argument is short.