The routing rule comes from plan cost against failures avoided

The argument that finally forced the measurement was two sentences long on each side. One engineer wanted planning off for anything under five steps, because the median task in the ops queue is four steps and paying 1,200 tokens and three seconds to plan a four-step task is silly. Another wanted it on everywhere, because the Saturday incident happened on a planned run and would have been worse on an unplanned one. Neither had a number. Both had been in the same argument twice before.

Both positions are about the same quantity and neither of them names it. Planning costs a fixed amount on every task it is applied to — one extra model call, whose output then sits in the message array and is re-sent with every later request — and it buys some reduction in the number of actions that fail. Whether it is worth applying to a particular task is whether that reduction is worth that cost on that task, and neither engineer had either figure. What settles an argument of that shape is a measurement, and what a measurement of this shape produces is a routing rule: a function from a task to the configuration the agent will run it under, evaluated per task rather than chosen once for the whole fleet.

The step count is the tempting input, because it is the one property of a task you can read off without running it, and on the queue below it very nearly works — a line drawn at five steps sorts all four of the trial's task classes the way the finished rule sorts them. That agreement is a coincidence of this queue, and the way to see that it is a coincidence is to ask the step count about a four-step task that issues refunds. It routes that task away from planning, because four is under five, while the finished rule routes it to planning without consulting a single success number. A property that agrees with the right answer on the cases you measured and disagrees on the case you care about is not the right input; it is a correlate of the right input on one sample.

The only comparison that settles it#

Take real tasks off the queue, run each one both ways, and record what happened. The design constraints are boring and every one of them is load-bearing.

The tasks have to be the same tasks in both arms, an arm being one configuration of the agent held fixed for the length of the trial. Comparing planned runs on migrations against interleaved runs on report extracts measures the difference between migrations and extracts. The team pulled 120 tasks from six weeks of the queue, kept the original request text verbatim, and rebuilt the world state each task started from — a snapshot of the warehouse, the bucket contents, and the vendor mock, restored before every run.

The step being skipped when that goes wrong is worth naming, because it is the same step in every botched comparison. A run's outcome is produced by two things at once: the task, and the configuration the agent ran it under. Measure a set of runs and you get one number that both of those contributed to. Change both at once between the two sets, and the difference you observe belongs to whichever of them you please, because nothing in the data separates them. Holding the task set identical is what makes the difference attributable, and the reason it works is that the task's contribution is then the same on both sides and subtracts out of the comparison.

Each arm runs multiple seeds — a seed being the value that fixes the model's random sampling for one run, so that changing it gives another draw of the same agent on the same task — because a single run of an agent is a sample from a distribution and the distribution is wide. Three seeds per task per arm gave 720 runs.

Seeds are doing the same job for a different source of confusion. The model samples, so the same agent given the same task twice can take two different routes and reach two different outcomes. One run per cell measures one draw. The difference between two draws is not the difference between two configurations, and with a single seed there is no way to tell the two apart, because both show up as one number moving. Several seeds per cell let the draw-to-draw variation appear as spread inside a cell rather than as signal between cells.

And the metrics have to include the ones that make planning look bad, or the trial is theatre. The team recorded six: task success against a written rubric per task, total tool calls, failed tool calls (a call returning an error or failing its post-check), wall-clock latency, total tokens across every model call in the run, and side-effect errors — writes to the wrong target, credits issued that should not have been.

def trial(task, arm, seed):
    world = snapshot.restore(task.fixture)
    t0, meter = time.monotonic(), TokenMeter()
    result = run_agent(task.text, mode=arm, seed=seed, world=world, meter=meter)
    return {
        "task": task.id, "arm": arm, "seed": seed,
        "success": rubric.grade(task, world),          # per-task, written before the trial
        "tool_calls": result.calls,
        "failed_actions": sum(1 for c in result.calls if c.errored or c.post_failed),
        "latency_s": time.monotonic() - t0,
        "tokens": meter.total,
        "side_effect_errors": world.diff_against(task.expected_effects),
    }

The one methodological trap worth naming before the results is that "planning" and "gating" travel together. The planned arm carries precondition probes, because probes are attached to plan steps; the interleaved arm has no plan and therefore no probes. Compare those two and the failed-actions column collapses, and you will attribute to planning what the probes did. The team added a third arm — interleaved reasoning with the same probes wrapped around destructive tools — which is what makes the numbers below mean anything.

Written out, the two-arm version is one equation in two unknowns. The number you observe when you turn the planned arm on against the interleaved one is a total, and that total is the sum of whatever the probes contributed and whatever the plan contributed. One measured quantity, two unknown ones. No arrangement of that single equation recovers either unknown, and every split you could write down fits the data exactly as well as every other — including the split that credits the plan with all of the gain, and the split that credits it with none. The third arm supplies the second equation, by holding the probes switched on and varying only the plan. Two equations and two unknowns, and the split stops being an assumption and becomes something you read off.

The three arms are nested, which is worth seeing before the numbers arrive: the interleaved arm has neither component, the middle arm has the probes, and the planned arm has both. Nesting forces the two gains to sum to the total exactly, whatever the numbers turn out to be, because the middle arm is the same measurement counted once as the end of the first gain and once as the start of the second. That is an identity rather than a finding, and it cuts both ways. It licenses the attribution, since given the total the middle arm fixes where the split falls. It certifies nothing at all about the total, which is still one measurement of one agent against one tool set on one queue, and which the arithmetic of splitting cannot make more true.

What came back#

Read that table down inside each class rather than across the classes. The levels are not comparable between classes, because a class is a set of tasks and those task sets differ in how hard they are — class D's success in the mid-nineties and class B's in the mid-sixties are facts about extracts and migrations, and swapping the arms would not close the distance between them. What is comparable is the gap between two arms inside one class, because that is the same task set measured twice with one thing changed. Every claim in the rest of this chapter is a gap, and none of them is a level.

Class A is the case for planning and it is a real one: 71% to 91%, with failed actions falling from 1.9 per run to 0.4. But the third arm splits that gain in two. Probes alone take it from 71% to 84%; planning adds the remaining 7 points. Somebody who had shipped planning and celebrated a 20-point improvement would have been crediting the wrong component for two-thirds of it.

Thirteen of those points belong to the probes and seven to the plan, which is the entire return on having run a third arm: with two arms there is a twenty-point gain and no way to divide it, and with three the division says the cheaper component did most of the work.

Class B behaves the same way and adds a wall-clock result that runs against intuition: the planned arm is 45 seconds faster despite generating a plan first, because the ordering-constraint failures from chapter 3 cost more time than the plan does. A run that waits 600 seconds on a lock is slower than a run that spends 3 seconds writing down that step 3 depends on step 2 finishing.

The step under that result is worth making explicit, because "planning made it faster" sounds like a contradiction and is not one. Planning adds a fixed few seconds to every run in the class. It removes a much larger delay from some fraction of the runs in the class. The sign of the average difference is therefore set by how often the large delay happens, not by how long the plan takes, and a delay measured in hundreds of seconds only has to be avoided occasionally to pay for a few seconds spent every time. That is why the same plan generation shows up as a cost on one class and a saving on another without anything about the plan changing.

Class C is where the argument that started this chapter dies. Planning makes investigations worse — four points of success, 1.8 extra tool calls, 45 extra seconds, 7,800 extra tokens — and the third arm beats both. This is the median four-step task. It is what the first engineer was pointing at, and the data agrees with the conclusion for a reason that has nothing to do with the step count: a plan written before the first observation on a task whose second step depends entirely on the first observation is a document that misleads the model for the rest of the run.

Class D is the pure-overhead case. Success is flat within noise across all three arms, and the planned arm costs 6,300 extra tokens and 19 seconds per run to achieve exactly nothing.

"Flat within noise" is a claim about the size of a difference against the size of the wobble, and it is checkable rather than decorative. Class D holds twenty-three tasks at three seeds, and its three arms sit inside a single point of each other while its token column moves by more than half again. A difference of one point on a sample that size is smaller than the amount an average of that size moves when the seeds are redrawn, so it is not evidence that any arm is better. The token difference is not in that position: it is large, it is in the same direction on every task, and it is paid whether or not a difference in success ever appears.

The arithmetic on the median task#

For a four-step task, plan generation is one extra model call producing about 1,200 tokens in roughly 3.1 seconds. The plan then sits in the message array for the five model calls that follow, so those tokens are billed as prompt input five times — 6,000 token-turns — on top of the 1,200 generated. Prompt caching recovers most of the price of the re-sends and none of the context window they occupy.

Written as algebra rather than as one worked case, that settles every task length instead of this one. The plan is generated once, so its output cost is paid once and does not depend on how long the task is. It then forms part of the prompt on every model call that follows, so its input cost is its size multiplied by the number of those calls. A task of n steps has n plus one calls after planning — one to choose each remaining action, and one more to produce the answer — which is where the multiplier of five on a four-step task comes from, and which says the resident cost of a plan grows with the task while the cost of writing it does not. A plan is cheap to produce and is charged rent for the rest of the run.

At the rate card this team pays, call it $3 per million input tokens and $15 per million output, the whole overhead on a class C run comes to about three cents. Three cents is not a reason to do anything. The costs that decide it are the 45 seconds, which the queue notices, and the four points of success, which the queue notices a great deal more.

Two routes reach that figure and they share no step, which is worth noticing because it is the closest thing here to a check on the accounting. The first route is the one just walked: a plan's own size, generated once, re-sent a number of times set by the task length, priced at the rate card. The second is the table, which recorded total tokens per run and puts class C's planned arm 7,800 tokens above its interleaved one; price that measured difference at the same rate card and it lands in the same place. One route is derived from the shape of the mechanism and the other is a subtraction between two columns of an experiment, and their agreement licenses a narrow but useful claim: the token accounting in this chapter is not a model of the cost, it is the cost. The residual is caching. Prompt caching cuts the price of the re-sent half of the first route and leaves the generated half alone, so the agreement holds between the uncached figure and the measurement, and a run on a warm cache pays less than either.

So on the median four-step task the ledger reads: 1,200 tokens and 3.1 seconds spent, 0.3 additional failed actions incurred, four points of success lost. Planning is not paying its overhead there; it is charging for the privilege of doing worse. Killing it for class C is the right call, and killing it globally on the strength of that same median would cost class A seven points and class B nine — on the tasks that move money.

Those three success gaps are not all measured from the same starting point, and stopping on that is worth a paragraph, because adding numbers taken from different baselines is the most reliable way a comparison like this goes wrong. The four points class C loses are planning measured against the plain interleaved arm. The seven and the nine are planning measured against the arm that already carries probes, which is the right reference for a decision about planning alone, since the probes stay switched on either way and are not part of what is being decided. Read against that same probes arm, class C's loss is a point wider rather than narrower. The direction is the same on both references and only the size moves, so the verdict on class C survives the choice; a reader who spots the mixed reference should know that it changes the number and not the answer.

The rule that falls out#

What the trial produces is a router, and its inputs are not the step count. Class C's median is four steps and class B's is five, and they sit on opposite sides of the line; class D is shorter than C and lands in the same place for a different reason.

Three features predict which side a task falls on.

Does any step have an effect of money or external? If so, plan, and route on this feature alone regardless of what the success numbers say. Every class A task has one, which is why class A is class A. This is also the feature the trial cannot measure, and that is the important caveat: the Saturday incident is roughly a 1-in-200 event, and a 120-task trial with 3 seeds will not contain one. The reason to plan an irreversible task is the reviewable artifact from chapter 2, and its value lives in a tail that means-of-120 will never show you. Do not let a trial talk you out of it.

The mechanism behind that caveat sits in the trial's design rather than in its size, and it is worth following through, because the same shape turns up in every benchmark anybody builds. Every run started from a restored snapshot: the warehouse, the bucket contents and the vendor mock all put back to a known state before the run began. That restoration is what makes the arms comparable, since it removes the world as a source of difference between them. It also removes the world as a source of failure. The Saturday incident happened because a lifecycle rule deleted an object while a run was in flight, and a fixture reset to a known state before every run cannot produce that. So the trial is not merely too small to see the event — its design forecloses it. A measurement whose inputs exclude the failure you are guarding against will report the same value whether or not that failure exists, and a number that cannot move is not evidence about the thing it cannot move for.

Is the path knowable before the first observation? Class A and B tasks are pipelines whose shape is fixed and whose dependencies are real properties of the systems involved. Class C tasks branch on their first result. A useful proxy that a router can actually compute: does this task class have a template — a checked-in SQL file, a runbook, a previous plan with the same fingerprint that succeeded? If there is a template, a plan is mostly a fill-in and it is cheap and accurate. If there is not, the plan is a guess.

What does the paired measurement say for this class? For everything not settled by the first two features, the trial's success gap is the tiebreaker, and it needs a threshold wide enough to survive the noise in 3 seeds over 20-odd tasks. Five points was the team's line; class C's −4 and class D's −1 both sit inside it in the wrong direction, and both route to interleaved.

Both of those gaps are quoted against the plain interleaved arm, which flatters planning slightly. Measured instead against the arm the router actually falls back to — the one with probes on — class C's gap is wider in the same direction and class D's is unchanged, so neither reaches the threshold on either reference. A threshold is needed at all because the gap is a difference between two averages, each of them an average over a handful of draws on a couple of dozen tasks, and averages of that size move on their own. A gap narrower than the amount they move on their own is not evidence, and a router that acts on one will flip an entire task class from one arm to the other when somebody re-runs the trial. The number to choose is the smallest gap you would still believe after a re-run.

def route(task):
    if any(s in task.declared_effects for s in ("money", "external")):
        return "planned"                     # tail risk, not mean performance
    if task.klass in TEMPLATED and CLASS_GAP[task.klass] >= 0.05:
        return "planned"
    return "interleaved"                     # probes are on in both arms, always

The comment on the last line is the finding that generalises furthest. Precondition probes on destructive tools helped or was neutral in every class, cost under 1.5k tokens and 15 seconds, and is the cheapest thing in the entire apparatus. If you ship one component from this topic, ship that one.

CLASS_GAP has to be re-measured, because it is a fact about this agent against this tool set on this queue, and all three change. The team re-runs the trial when a tool's behaviour changes, when the model changes, and quarterly otherwise. A routing rule derived from a trial nobody has repeated in a year is a preference again, just one with a spreadsheet behind it.

That decay has a shape worth naming, because it is not a failure of diligence. The number in the config file cannot change on its own, while the agent, the tool set and the queue all move underneath it, and every one of those movements changes the gap the number is supposed to stand for. Staleness is therefore invisible from inside the thing being aged: no reading you can take of a stale gap distinguishes it from a fresh one, because the two are the same digits. That is why the date has to be stored beside the gap. The date is the only part of the record that goes wrong in a way you can see.

Shapes that trade the same quantities differently#

The plan-then-execute shape measured here is the simplest member of a family, and the other members trade the same quantities differently. ReWOO, short for Reasoning WithOut Observation, removes observations from the planning loop entirely — the planner writes the whole path with variable placeholders, a worker fills them in, and the model is called far fewer times, which cuts tokens substantially and gives up the ability to react. LLMCompiler goes the other way and treats the plan as a DAG whose independent branches execute in parallel, which is aimed squarely at class B, where the trial's 6m55s is mostly serial waiting on steps that do not depend on each other. LangChain's Plan-and-Execute Agents write-up compares the three on latency and cost directly, and its Plan-and-Execute notebook is a working implementation of the plan, execute, replan cycle if you want to read one end to end rather than assemble it.

None of them removes the need for the trial. They change the cost side of the ledger — fewer model calls, or less wall-clock — and leave the benefit side exactly where it is, which means the routing question is the same question with different constants.

There is one residual inside the router itself that the trial does not measure. CLASS_GAP is indexed by task class, so a task inherits whatever gap was measured for the class it was filed under, and the filing is done by something upstream of the router. A task that lands in the wrong class gets the wrong arm, and no column in the trial reports it, because the trial's tasks arrived with their classes already attached and were never asked to be sorted. The other two features do not have this exposure: a declared money effect is a property of the task itself, and a template either exists for it or does not. That is a second reason to let those two decide first and to leave the measured gap as the tiebreaker it already is.

The order to build this in is the order the numbers justify. Probes on destructive tools first, in whatever architecture you already have, because they were the best return in every class and they do not require planning at all. Then a plan artifact with a schema and a pre-execution check, for the task classes with irreversible steps, because that is where the tail risk lives. Then dependencies and success criteria, when a task class has real ordering constraints and you have watched a run fail on one. Then the gated executor and the convergence rules, which are the machinery that makes replanning safe, and which you do not need until replanning exists.

Then run the trial. Twelve tasks per class and three seeds is enough to see a ten-point gap, it costs an afternoon, and it will tell you at least one thing you were confident about and wrong on — for this team it was that two-thirds of their planning win belonged to the probes. Write the router, put the class gaps in a config file with the date they were measured, and put a calendar reminder on it. The rule you end up with is not a view about how long a task is; it is a pair of features you can read off a task before running it, plus one number you measured and will have to measure again.