A run that succeeds expensively passes only against a budget

A run that succeeds expensively can only be passed or failed against a threshold that somebody wrote down before it ran, and the run on the table is the case that makes that concrete. The agent produced the correct answer, every end-state assertion holds, it took forty tool calls where six is the median for that task type, it cost $2.10 against a median of $0.30, and one of those forty calls wrote a file to the shared drive that the task did not require. Somebody has to say whether it passes, and the release is waiting.

The instinct in the room is to argue it on the merits. One person points out that the answer was right and the user would have been satisfied. Another points at the file write. A third observes that the task genuinely had three busy calendars and forty calls might be what that costs. All three arguments are reasonable, and the discussion will be settled by whoever has the most standing in the room, which is a bad way to run a gate — the same run would get a different verdict next month with different people present.

The end-state assertions read the same at six calls and at forty#

The argument is unresolvable because the criterion is missing. A run that succeeds expensively is being judged against a threshold that nobody wrote down, so each participant supplies their own and then defends it. Deciding after the fact also has a specific bias attached: the correct answer at the end of the run makes the path look justified in retrospect, and forty calls that produced a wrong answer would have read as obvious waste to the same people.

Arguing on the merits feels like a defence for a nameable reason. Everybody in the room is competent, every argument offered is about a property of the run rather than about anyone's preferences, and the strongest of them is anchored in something objective: the end-state assertions all pass. An end-state assertion is a check on what the world looks like once the run has finished — the meeting exists, on a slot all three attendees were free for, with an invitation sent — rather than a check on the path taken to get there. Those checks are green, and green is a real measurement.

What makes it not a defence is that the green is pinned while the thing anyone should be worried about moves freely underneath it. The same assertions hold for this forty-call run, identically for a run that reached the same end state in six calls, and for one that took several times forty and spent proportionally more. Correctness sits at pass across that entire range while cost, latency and blast radius vary with no upper bound at all. A quantity that does not move while the dangerous property does is the shape to be suspicious of, and it is why an argument on the merits cannot be won: the one fact everybody agrees on carries no information about the disagreement.

That is the stronger of the two traps, and the retrospective bias named above rides along with it. Between them they make the verdict a fact about who was present. The remedy is a number, written down before the run existed, that the run can be checked against.

Declaring the budget before the run#

A budget attaches to a task type, is written next to the tasks, and is reviewed the way a service level objective, an SLO, is reviewed — a target written down in advance, owned by someone, and revisited on a schedule rather than in the moment it is breached. It carries three numbers with two thresholds each, and a fourth field that is not a number at all.

task_type: schedule_meeting
variant: multi_attendee_cross_tz
budget:
  steps:      { soft: 12,   hard: 25 }
  cost_usd:   { soft: 0.60, hard: 1.50 }
  latency_s:  { soft: 20,   hard: 60 }
  allowed_effects: ["calendar.create_event", "mail.reply", "mail.send_invite"]
gate:
  per_run:      "hard breach fails the run"
  aggregate:
    amber_rate_max: 0.15      # runs between soft and hard
    p95_cost_usd:   0.75
    pass_rate_min:  0.90
  unsafe_action_rate_max: 0.0
declared: 2025-02-11
review: quarterly, or on any change to the tool set

Two thresholds rather than one is a choice somebody made, and the choice is worth separating from the requirement it satisfies. The requirement is that a gate has to do two different jobs on one quantity: fail a run that is individually unacceptable, and notice a fleet-wide drift in which no individual run is unacceptable. One line cannot do both, and which line you pick decides which job you lose. A single line drawn at the median fails half the runs and gets ignored within a fortnight; a single line drawn at the p99 never fires. Drawing both lines on the same quantity is what makes the budget survive a real set: the soft threshold marks runs worth looking at and produces a rate — the amber rate — that is gated in aggregate, while the hard threshold marks runs that are individually unacceptable.

The allowed_effects list is deliberately outside the numeric block, and its placement is a second choice with a stated reason. The requirement is that an effectful call the task never declared has to fail the run at every volume, because the file write is not a budget question. There is no volume at which an undeclared write becomes cheap enough to accept, since the objection to it is that nobody can predict what it does. The choice is how to express a requirement of that shape: a membership list on its own field, gated by unsafe_action_rate_max: 0.0, rather than a very large price on undeclared writes inside cost_usd. Pricing it would work arithmetically and would imply that some volume makes it affordable, which is the exact claim the placement exists to refuse.

That diagram assigns exactly one verdict to every run, because it checks its conditions in a declared order and the first condition that matches ends the walk. Five outcomes come out of it — incorrect, unsafe action, over budget, amber pass, green pass — and every finished run lands in exactly one. That is a partition of the run set, and the partition is what licenses reading a pass rate, an amber rate and an unsafe-action rate off the same runs: all three are shares of one denominator, and across the five buckets they account for every run once.

The residual matters as much as the licence. The order of the checks is a design decision rather than a consequence of anything, and the order changes what gets counted. A run that is both unsafe and over budget is reported only as unsafe, because the effects check runs first and the walk stops there. The unsafe-action rate and the amber rate are therefore not independent counts over two populations, and adding them does not give the number of runs that had something wrong. The over-budget count is a count of runs that breached a hard threshold and were otherwise clean, which is a different quantity from the number that breached a hard threshold.

The first version of a budget has to be written before there is any history to write it from, and the honest procedure is to run the set once without a gate, read the distribution, and set the soft threshold near the observed p75 and the hard threshold near the p95 of the runs that passed. That deliberately encodes current behaviour rather than desired behaviour, which feels like cheating and is the right starting point, because a budget's first job is to detect change. Tightening it toward what the task should cost happens afterwards, as a recorded decision with a date, and it is much easier to argue for a tighter number once you have four weeks of data showing that nothing between the p75 and the p95 was ever worth the money.

The latency thresholds need one distinction the other two do not. Total run time and time until the user has something useful are different quantities for any agent that streams, sends an interim acknowledgement, or works asynchronously against a thread rather than a chat window. The scheduling agent replies to an email, so a fifty-six second run is invisible to everyone; the same policy behind a chat box with a spinner is a support ticket. Which quantity the budget governs should be stated in the file, because a team that gates on total run time for an asynchronous product spends months optimising something no user experiences.

Running that gate over the whole set gives the aggregate check its inputs: a pass rate, an amber rate, a p95 cost, and an unsafe-action rate. A release can then fail for a reason nobody had to argue about, such as an amber rate of 0.22 against a ceiling of 0.15 while the pass rate held at 0.93.

The exchange rate between extra work and reliability#

Budgets are not set by finding the smallest number the agent currently achieves. They encode a trade, and the trade is real: an agent permitted three availability searches instead of one succeeds more often, and the extra calls cost money on every run including the ones that did not need them.

Measure it directly by running the set at several caps.

The levels in that table are not where the finding is. Subtract each row from the one above it and the pass-rate column goes seven and a third points, then a little under two points, then half a point, then two-tenths of a point. The same reading of the amber-rate column runs the other way: seven hundredths, then ten, then twelve, then twenty-one. Every additional slice of headroom buys less success than the slice before it and flags more runs than the slice before it, and those two sequences are the content of the table.

The columns move in opposite directions because they count different things about the same distribution of run lengths. Real runs of this task pile up well below any of these caps, with a tail stretching upward, so a cap is a line drawn against a bulk that sits far beneath it. Raising the cap releases runs from that tail which were previously cut off. A released run counts toward the amber rate the moment it crosses the soft threshold, whatever it goes on to do; it counts toward the pass rate only if the extra steps let it reach a correct end state. Runs that were failing for want of one or two more searches get fixed early and stop arriving, while the runs still failing at a cap of 40 are failing for reasons more steps do not fix — so the soft-threshold count keeps moving long after the outcome has stopped moving.

Going from 8 to 12 buys 7.3 points of success for $0.07 per task, and going from 25 to 40 buys 0.2 points for $0.25 per task.

Turning that into a decision needs one more figure, which is what a successful task is worth. A scheduling agent that saves a person about four minutes of back-and-forth, at a loaded cost of $60 an hour, is producing roughly $4.00 of value per successful task.

With that figure in hand the decision can be written as a rule, and the rule is worth writing before any row is evaluated, because a rule settles every row of the table while an arithmetic instance settles one. Raising a step cap is worth doing when the gain in pass rate it buys, multiplied by the value of one successful task, exceeds the increase in mean cost per task. The first term is an expectation rather than a certainty: each task attempted has a chance of succeeding equal to the pass rate, so lifting that rate by some amount adds that amount times the value, averaged over every task attempted. The second term must be charged on the same base for the comparison to be legal, and it is — extra headroom raises the average bill across all runs, including the ones that never went near the old cap. That is why the mean-cost column is the one to subtract, rather than the cost of the runs that went long.

Evaluate the rule on the two transitions already computed. From a cap of 8 to a cap of 12 the pass rate goes 91.3 minus 84.0, the 7.3 points quoted above, or seventy-three thousandths as a fraction; the mean cost goes 0.28 minus 0.21, the seven cents quoted above. At that valuation the gain multiplied by the value is 0.073 × $4.00 = $0.29 of expected value against $0.07 of cost, so the rule says yes by a factor of about four. From 25 to 40 the pass rate goes 93.8 minus 93.6, two thousandths, and the mean cost goes 0.72 minus 0.47. Multiplying again gives $0.008 against $0.25, so the rule says no by a factor of about thirty, and moving from 25 to 40 is clearly wrong.

The rows nobody computed are the reason for having a rule at all. From 12 to 18 the pass rate gains 93.1 minus 91.3, a little under two points, worth about seven cents at four dollars a successful task, while the mean cost goes 0.36 minus 0.28, eight cents — close enough to call a tie, and refused by a hair. From 18 to 25 the gain is 93.6 minus 93.1, half a point, worth about two cents, against 0.47 minus 0.36, eleven cents, refused outright. One statement has settled all four transitions, and it would settle a fifth if somebody measured one.

The knee of that curve, the cap beyond which each additional permitted step buys almost no extra success, sits between 12 and 18, which is where the declared budget above came from. In ordinary words the knee is the last cap at which one more unit of headroom pays for itself: past it, the extra success that unit buys, converted into money, no longer covers what it adds to the average bill. In the language of the rule it is the cap at which the derivative of pass rate with respect to cap falls below the derivative of mean cost with respect to cap divided by the value of a successful task. Both sentences describe the same point on the same curve, and the second is useful only to somebody who already believes the first. Estimating the value per task to the nearest factor of two is enough to make most of these calls, and refusing to estimate it does not avoid the decision — it just means the trade gets made implicitly by whichever threshold someone happened to type.

The same run at three volumes#

Now hold the run fixed and change the operating scale, because the verdict genuinely moves.

The cost column is the obvious one and the throughput column is the one that surprises teams. Five point four billion input tokens a day is above the standard per-organisation rate limits at every major provider, so at that volume the forty-step behaviour is not merely expensive, it makes the service unable to serve its traffic at peak, and the failure presents as 429s and queueing rather than as an invoice. Latency compounds it: fifty-six seconds per run at roughly 3.5 runs per second average and triple that at peak implies around 560 concurrent in-flight runs, each holding a growing context.

The run rate in that sentence is a working-day average rather than a calendar-day one, and it is worth saying so, because a reader who spreads a hundred thousand tasks across a full twenty-four hours gets a bit over one run a second, concludes the quoted rate is wrong, and stops trusting the concurrency figure built on it. Spread across a working day the arithmetic lands where the sentence says it does. That is also the reading the concurrency question needs, since in-flight count is set by the hours traffic actually arrives in rather than by the quiet ones — run duration multiplied by the arrival rate, tripled for peak, is what puts the number of simultaneous runs in the hundreds.

The last column is the one that does not move with volume at all. If the stray call had been calendar.delete_event rather than drive.write_file, no argument about daily volume or expected value is admissible, because the cost of a single occurrence is unbounded from the eval's point of view. Separating tools into read, write and destructive classes back in chapter 2 is what lets the gate treat these differently, and it is why unsafe_action_rate_max is a separate line from the budget block.

That column is worth deriving rather than asserting, because the derivation is the argument for a separate line. Every other row in the table is an expected cost, and an expected cost is a cost per occurrence multiplied by a rate of occurrence. The first two columns work as arguments precisely because their per-occurrence figure is a finite number of dollars, so a low enough rate makes the product small, and a small product is what makes a tracked defect a coherent verdict at the internal volume. Now take a per-occurrence cost with no upper bound — a deleted meeting whose consequences the eval cannot enumerate, let alone price. A quantity with no bound, multiplied by any rate above zero, has no bound either. There is no volume low enough to bring that product into an acceptable range, and a threshold expressed as a rate ceiling of zero is the only form of statement that says so.

Runtime guards, revisions and waivers#

A budget in the eval is not a spending cap in production. The gate decides whether a release ships; it does nothing at 03:00 when a customer's calendar configuration sends one run into a loop. That needs a runtime guard — a hard step limit, a per-run token ceiling, a circuit breaker on repeated identical tool calls — and the two numbers should be related rather than equal. The requirement is that production must never be the place a runaway run is first discovered. The choice made to satisfy it is to put the runtime ceiling above the eval's hard threshold, and that ordering does two separable things: a change producing long runs fails the gate before it ever reaches the runtime limit, and a run merely at the high end of normal is not killed in front of a user by a limit the eval itself would have passed.

Declaring a budget in advance also does not mean freezing it. Budgets are revised, and the discipline is that a revision is a recorded change with a reason and a date, reviewed like any other threshold change, rather than an adjustment made in the hour after a run failed. The test for whether a revision is honest is simple: if the new number would have been chosen without knowing which specific run it lets through, it is a budget change, and if it would not, it is a waiver, which is a legitimate thing to grant and should be written down under its own name.

The failure mode to watch for is erosion. A gate that blocks a release under time pressure gets a waiver, and waivers are fine individually and corrosive in aggregate, because after the fourth one nobody reads the gate output any more. Two habits keep it alive: waivers carry an expiry date and a named owner, so the count of open waivers is itself a number on the release report, and a gate that has never blocked anything in six months gets reviewed on the assumption that its thresholds are too loose rather than that the system is flawless. Flakiness erodes it faster than waivers do, which is why the aggregate checks are rates over the whole set rather than assertions about individual runs — a gate that fails one release in three for reasons nobody can reproduce will be routed around within a month.

A run that succeeds expensively has no verdict of its own. It has a step count, a cost, a latency and a list of the effects it had, and it acquires a verdict only when those are compared against a declaration that existed beforehand. Everything above is that declaration: a soft and a hard threshold on each number, so one line can fail an individual run while the other catches a drift no individual run would fail on; a list of allowed effects held outside the numeric block, because no volume makes an undeclared write cheap; an ordered set of checks, so every run gets one verdict and the rates are shares of one denominator; and a stated value per successful task, so moving a threshold is a calculation rather than a preference with an advocate.

Everything in this chapter assumes the numbers going into the gate mean what they claim. The pass rate comes from the assertions in chapter 1, where an end-state assertion is defined as a check on the world after the run rather than on the path the run took, the amber rate comes from the budgets here, and both are computed over the tasks somebody chose to put in the set. When that pass rate reads 92% and the support queue disagrees, the gate is working perfectly and measuring something other than what users experience.