Choosing the cap, and the stops a step count cannot express

An iteration cap is the largest number of model calls one run may make before the loop stops it and returns without an answer — the MAX_STEPS in for step in range(MAX_STEPS), or the max_turns a framework sets on your behalf. Every loop has one because none of its other exits is guaranteed to fire. The terminal turn (chapter 5, where the loop ends when the model returns a turn carrying no tool calls) is a state the model has to reach, and a model that keeps asking for tools never reaches it. The cap is the only exit that does not depend on the model's cooperation, which is both why it exists and why the number on it deserves an argument rather than a guess.

The dashboard says 4% of runs hit the iteration cap. The cap is 10. The product manager's suggestion is entirely reasonable on its face: raise it to 25, ship it this afternoon, revisit if it becomes a problem. Some tasks are just harder than others, and a limit that fires on one task in twenty-five is obviously too tight.

The suggestion is wrong, and the reason is not that 25 is too high. It is that the 4% is currently the only instrument the system has for a class of failure that has nothing to do with how many steps a task needs, and raising the number turns that instrument off while quadrupling what the failures cost. Before arguing that, though, it is worth being able to say where 10 came from — because in most teams the honest answer is that somebody typed a round number.

Where the number should come from#

The cap is derived from the step counts of runs that succeeded. Collect the steps field from a few thousand finished RunResults, filter to stop == "end_turn", and look at the distribution rather than the mean — at the percentiles, of which the one that matters here is the p95, the step count that 95% of successful runs finish at or under — because the distribution of agent step counts is long-tailed and the mean sits somewhere with almost no runs in it.

Read that table as gaps rather than as levels and it says something the levels on their own do not. From p50 to p95 the step count rises by two at every stop — three, five, seven, nine — so nearly half of all successful runs are packed into a band six steps wide. Then the gaps open. The p99 row sits five steps above p95, and the longest run in the sample sits seventeen steps above p99. That shape is what a long tail looks like when you write it down, and a cap is a horizontal line drawn across it: where you put the line decides how much of the tail you have declared broken.

The cap goes at p95 plus headroom — here, 11 or 12 — for a reason that is about what the number is for. Set it at p50 and you truncate half your successful work. Set it above p99 and it stops distinguishing hard tasks from broken ones, which is its only diagnostic value. At p95 plus two, you are asserting that a run needing more than twelve steps is more likely to be malfunctioning than to be unusually hard, and you have the p99 row to tell you how often that assertion will be wrong.

The percentile table prices that assertion without any further measurement, which is worth spelling out because it is the whole reason the cap is derived from a distribution rather than chosen. The p95 row says one successful run in twenty needs ten steps or more. The p99 row says one in a hundred needs fifteen or more. A cap of eleven truncates the runs that need twelve or more, and that population is smaller than the first of those fractions and larger than the second — so the table bounds the damage from both sides before anything ships. A cap of thirty-one would make the assertion vacuous instead, since every successful run in the sample already satisfies it and nothing can trip the line.

What eleven steps and twenty-five steps cost#

Then check the number against the two budgets it spends. At the request sizes from the trajectory in the previous chapter — around 1,420 input tokens on the first call, growing by roughly 700 per step as the history accumulates, with 350 output tokens per step — the arithmetic is:

Every cell in that table follows from the three premises above it, and doing the derivation once is worth more than trusting the cells, because the shape of the answer is the argument the rest of the chapter makes. A run capped at n steps makes n requests. The first carries about 1,420 input tokens. The second carries that plus roughly 700, because the assistant turn and the tool result from the first step are now part of the history and the whole history is resent. The i-th carries 1,420 plus 700 multiplied by i minus one. Adding all n of them gives 1,420 times n, plus 700 times the sum of every whole number from nought up to n minus one. That second sum is the triangular number of n minus one, and it is what makes the total grow with the square of the cap rather than in step with it.

Take the median row, where n is three, because the whole calculation is visible at that size. The three requests carry about 1,420, about two thousand one hundred and twenty, and about two thousand eight hundred and twenty input tokens. The whole numbers being summed are nought, one and two, which add to three, so the total is three times 1,420 — four thousand two hundred and sixty — plus 700 times three, which is two thousand one hundred. That comes to six thousand three hundred and sixty, against the 6,400 printed in the table. The table's input column is the derived figure rounded to the nearest hundred, which is why the twenty-five-step row needs no rounding at all: two hundred and forty-five thousand five hundred is already a multiple of a hundred. Run the same two multiplications for eleven and for twenty-five and the other two cells appear without any adjustment, which is what an algebraic form buys — it settles every cap rather than the one you happened to compute.

Those are Claude Opus 5 list prices, $5 per million input tokens and $25 per million output. Two things fall out. Cost grows quadratically in the cap, not linearly, because every step resends the whole history — the jump from 11 to 25 is 2.3× the steps and 3.9× the money. And 66 seconds is past the point where a chat interface with a 45-second request timeout has already given up, which means a cap of 25 would produce runs that cost $1.45 and are never seen by anyone. Prompt caching changes the first column substantially and not the second: with the growing prefix cached, the input side of an 11-step run falls from about $0.27 to under $0.05, while the wall time is unchanged.

With the prices in hand the money column reproduces on the same row. Three steps produce six thousand three hundred and sixty input tokens and, at 350 output tokens a step, one thousand and fifty output tokens. At five dollars a million the input side is a little over three cents; at twenty-five dollars a million the output side is a little over two and a half cents; together they come to just under six cents, which is the $0.06 in the table. The two ratios quoted above are then consequences of the algebra rather than observations about these particular figures. Twenty-five steps against eleven is a little over two and a quarter times as many, which is the 2.3× stated; the money carries the triangular term as well as the linear one, so it rises by a factor close to four. No choice of first-request size or growth rate would make the money ratio smaller than the step ratio, because the triangular term is larger than the linear one at any cap worth arguing about — which is what "quadratically" means when it is unpacked.

The wall-time column repays the same treatment, and it is the one place in the table where the printed figures look as though they disagree with the header. Twenty-five steps at the stated 2.6 seconds a step is sixty-five seconds, and the table says 66. Read the column as gaps rather than levels and the disagreement resolves. Between the three-step and eleven-step rows the time rises by twenty-one seconds over eight steps; between the eleven-step and twenty-five-step rows it rises by thirty-seven seconds over fourteen. Both differences put the underlying rate a little above two and six-tenths of a second, and at a rate in that band the three levels come out at just under eight seconds, a shade over twenty-nine, and sixty-six exactly — which is what the table prints. The header is that rate rounded to one decimal place, so the rounding survives in the levels and cancels in the differences. The 66 is right, and reproducing it from the header is what is wrong.

The $0.27 in the caching sentence is not a fourth measurement either; it is the input half of the $0.37 the derivation just produced, and splitting the total into its two halves is what makes the size of the saving predictable in advance. Of an eleven-step run's cost, a little over twenty-seven cents is input and a little under ten cents is output. Prompt caching — the provider charging a reduced rate for a prefix it has already processed — can act only on the first of those. The output cost is the residual, and it is untouched: however good the cache hit rate, an eleven-step run cannot fall below the ten cents it spends on tokens the model writes. That residual is also why the saving is worth having, since twenty-seven cents of thirty-seven is the majority of the bill.

When there is no distribution yet#

On day one there are no thousands of finished runs to take percentiles from, and picking p95 from a sample of eleven is arithmetic theatre. The order of operations that works is to start deliberately loose and tighten once the data exists: set the cap at something generous — 20 is a reasonable first guess for a tool-using agent with a handful of tools — put the wall-clock deadline in from the beginning — a budget in elapsed real seconds, which stops the run at a fixed time however few or many steps it has taken — at whatever your caller's timeout is minus a margin, and let the deadline rather than the step count be the thing that actually protects you for the first few weeks.

The deadline is the safer of the two to lean on early because its budget is one you already know. A chat interface that times out at 45 seconds gives you a 40-second deadline whether or not you understand the task's step distribution, whereas a step cap chosen without data is a guess about a quantity you have never measured.

Tightening also has to be done per task type once the workloads diverge, and the signal that they have diverged is a step distribution with two humps in it. A single cap over a bimodal distribution is wrong for both populations: too tight for the long one, useless as a detector for the short one.

The two-hump shape is worth being able to recognise, because it is the one case where no single number can be right. A histogram with a hump at three steps and another at thirteen is two workloads sharing one loop. A cap set for the short hump truncates the long workload wholesale. A cap set for the long hump sits so far above the short workload that one of its runs could spin for ten steps without tripping anything. There is no compromise position between them, because the quantity the cap is measuring is how far this run is from normal, and there are two different normals underneath it.

The 4% is a detector#

Raising the cap treats it as a constraint on how much work a task is allowed to do. Its more useful role is as a tripwire, and the way to find out which role it is playing today is to bucket the runs that hit it.

The reason bucketing settles the question rather than merely informing it is that a cap-hit rate is a rate over a mixture. The runs that hit the cap are not one population with one cause; they are however many populations happen to share the property of having run out of steps, and the rate reports the size of their union while saying nothing about its composition. Raising the cap removes exactly one kind of member from that union — the runs whose genuine step requirement lies between the old cap and the new one — and leaves every other kind exactly where it was. The rate afterwards is therefore the count of runs that were never about step count at all, divided by the same denominator as before. Two consequences follow and both are checkable on paper. The bucket sizes must add up to the original count, and a bucketing that leaves a remainder has missed a population. And the fall in the rate can be predicted from the bucket sizes before a line of code changes, which means a fall of the predicted size is evidence of nothing whatever.

Both checks from the general rule hold on those figures, and they are worth performing rather than assuming. The three buckets add to the 160, exactly and with nothing left over, which is what licenses treating the 64 as the whole of the population that raising the cap does not touch — had the sum come to less, some fourth kind of run would be hiding in the difference and the prediction below would be wrong. And 64 runs against the same 4,000 is 1.6%, so the fall the dashboard would show is arithmetic performed on an unchanged set of failures. Anyone can compute it in advance of the change, which is precisely what makes it worthless as evidence that the change helped.

Exhaustion is a result, not an absence#

When the cap does fire, the loop must return something a caller can act on. The failure mode to avoid is a loop that returns None, logs a warning, and drops the trajectory — which is what makes the 4% bucketing exercise above impossible in the first place.

return RunResult(
    stop="max_steps",
    output=None,
    partial=last_assistant_text(messages),   # whatever it had said before the cut
    messages=messages,                       # the full trajectory, for diagnosis
    steps=MAX_STEPS,
    usage=usage,
)

The caller then has real choices: escalate to a human queue with the transcript attached, resume the run with a higher cap because the task type is known to be long, or record it in the dataset that the next cap review reads. The partial field matters more than it looks — in the multi-invoice population, the model has usually already refunded four of six invoices by the time the cap fires, and a caller that knows this can finish the job rather than starting over and refunding the first four again.

The stop a step count cannot express#

A cap counts model calls. It cannot see anything that happens between them, which means it is blind to the single most common way an agent run goes wrong in production: a tool that never returns. An HTTP client with no timeout, a database query behind a lock, a queue read that blocks — any of these parks the loop inside dispatch_one at step 3 of 11, with 8 steps of headroom remaining and no mechanism that will ever consume them. The run is not stopped by the cap. It is not stopped by anything.

The belief that makes that expensive is not that the cap is useless; it is that a cap is a bound, and bounds are supposed to bound, so a run with eight steps left over must have something still holding it. The tempting part is that the reassuring quantity really is constant: eight of eleven steps remain, and they remain no matter how long the run sits there. The dangerous quantity is the one varying underneath, which is elapsed real time — and a cap cannot observe elapsed time at all. It is a counter, it advances only when a model call returns, and a tool blocked in a socket read produces no model call for it to count. A bound on a quantity that has stopped changing is not a bound on anything.

Every production loop therefore needs a wall-clock deadline beside the step cap, and the deadline needs teeth in two places.

def run(user_input: str, max_steps: int = 11, budget_s: float = 45.0) -> RunResult:
    deadline = time.monotonic() + budget_s
    messages = [{"role": "user", "content": user_input}]

    for step in range(max_steps):
        if time.monotonic() >= deadline:
            return RunResult(stop="deadline", output=None,
                             partial=last_assistant_text(messages),
                             messages=messages, steps=step, usage=usage)

        response = client.messages.create(..., messages=messages)
        ...
        remaining = deadline - time.monotonic()
        results = run_concurrently(calls, dispatch, timeout=min(5.0, remaining))

Checking the deadline before each model call is the cheap half and stops a run that is merely slow. Passing the remaining budget down as a per-call timeout is the half that actually addresses hung tools, because a check at the top of the loop is never reached by a thread blocked in a socket read. Every tool needs a timeout, and the loop needs to be the thing that enforces it rather than trusting each tool's own client configuration.

A third bound is worth adding once you have the first two: a cumulative token budget, checked against usage after each step. It catches the case that neither of the others does — a run that stays under both the step cap and the deadline while one tool returns 40,000 tokens of JSON per call, which is fast, legal, and ruinous. Providers are also starting to expose an advisory version of this to the model itself, where a declared task budget lets the model pace its own work and wrap up as it approaches the limit; that is a complement to your hard bound rather than a replacement, since an advisory the model can ignore is not a bound.

That third case is the same shape a third time, and recognising the shape is cheaper than meeting each instance separately. Two comfortable constant quantities — steps remaining and seconds remaining — sit above a third that nobody is watching. The general rule underneath all three is that a bound protects the resource it counts and no other, so the question to ask when you add one is which resource can still be exhausted while the new bound reads healthy.

Authorisation, correctness and termination are elsewhere#

Three misreadings are common enough to be worth ruling out. The cap is not a safety mechanism: a run that issues an unauthorised refund does so on step 2, well inside any limit, and the thing that prevents it is authorisation inside issue_refund. It is not a correctness bound: nothing about finishing in nine steps makes the answer right, and a low cap mostly converts wrong answers into missing ones. And it is not a substitute for the exit condition designed two chapters ago — a run that ends by exhausting its cap has, by definition, failed to reach a state the model considered terminal, and a system where that is the usual way runs end is a system whose exit design is not working.

Those three are one reasoning error wearing three costumes, which is why they arrive together and why the same test disposes of all of them. It is the test the hung tool already established, reused: name the quantity the bound counts, then ask whether the harm you are worried about is a function of that quantity. An unauthorised refund is a function of what issue_refund permits its caller to do. A wrong answer is a function of what the tools returned and what the model made of it. A run that never reaches a terminal state is a function of the exit condition (chapter 5, where the loop reads stop_reason rather than treating the absence of tool calls as an answer). None of the three is a function of how many model calls were made, so no setting of the cap moves any of them.

The number therefore has two separate justifications and they should be argued separately. Its size comes from the step counts of runs that already worked, taken at the p95 and given headroom, and then checked against what a run of that length costs in money and in seconds — all of which is arithmetic on a distribution you already have, done before anything ships. Its value comes from what it catches, which is why a cap-hit rate is read as a mixture rather than minimised. Raising the number until the dashboard is quiet spends the second justification to buy nothing on the first, and the trade is hard to see because the dashboard improves either way. Beside the cap goes a wall-clock deadline, because the two bounds count different quantities and only one of them advances while a tool is hung.

The remaining question is the one the 4% bucketing raised and did not answer. A trace showing the same call over and over, a trace showing malformed arguments, a trace full of 503s, and a trace where the model simply never stops all look alike from the outside: a run that hit the cap. They are four different bugs with four different fixes, and the only thing that tells them apart is the trajectory.