Per-agent caps bound the hop, never the number of hops

The line item that started the argument was $38.58 against a single conversation, run_b41f07, which ran for nine minutes and fourteen seconds before the customer closed the tab. No alert fired, no error was logged, and every per-agent dashboard was green for the entire duration, because every agent was behaving exactly as configured.

$ trace summary run_b41f07

transfers            41          wall clock        9m14s
model calls         306          tokens in         6.92M
                                 tokens out        159k
cost                $38.58       (in $5/M, out $25/M)

per-turn caps:      10 tool calls, 25k input tokens
max observed:        9 tool calls, 24.1k input tokens   ← no cap breached
mean per turn:      7.5 tool calls, 22.6k input tokens

top edges:  refunds→triage 9x   triage→refunds 8x   triage→billing 7x
            billing→triage 7x   refunds→notifications 4x
state_after == state_before on 29 of 41 hops

Every number in the top half is inside its limit and every number in the bottom half is a description of a loop. Control circulated between triage and two specialists, twenty-nine of the forty-one hops established nothing new, and the run had no property that could have stopped it, because the only limits in the system were properties of a turn and a turn is not the thing that ran away. Without the run-scoped trace from the previous chapter this would still be unexplained: a per-agent log has no record of a transfer at all, so the same run appears in the logs as forty-one unremarkable turns spread across three agents, none of which looks wrong.

The arithmetic of a cap#

The cost of a run is the number of hops multiplied by the cost of a hop, and a per-turn cap is a bound on the second factor only. At 22.6k input tokens and 7.5 tool calls, one hop of run_b41f07 cost about $0.94, and forty-one of them cost $38.58; the cap did its job perfectly on each of the forty-one.

The proposal that follows an incident like this is to tighten the caps — four tool calls and eight thousand input tokens, on the reasoning that a $38 run becomes a $10 run. Replaying the same conversation under those caps is worth doing before agreeing, because the outcome is not the one the proposal predicts.

The bill goes down and nothing has been fixed. The run still does not end; it ends when the customer leaves, and under the tighter caps the customer would have left seven minutes earlier having received less, because each agent can now do less per turn and therefore transfers sooner. The 78 hops are the same loop taking more laps. Meanwhile the change breaks the legitimate long turns: 6.3% of production turns used more than four tool calls, and those are the conversations where an agent looks up an order, checks two suppliers, reads a policy and computes a partial refund. Capping at four turns those into wrong answers, complaints follow, the cap gets reverted within a fortnight, and the actual defect — a run with no budget of its own — is still there.

Budgets that belong to the run#

The fix is a small object that is created once, travels in the payload, and is decremented at the transfer boundary, which is the same wrapper that already checks the declared graph and emits the routing event. Sizing it comes from the distribution of the runs that work rather than from a round number: on resolved conversations, hops were 4 at p50 and 9 at p99, so 14 leaves headroom for the multi-issue tail while cutting run_b41f07 off at hop 14 for about $13 and three minutes.

@dataclass
class RunBudget:
    hops_left: int          # 14
    tokens_left: int        # 400_000
    spend_left_usd: float   # 4.00
    deadline_ms: int        # start + 180_000

    def charge(self, tokens: int, cost: float) -> None:
        self.hops_left -= 1
        self.tokens_left -= tokens
        self.spend_left_usd -= cost

    def exhausted(self) -> str | None:
        if self.hops_left <= 0:        return "hop budget"
        if self.tokens_left <= 0:      return "token budget"
        if self.spend_left_usd <= 0:   return "spend budget"
        if now_ms() > self.deadline_ms: return "wall-clock deadline"
        return None

The part that determines whether this helps or merely fails differently is what happens when exhausted() returns a string. Raising an exception produces a dropped conversation, a stack trace, and a customer with a refund half-issued, which is a worse outcome than the $38 run rather than a better one. Exhaustion has to route, and the destination is the terminal transfer from chapter 3.

def apply_transfer(state, to_agent, reason, payload):
    state.budget.charge(state.turn_tokens_total, state.turn_cost)
    payload.budget = state.budget
    if (why := state.budget.exhausted()):
        trace.emit(RoutingEvent(..., to_agent="resolve", reason=f"exhausted: {why}"))
        return force_resolve(state, cause=why)     # terminal, with gaps named
    if to_agent not in ALLOWED[state.current_agent]:
        return ToolError(...)
    trace.emit(RoutingEvent(...))
    return AGENTS_BY_NAME[to_agent]

Different budgets have to be enforced in different places, because they are exceeded in different ways, and a single check at the transfer boundary misses the run that burns its money inside one very long turn.

The deadline appears twice on purpose. Enforced only in the driver, which is the outer loop from chapter 1 that keeps invoking whichever agent currently holds control, it stops the run at 180 s regardless of what an agent has just started; carried in the payload as well, it lets a receiving agent see that it has eleven seconds left and decline to begin a supplier lookup that takes forty, resolving with what it has instead.

Where to put the ceiling, and what it costs to put it there#

Fourteen hops was chosen from a distribution, and the choice is worth showing as arithmetic because every ceiling cuts some run that would have succeeded. Replaying six weeks of traffic against candidate ceilings gives the shape of that trade directly.

Going from 14 down to 11 saves another $24 a day and triples the number of runs that would have finished but did not, each of which becomes a customer waiting on a person. At a fully loaded $4 per handled ticket, meaning the agent's time plus the share of everything behind it, those 19 extra tickets cost $76, so the tighter ceiling is worse on the same ledger. Going up to 20 gives back $46 a day for six fewer truncations. The flat region between 11 and 20 is the useful thing in this table, because it says the exact number does not matter much and the existence of a number matters a great deal, which is also why arguing about it in review is usually a poor use of an hour.

Telling the run it is running low#

A hard ceiling produces a run that is working normally at hop 13 and terminated at hop 14, and the agent holding it had no way to see that coming. Exposing the remaining budget in the prompt, which the payload from chapter 2 already carries, lets the run change its own behaviour before it hits the wall. The rule that worked is a soft threshold at 70% of any single budget: past that point the receiver's system message gains a line instructing it to prefer finishing over transferring, to skip optional verification, and to call resolve with named gaps rather than hand on a task it cannot see the end of.

There is no edge back from converging to open, because budgets only ever decrease and a run that re-entered the relaxed state would oscillate at the boundary. The measured effect was on the tail rather than the median: p99 hops fell from 24 to 15, hard exhaustions fell from 1.9% of runs to 0.7%, and the overall rate of runs ending partial stayed near 4%, which is the outcome to want. The same conversations end incomplete; they end incomplete earlier, more cheaply, and with the agent's own account of what is missing rather than with whatever it happened to be saying when the counter reached zero.

One detail in the transfer wrapper above is load-bearing and easy to drop in review. The line payload.budget = state.budget overwrites whatever the sending agent wrote into the budget field, and it has to, because the payload is generated by a model and the model has been reading budget numbers in its prompt all run. In the first week, three runs carried a hops_left the sender had copied from an earlier hop's prompt rather than the current value, and one carried a number larger than it had ever been given. A budget that a model can write is not a budget; the authoritative copy lives in the run state, the payload carries a rendering of it, and the direction of that copy is one way at every transfer.

Duplicate work is a budget too#

The row at the bottom of that table is the one that has no token cost and the highest real cost. Two agents in the same run contacting the same supplier is not caught by hops, tokens, spend or time, because both hops are cheap and legal, and the effect is an external side effect that cannot be rolled back. The bound is on identity rather than on quantity: name the effect, record it against the run, and refuse the second one.

def claim(run_id: str, identity: tuple[str, ...]) -> bool:
    """('supplier_email', 'SUP-118', 'stock_query') — first caller wins."""
    return redis.set(f"claim:{run_id}:{'|'.join(identity)}", "1",
                     nx=True, ex=3600) is not None

def email_supplier(ctx, supplier_id: str, topic: str, body: str):
    if not claim(ctx.run_id, ("supplier_email", supplier_id, topic)):
        return ToolError(f"Already contacted {supplier_id} about {topic} in this run. "
                         f"The result is in the handoff payload under established facts.")
    return smtp.send(...)

The error text is doing work there. An agent told only "denied" will retry or transfer; an agent told where the existing answer lives can use it, which is the difference between the guard preventing a duplicate email and the guard preventing a duplicate email without stalling the run.

Running out is a normal outcome, not an error#

A run that exhausts its budget has reached a state the design should have an answer for, and so has the run from chapter 3 where the agent holding control has a partial answer, low confidence, and no destination that looks right. Those two states are the same state arrived at from different directions, and both are handled by the payload carrying the termination conditions and the remaining budget, so that the agent currently holding the conversation can evaluate its position rather than infer it. With stop_conditions and budget rendered into its prompt, "none of the stop conditions are met, one hop remains, and the outstanding item needs a supplier reply that will take a day" is a comparison, and its correct outcome is a terminal call with the shortfall stated.

{"status": "partial", "cause": "hop budget",
 "delivered": ["Refund of £68.40 issued to the original card, ref RF-90412."],
 "gaps": ["Duplicate shipping surcharge on INV-8871 not resolved. Billing agent
           has the invoice id; no charge reversal was attempted."],
 "next_owner": "human_queue:billing"}

That object is what makes the third recurring incident — a confident partial answer nobody noticed was partial — a thing the system can detect, since status is a field rather than a tone of voice. Runs ending partial are routed to a human queue and counted; the count on this workload settled at 4.1% of conversations, which is a number to work on rather than a surprise.

None of this makes the routing better. A bounded swarm still sends conversations to the wrong agent at the rate its handoff descriptions permit, still loses constraints at the rate the payload composition permits, and now stops circulating at hop 14 with a partial answer instead of at minute nine with an abandoned customer. What budgets buy is that the worst case is known in advance and priced: $4.00 and 180 seconds, every time, whatever goes wrong.

That is enough machinery — contract, graph, trace, budget — for the swarm to be defensible in production. Whether it should be the design at all is a separate question, and it is one a director is entitled to ask with numbers on the table.