Exhausting a budget mid-task leaves state somebody has to own

Several of the mechanisms in this topic end a run early. A budget fires, a repeated action is detected, a wait times out; in each case the loop stops calling the model and returns. Stopping is the part that is easy to write and easy to test, and it is half of the work. The other half is that the run was in the middle of something when it stopped, and "in the middle of something" is not a figure of speech for an agent whose tools move money.

Two terms need fixing before anything else, because the whole argument is about the relationship between them. An external effect is anything a tool did that outlives the run: a refund posted to a payment provider, an email delivered to a customer, a row written into a system somebody else reads. A terminal result is the object the loop returns when it ends — the interface between a run and whoever asked for it. External effects are permanent and, from outside the process, invisible. The terminal result is the only thing that can make them visible. When it does not, the next component to touch that task proceeds as though they never happened, and on a billing agent that means a customer is refunded twice and told twice.

A stop ends the model's turn and reaches nothing downstream#

It is worth walking the reason a stop cannot undo anything, because the phrase "the budget stopped the run" invites the belief that it stopped the run's consequences too. The loop's entire influence over the world is the tool calls it dispatches. A tool call that has already returned has already had whatever effect it was going to have, out in a payment provider or a mail server that has no idea an agent loop exists. Ending the loop removes the agent's ability to make further calls, and it has no reach at all over calls already made. So at the instant a budget fires, the set of effects that landed is fixed, and it is known to exactly one component — the run that made them, which is the component about to exit.

That is the causal step, and everything else follows from it. If the exiting run does not write the set down, the set is unrecoverable from anywhere else at reasonable cost: a payment provider's records can be reconciled against, eventually, by somebody who already suspects there is something to look for. The terminal result is the only cheap moment at which that knowledge exists in one place.

Now add the caller. A queue consumer configured for at-least-once delivery — the setting that would rather hand the same message to a worker twice than risk dropping it — has exactly two responses to a worker that reports failure. It can drop the message, which loses the task. Or it can redeliver it, which repeats the task. It cannot choose between those two intelligently unless the failure report tells it which effects already landed, and a report of the shape below tells it nothing at all.

{"error": "budget_exhausted", "budget": "money"}

A consumer holding that dictionary knows one thing: the work did not finish. Redelivering is the correct response to an unfinished task, so it redelivers. The second run reads the same input, is the same agent under the same prompt, and therefore plans much the same path — which means it re-issues every effect the first run issued before it stopped. Duplication here is not bad luck. It is the specified behaviour of that pair of components, and it will reproduce every time the pair is assembled.

What a terminal result has to carry#

The object a run returns is an interface, and the callers of it are a queue consumer deciding whether to redeliver, an on-call engineer reading a trace at 3am, and quite often another agent that dispatched this one as a subtask. None of those three can do anything useful with a two-key error dictionary. The contract that serves all of them looks more like this:

{
  "run_id": "9f2a1c",
  "status": "stopped",
  "stop_reason": "budget_exhausted:money",
  "resumable": true,
  "budgets": {
    "steps":  {"used": 9,      "limit": 12},
    "tokens": {"used": 48310,  "limit": 60000},
    "seconds":{"used": 31.4,   "limit": 90},
    "cost_usd":{"used": 0.251, "limit": 0.25}
  },
  "partial_output": "Refund of $24.00 issued for order 4471 and confirmation sent. Internal ticket note not written.",
  "plan": {
    "done": ["verify_charge", "issue_refund", "notify_customer"],
    "remaining": ["write_ticket_note", "close_ticket"]
  },
  "effects": [
    {"tool": "issue_refund", "key": "9f2a1c:5", "state": "confirmed",
     "detail": {"order_id": "4471", "amount_cents": 2400,
                "refund_id": "re_1Qk8"}},
    {"tool": "send_email", "key": "9f2a1c:7", "state": "confirmed",
     "detail": {"to": "c***@example.com", "template": "refund_sent"}}
  ],
  "messages_ref": "s3://agent-traces/2026/03/9f2a1c.json"
}

Four things in there are load-bearing in a way the original two-key version was not. stop_reason names which budget fired, so the consumer can distinguish a run that was too expensive from a run that was stuck, and those two want opposite handling. plan says what remains, which is what makes continuation cheaper than restarting. effects is the record whose absence cost $1,486. And resumable is a decision the run makes about itself rather than a guess the caller has to make, because some stop reasons are safe to continue from and some are a bug that will reproduce.

The budgets block deserves a second look, because it does something the other fields cannot. It reports all four limits — the same four the earlier chapter set at twelve steps, sixty thousand tokens, ninety seconds and a quarter of a dollar — with the used value beside the limit in each case. Read down the four pairs and exactly one of them is over its line: steps and tokens and seconds all sit below, and only the cost figure has crossed. That means stop_reason is not a claim the caller has to take on trust. The caller can recompute which budget fired from the numbers printed next to it, by a route that shares no step with whatever code set the field, and get the same answer. What that licenses is narrow and useful: this object certifies which limit was exceeded without the reader trusting the run's own account of itself, and it certifies nothing whatsoever about whether exceeding that limit was the right thing to do.

The reverse case is the one worth holding on to. If a terminal result ever arrives with stop_reason naming money and no budget over its line, or with two budgets over at once, the field and the block disagree, and a disagreement between two independently produced descriptions of the same event is a bug report that costs nothing to generate. A single-field stop reason has no counterpart to disagree with, which is why nothing ever catches it being wrong.

The rows are more useful read as answers to one question than as a list of policies. The question is whether running the same task again, with the same inputs, would produce a different outcome, and each row is a different answer to it. A budget ran out, so the same work with more allowance finishes: yes. A deadline passed, so it depends on whether the deadline still means anything: sometimes. A dependency was down, so it depends on the clock rather than on the task: yes, later. And a loop was detected, which is the row where the answer is provably no — the detector fired precisely because the run had established that its own repetition changed nothing, so the same inputs will produce the same plan, the same repetition and the same bill. The gap between that row and the ones above it is the whole content of the table.

Auto-resume on loop_detected is the mistake this table exists to prevent, and it is easy to arrive at by treating every non-success as retryable, which is precisely what the queue consumer in the incident did. A loop is deterministic in the way that matters: same inputs, same plan, same repetition, three times the cost.

The postmortem needs a budget of its own#

There is a circularity in producing that partial_output. Summarising what the run accomplished is a model call, and the run has just established that it has no money left. A cap enforced with no slack means either the wrap-up cannot happen or the cap is not really a cap.

The way out is to reserve a slice up front. On the email agent that means the enforcement limits sit at 92% of the stated budget — the check fires at $0.23 of a $0.25 cap, at 55,000 tokens of 60,000, and at 80 seconds of 90 — and the remainder funds exactly one further model call with tools=[] so it can describe but not act. That call takes the message list and asks for two things: a plain-language account of what was completed, and the remaining plan items. It costs about $0.008 and 1.4 seconds on this agent, which is well inside the reserve, and it is the difference between a caller that knows a refund went out and a caller that knows a number was exceeded.

That percentage is a floor rather than an exact division, and the three enforcement limits show why. The money figure lands on the stated share exactly. The token figure and the seconds figure are each rounded down to a round number, which puts them a little below the share and therefore leaves a little more in reserve than the rule strictly asks for. Rounding in that direction is the safe one: a reserve that is slightly too large costs a fraction of the allowance, and a reserve that is slightly too small costs the wrap-up call, which is the one call whose failure the whole arrangement exists to prevent.

The reserve also has to cover the case where the wrap-up call itself fails, so the terminal result is assembled from structured state first and enriched by the model second. If the summarisation call errors or times out, partial_output falls back to a rendering of the plan and the effects ledger, which is less readable and contains every fact that matters.

Recording effects before they happen#

The ledger is the piece that prevents the duplicate refund, and its one non-obvious property is that the entry is written before the call, not after.

def dispatch_with_ledger(call, run_id: str, step: int, ledger: Ledger) -> str:
    key = f"{run_id}:{step}"
    if (prior := ledger.get(key)) and prior.state == "confirmed":
        return prior.result                      # replay: do not call again

    if MUTATING.get(call.name):
        ledger.put(key, state="intended", tool=call.name, args=call.input)

    try:
        result = call_tool(call.name, call.input, idempotency_key=key)
    except ToolFailure as failure:
        if MUTATING.get(call.name):
            ledger.put(key, state="unknown" if failure.retryable else "failed",
                       error=failure.message)
        raise

    if MUTATING.get(call.name):
        ledger.put(key, state="confirmed", result=result)
    return result

The intended write exists for the window between deciding to refund and knowing whether the refund happened, which is where the process can be killed by an OOM — the kernel terminating it for running out of memory — a deploy, or a spot instance reclaim. A ledger that only records completed effects tells a resuming run nothing about that window, and the resuming run will make the call again. A ledger that records intent first turns the ambiguity into something a human or a reconciliation job can act on: there is a row saying a refund for order 4471 was about to be issued, and someone can go and look.

The reason a completed-effects ledger fails there is worth stating as a property rather than as a story, because it is the sharpest thing in the chapter. Ask such a ledger whether a given refund happened. It answers by looking for a row, and a row is written only after a response comes back. So it returns "no row" for a call that was never dispatched, and it returns "no row" for a call that was dispatched, executed at the provider, and killed before the response landed. Those two situations differ in the only way anybody cares about — one owes the customer money and the other does not — and the ledger has no term in it that varies between them. Its answer is constant across the distinction. What a completed-effects ledger licenses is therefore exactly one claim: that an effect finished. It never licenses the claim that an effect did not start, and a resume is built entirely on the second claim.

Writing the intent first adds the missing term. A row in intended and no confirmation means the call was dispatched and the answer is unknown, which is a third outcome and reads as one.

state has three values because there are genuinely three outcomes, and collapsing them loses the only distinction that matters. Topic 2.11 established what the idempotency key does about the middle one, and the key here is derived from the run and step so that a resume reproduces it exactly; the ledger's job is the part a key cannot do, which is to survive the process and be readable by whoever comes next.

That diagram traces one effect from intent to replay, which is what makes the mechanism visible; the same three writes happen for the email at step 7, and a resume walks both rows before it walks anything new.

The keying decision in that diagram is worth stating explicitly because it is where teams get it wrong under pressure. Keying on run and step makes replay work only when the resume reproduces the same plan, which it does when the resume starts from the same message list, and does not when the resume is a fresh run from the original queue message. For that second case the key has to come from the task rather than the run — the ticket ID and the effect type, ticket_88231:refund — so that a completely independent second run still collides with the first attempt's row. The email agent uses both: the run-step key for the provider's idempotency header, and a task-scoped key as a unique index on the ledger table, so a duplicate refund for the same ticket is rejected by the database whatever the agent believes.

One key cannot cover both cases, and the reason is a property of what each key is made of rather than a matter of preference. A run-step key is built from a run identifier and a position in that run, so two attempts collide only when both parts match — which requires the second attempt to be the same run, resumed, with its plan reproduced step for step. A fresh delivery from the queue is a new run identifier by construction, so its keys cannot collide with the first attempt's however identical the work is. A task-scoped key is built from something both attempts share, the ticket and the kind of effect, so it collides across independent runs. It also cannot serve as the provider's idempotency header on its own, because it does not distinguish a legitimate second refund on the same ticket from a duplicate of the first. Each key answers the question the other cannot, which is why a system that needs both answers carries both.

A resume also has to tell the model what already happened, which is a separate mechanism from skipping the calls. Replaying the stored message list is the cleanest option when the trace is available, since the model then sees the refund it issued and plans from there. When the trace is gone or too large to resend — the email agent stores its messages in object storage and a 48,000-token replay would consume most of a fresh budget — the resume starts from the original task plus a rendered summary of the ledger: "Already completed in an earlier attempt: refund re_1Qk8 of $24.00 for order 4471; confirmation email sent. Do not repeat these." That block costs about 60 tokens and it is what stops the model from planning the refund again and then being silently blocked by the ledger, which produces a run that spends steps discovering it has nothing to do.

The size comparison in that sentence is not incidental. The stored trace and the fresh allowance are the same order of magnitude, so replaying the messages consumes most of what the resumed run has to work with before it does anything; the rendered summary is smaller by three orders of magnitude and carries the facts a resume actually needs. That is the trade being made, and it is a trade rather than a free win — the summary drops the model's own reasoning about why it refunded, which is exactly the material a resume would need if the remaining work depended on it.

For the stop reasons that are not resumable, safety means something different and needs somewhere to go. A loop_detected result should land in a human queue carrying the same terminal object — the effects that did happen, the plan that did not, the trace reference — because the person picking it up needs to know whether a refund went out before deciding what to tell the customer. The email agent routes those to the same support queue the drafts go to, with the summary as the first comment, and the measure that made it worth doing is that median time from a stopped run to a resolved ticket fell from just over two days, when stopped runs were only visible in logs, to about four hours.

Durable execution, and the window nothing closes#

None of this is durable execution. A workflow engine that checkpoints every step and replays deterministically after a crash — the model Samuel Colvin discusses in Durable Agents: How to Build AI Systems That Survive a Crash — solves a strictly larger problem, and if you are already running one, the ledger described here is a weaker version of something it gives you. The reason to build the small version anyway is that it fits in a queue consumer you already have, and the terminal result contract is required whether or not the runtime underneath is durable, because the contract is about what the caller learns rather than about how the run is stored.

It is also not a guarantee. There is an irreducible window between the request leaving your process and the ledger recording the response, and a machine that dies inside that window leaves an intended row and no answer. Provider-side idempotency closes it for providers that offer it, reconciliation against the provider's own records closes it for those that do not, and neither is something the agent loop can do by itself. What the ledger buys is that the ambiguity is small, bounded, recorded, and attached to a specific order ID, rather than a discovery finance makes nine days later from a spreadsheet.

That residual deserves naming rather than burying, because it is what the whole design reduces to and not what it removes. Before the ledger, the set of possibly-duplicated effects was every effect any stopped run had ever made, and finding it meant reconciling a month of provider records against a month of traces. After the ledger, the set is the rows sitting in intended with no confirmation, which is normally a handful, each one naming an order. The design converts an unbounded, undirected search into a short worklist. It does not convert an uncertain outcome into a certain one, and a review that reads it as doing so will be surprised the first time an intended row turns out to have gone through.

The last thing to say about the incident is that everything in this chapter existed as a plan before it happened. The budgets were configured, the stop was clean, and the team had discussed a ledger. What did not exist was a test that ran the agent into its money cap mid-task and asserted anything at all about what came out, which is why the two-key error dictionary survived code review and shipped.

That absence is a specific kind, and it is worth being precise about it rather than filing it under insufficient testing. The suite tested that the cap fired, and the cap did fire, on the number it was set to, every time. What nothing tested was the object produced on the way out, so the two-key dictionary was never wrong in any test's opinion — no assertion was computed from it. A control can be exercised thoroughly and its output never inspected, and the incident that follows will look like a failure of the control that worked perfectly. The next chapter takes that observation and turns it on the whole topic, because the same gap that let a two-key dictionary ship also lets a fleet report a month with no failures in it.