Tool timeouts and cancellation

A timeout on a tool call is the point at which your code stops waiting for a dependency and tells the agent something went wrong. Three questions decide whether it helps: where the number comes from, what actually stops when it fires, and what the agent is supposed to believe afterwards. Most implementations answer only the first, with the rule that fits a synchronous request path rather than an agent task.

The distinction underneath all three is between a bound on your waiting and a bound on the work. Where those two bounds diverge, a timeout that looks like a safety control is a latency control with a safety story attached.

The number belongs to the task, not to the dependency#

Take a real distribution to argue against. An internal reporting service has a p50 of 400ms, a p90 of 1.8 seconds, a p95 of 3.5 seconds and a p99 of 9 seconds, those percentiles being the latencies that 50%, 90%, 95% and 99% of calls come in under, which is slow but survivable for a service that aggregates across a fact table with 2.4 billion rows. What is not survivable is the tail past that: when a query misses the result cache and the underlying partitions have been evicted, the service can take up to 90 seconds, and roughly one call in five hundred does.

Set the timeout for an agent's fetch_report tool at 12 seconds, on the standard reasoning that p99 plus a bit of headroom is where a timeout belongs. The agent makes up to six of these calls in a single task, and the product commitment for the whole task is 30 seconds. That arithmetic does not close: six calls at 12 seconds is 72 seconds of tool time alone, against a budget of 30 for everything including the model's own generation latency, so the design has already failed at the specification stage.

Percentiles of a dependency tell you what will happen if you wait. They do not tell you how long you are entitled to wait, and that is the question a timeout answers. The entitlement comes from the end-to-end target, divided across everything the task must do inside it.

For a 30-second product target, the division looks roughly like this. Reserve two seconds for the gateway, request parsing and the final response write. The model itself, across the turns needed to plan, call six tools and compose an answer, is the largest single consumer: at four assistant turns averaging 1.6 seconds of time to first token, the delay before the model emits its first word, plus generation, that is around 8 seconds. Leave 2 seconds of slack so the task does not fail at exactly the boundary. What remains for tool execution is about 18 seconds, and that 18 seconds is the real budget being divided.

That division is worth checking rather than reading, because it is a subtraction and it closes. Two seconds of gateway, about eight seconds of model, two seconds of slack and eighteen seconds of tools add up to the thirty the product committed to, with nothing left over and no line that has not been named. The model's own share is the one worth expanding, since it is the largest and the least visible: four assistant turns, each waiting a bit over a second and a half before the first token appears, is already more than six seconds of pure waiting, and the words themselves carry it to around eight. Two properties follow from the sum closing exactly. The tool budget is a residual rather than a choice, so anything added to another row is taken out of this one — a fifth assistant turn costs the tools a second and a half before anybody has written a line of code. And there is no headroom hiding anywhere, which means a per-call number chosen without reference to this table cannot be checked against anything, and will not be.

Dividing it evenly gives 3 seconds per call, and 3 seconds is below the service's p95. Against the distribution above, a 3-second cap times out something like 6% of calls, and with six calls in a task the probability that at least one of them times out is 1 − 0.94⁶, or about 31%. A third of tasks hitting a tool failure is not a safety improvement; it is an availability regression with a safety justification attached.

The step from six per cent to a third is where the intuition usually fails, so it is worth writing out. The quantity that multiplies is not the failure rate but its complement: a call survives with probability ninety-four per cent, and the task survives only if every one of its six calls does, so the survival probabilities multiply while the failure rates do not. Six factors of ninety-four per cent come to a bit over sixty-nine per cent, and the task failure rate is what is left when that is taken from one. The reason this is counter-intuitive is that a small per-call number invites addition, and adding gives thirty-six per cent, which happens to be close enough to the right answer that nobody notices the method is wrong until the call count grows. Two conditions are being assumed and should be said aloud. The calls are treated as independent, which they are not — six calls in one task hit the same cache in the same state, so real tasks cluster into all-fast and all-slow more than the multiplication predicts, which makes the true figure somewhat lower. And the six is itself a model's choice at runtime rather than a constant, so the exponent moves with the task.

The p99-plus-headroom rule is not folklore, and it is worth understanding why it is usually right and wrong here. It comes from the ordinary synchronous request path, where one service makes one call to one dependency and the caller's own service level objective (SLO), the latency target it has committed to meeting for some stated share of requests, is set with that call's distribution in mind. In that setting the timeout exists to catch the pathological tail without failing calls that were merely slow, so anchoring it above p99 is exactly correct. An agent task breaks the assumption the rule was built on, because the number of calls is decided at runtime by the model rather than fixed by the code, and a per-call rule that is correct in isolation composes into a task-level budget nobody chose. Whenever the call count is variable, the timeout has to be derived downward from the total rather than upward from the dependency.

The way out is that the calls do not each need their own private share. Tool latency is heavily skewed, so the typical task spends 6 × 400ms ≈ 2.4 seconds on tools and leaves nearly all of the 18-second budget unused. A deadline held at the task level, with each call capped at whatever remains, spends that slack where it is needed instead of pre-allocating it evenly.

The third row is the design worth building. A task deadline of 18 seconds of tool time is established when the run starts, every call is capped at the smaller of 9 seconds and the remaining budget, and a call that would start with less than a second left is refused before it is issued rather than started and abandoned. The 9-second cap is not the dependency's p99 chosen for its own sake; it is the largest single call the budget can absorb while leaving room for the calls that usually follow it.

Every row of that table is the same two computations with a different per-call cap, which is what makes it checkable rather than a set of quoted results. The worst-case column is the cap multiplied by the call count, and the third row's figure is the deadline itself rather than a product, because the deadline is what caps the sum directly. The last column is the calculation of the paragraph above, run on that row's own timeout rate: raise the survival rate to the sixth power, subtract from one. Do it on the top row and a per-call rate of about seven in a thousand comes out at roughly four tasks in a hundred; do it on the bottom row and one and a half per cent comes out at about eight. The middle row is the one everybody would have designed, and it is the worst of the three on the column that decides whether the feature works.

The nine-second cap earns its number from the same table rather than from the dependency's dashboard. Spend the largest permitted call and half the eighteen seconds is gone, leaving nine for whatever follows. What usually follows is five more calls at the median, which is two seconds of work — so even the worst single call the design permits leaves four times the budget the rest of the task typically needs. Push the cap higher and that margin closes; pull it lower and the cap starts refusing calls that the tail of the distribution would have completed, which is the even-split failure creeping back in. The number is the largest one that keeps the typical remainder comfortably affordable after a worst case, and that it also happens to sit at the dependency's p99 is not the reason it was chosen.

import time, asyncio

class Deadline:
    def __init__(self, seconds: float, per_call_cap: float):
        self.expires_at = time.monotonic() + seconds
        self.cap = per_call_cap

    def slice(self, minimum: float = 1.0) -> float:
        remaining = self.expires_at - time.monotonic()
        if remaining < minimum:
            raise BudgetExhausted(f"{remaining:.2f}s left, refusing to start")
        return min(self.cap, remaining)

async def call_tool(client, deadline: Deadline, path: str, body: dict):
    budget = deadline.slice()
    # The server is told the deadline too; see the next section for why.
    headers = {"x-request-deadline-ms": str(int(budget * 1000))}
    return await asyncio.wait_for(
        client.post(path, json=body, headers=headers), timeout=budget
    )

BudgetExhausted is a useful error to have, because it is the one that lets an agent degrade honestly. A task that has spent its tool budget can return what it has with a note that the last two reports were not fetched, which is a much better outcome than a task that runs to 74 seconds and gets killed by something that has no idea what it was doing.

A timeout that does not cancel has not stopped anything#

The 12-second number was the visible problem. The one that survives fixing it is that firing a timeout in the caller frequently stops no work at all. asyncio.wait_for cancels a coroutine, which unwinds the client-side await and closes or returns the socket. The reporting service, on the other side of that socket, has already parsed the request, dispatched a query and is holding a warehouse slot; nothing about the client giving up is communicated to it, and it will happily finish its 71-second scan and try to write a response to a connection that is gone. Under a load spike, the effect of aggressive client timeouts without cancellation is that the dependency's queue fills with work nobody is waiting for.

The step the diagram skips is why the service does not simply notice. Closing a socket is not an interrupt delivered to whatever the peer happens to be doing; it is a packet that arrives at the far side's operating system and changes the state of a file descriptor. The reporting service's thread is not looking at that descriptor. It is blocked inside a database driver waiting for rows, and it stays blocked there until the rows arrive, because nothing on that code path polls the client connection. The service finds out that the client is gone at the only moment it looks, which is when it tries to write the response, and by then the expensive work is done. Written as a chain: your cancellation frees your thread, frees your socket and frees your budget, and reaches nothing on the far side, because the far side was never waiting on you. That is what the timeout is worth as a protection for the dependency. Its effect on the dependency's load is nil when the dependency is healthy and nil when it is melting, which is the defect chapter 1, on dry runs and pinned proposals, gave a name to: a defence that does not vary with the condition it is supposed to answer.

Subprocesses have the sharper version of this. subprocess.run(cmd, timeout=30) raises TimeoutExpired and kills the process it spawned, which is often a shell, and the shell's children — the psql, the ffmpeg, the migration script — are left orphaned and running. Getting this right means putting the child in its own process group and signalling the group, then escalating if it does not exit.

import os, signal, subprocess

def run_bounded(cmd: list[str], budget: float) -> subprocess.CompletedProcess:
    proc = subprocess.Popen(cmd, start_new_session=True,          # its own process group
                            stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    try:
        out, err = proc.communicate(timeout=budget)
    except subprocess.TimeoutExpired:
        os.killpg(proc.pid, signal.SIGTERM)                        # the whole tree
        try:
            out, err = proc.communicate(timeout=2.0)
        except subprocess.TimeoutExpired:
            os.killpg(proc.pid, signal.SIGKILL)
            out, err = proc.communicate()
        raise ToolTimeout(cmd=cmd[0], budget=budget, partial_stderr=err[-2000:])
    return subprocess.CompletedProcess(cmd, proc.returncode, out, err)

For network calls, cancellation has to travel to the other end, which requires the other end to cooperate. gRPC carries this natively through deadlines, which propagate down the call chain and cause servers to abort their own work when the deadline passes. Over HTTP it takes a convention — the x-request-deadline-ms header above, or the emerging Deadline field in your internal RPC framework — and a server that actually reads it and sets a matching bound on whatever it does next. For a database that means SET LOCAL statement_timeout = 8500 inside the request, so the query dies with the request instead of outliving it. Without that, the client timeout is a decision about how long you wait, and nothing more.

A timeout is an ambiguous outcome, not a failure#

The reporting service sometimes returns after the cap has fired. For that to be safe, three things have to be true, and they are three separate pieces of work.

The late response must not be applied. If the agent has moved on and the HTTP client's callback still fires, or a retry's response and the original's response both arrive, something must decide which one counts. The usual mechanism is an attempt identifier carried on the call and checked before any result is written into the run's state, so that a result belonging to attempt 1 is discarded once attempt 2 has been started. Without it you get results interleaving out of order, which produces symptoms far stranger than a plain error.

The resources must be released. A timeout that leaves the connection in the pool half-read will eventually exhaust the pool, and the failure then presents as every tool call hanging, which is a much harder incident to read than the one you started with.

And the agent's model of the world has to treat the timeout as unknown rather than as failed. This is where read tools and write tools diverge sharply. A timed-out fetch_report did nothing to the world and can be retried or abandoned freely. A timed-out cancel_subscription may have cancelled the subscription, may have cancelled it and failed to return, or may never have reached the service, and the agent has no way to tell from the client side. The correct handling is to reconcile before doing anything else: query for the subscription's current state, and only then decide whether to retry. Tools that mutate should also accept an idempotency key generated once per logical intent, so that a retry with the same key is recognised by the dependency as the same operation rather than a second one.

async def cancel_subscription(sub_id: str, idem_key: str, deadline: Deadline):
    try:
        return await post(f"/subscriptions/{sub_id}/cancel",
                          headers={"Idempotency-Key": idem_key},
                          timeout=deadline.slice())
    except asyncio.TimeoutError:
        state = await get(f"/subscriptions/{sub_id}", timeout=deadline.slice(minimum=0.5))
        if state["status"] == "canceled":
            return state                      # it happened; report it as such
        raise ToolTimeout(op="cancel_subscription", state="unknown", sub_id=sub_id)

The idempotency key has a condition attached that decides whether it does anything, and it is easy to implement backwards. The key has to name the intent — this run, this turn, this subscription — so that every attempt at the same intent carries the same key and the dependency can recognise the second one as the first. A key minted inside the retry path, one per attempt, is a fresh value on every call, which is exactly the same as sending no key at all: the dependency sees two distinct operations and performs two. The tempting version is the wrong one, because generating an identifier at the point of sending is the natural place to generate identifiers. Notice also which direction the key protects. It stops a retry becoming a second cancellation; it does nothing about the first attempt having succeeded and the client believing it failed, which is why the reconciling read comes before the retry rather than instead of it.

When retrying makes the outage#

The instinct after adding a timeout is to add a retry, and for the reporting service specifically that instinct is wrong in a way worth being precise about. The tail past 9 seconds comes from cache misses on evicted partitions, which means the slow calls are slow because the service is doing expensive work. Retrying issues a second expensive request while the first is still running and still holding a warehouse slot, so the retry does not replace the original load, it adds to it. If the tail was caused by a load spike, every client retrying at once converts a latency problem into a saturation problem, and the classic shape of that incident is a service that recovers only after clients are throttled.

Counting warehouse slots makes the amplification concrete. At the instant the cap fires, the task is holding one slot on a scan that has most of its ninety seconds still to run, and — for the reason the previous section gave — the client giving up does not release it. Issuing a retry makes it two slots for that same task, and the first is returned when the original query finishes rather than when the retry is sent. So the retry does not swap a slow call for a fast one; it holds both for the remainder of the slow one's life. Two conditions turn that from a nuisance into an outage. The slow calls are correlated across clients, since they are slow because the same partitions were evicted, so many clients reach the cap at once. And the doubling is per client, so the demand on the pool roughly doubles at exactly the moment the pool is already the constraint. Retrying is a good strategy when a request failed for a reason local to that request, and a bad one when the reason is that the dependency is busy — and a timeout on its own cannot tell you which of those you are in.

There is an honest counter-argument, which is that the first request may have warmed the cache, so a retry a few seconds later could return in 400ms. That is sometimes true and it is not a reason to retry blindly. If you want the benefit, the shape that gets it without the amplification is a hedged request — a second copy of the call issued once the first has passed p95 without answering, with whichever response arrives first being used — issued at p95 rather than after a full timeout, capped at one hedge, and only for idempotent reads. Even then it belongs under a retry budget, where the client refuses to spend more than about 10% of its total request volume on retries and stops entirely when a circuit breaker, which cuts off traffic to a dependency once its error rate crosses a threshold and lets it recover, sees that error rate climb. For the six-call task above, none of that fits in an 18-second budget alongside the original calls, so the right answer for this tool is to fail the call, record it, and let the agent proceed with a partial answer.

Timeouts are also frequently asked to do jobs they cannot do. They are not a cost control, because a query that returns in 2 seconds can scan 400GB and cost more than one that takes 40. They are not a rate limit, because a tool that returns in 300ms can be called eight hundred times in a task. And they are not a substitute for concurrency limits on the dependency, which is the control that actually protects it.

What a timeout gives you is a bound on how long a single call ties up the task, plus — when cancellation genuinely propagates — a bound on how long it ties up the dependency. The first is bought by dividing an end-to-end target rather than by reading a dashboard; the second by process groups, deadline headers and a server willing to honour them. A cap taken from the dependency's p99 and enforced with wait_for alone delivers neither, which is why it survives so long: it looks like both.

The state="unknown" case is where this chapter runs out of road. When a write times out with the outcome undetermined, when a retry may or may not have duplicated it, and when someone asks two weeks later what the agent actually did, the answer is not available from the tool's return value, because there wasn't one. It has to come from what was recorded while it happened.