Retrying a failed API is not a reason to add a loop

One sentence turns up in design documents often enough to be worth taking apart on its own: we made it an agent so that it can retry when the API fails. It is persuasive because it joins two things that sound like one thing. There is a failure that needs handling, and there is a shape of program that will handle it, and the sentence implies the second follows from the first. It does not. A retry policy is what a program does when a dependency it called returns an error that it might not return on the next attempt. Control flow is the part of a program that decides which step runs next. The first is a property of one step; the second is a property of the whole program.

Retries, exponential backoff, jitter, circuit breakers, idempotency keys — a token sent with a request so that a repeat of it is recognised and applied once, which is what makes an operation idempotent, or safe to send twice — timeouts and fallbacks are things that every architecture that talks to a network has, and has had since long before anyone put a language model in one. A cron job has them. A Kafka consumer has them. A three-line shell script that curls an endpoint has them, badly. The question of what your program does when a dependency returns 502 lives entirely inside a single step, and the answer is a policy attached to that step.

A retry decision never leaves the step that made it#

The reason the policy stays inside one step is worth walking rather than asserting, because it is the load-bearing link in the whole argument. A transport failure — a connection reset, a gateway error, a rate limit — tells you that a call did not complete. It does not tell you anything about the task the call was part of. The information content of a transport failure is exhausted by the status code itself, so the set of sensible responses is fixed before the run starts: wait and send the same request again, wait longer and send it again, or stop. Nothing about the surrounding work narrows that set, which means nothing outside the step needs to see the failure in order to choose among them. A decision whose inputs are all local is a decision that can be made locally, and making it anywhere else is strictly more machinery for the same outcome.

Written as code, that policy is a decorator on the function that makes the call. This one belongs to a service that reads scanned documents, where optical character recognition — OCR, the step that turns a page image into text — is supplied by an outside vendor:

@retry(
    retry=retry_if_exception_type(TransientOCRError),
    wait=wait_exponential_jitter(initial=0.5, max=8),
    stop=stop_after_attempt(4),
    reraise=True,
)
def ocr(document: bytes) -> OcrResult:
    resp = httpx.post(OCR_URL, content=document, timeout=20.0)
    if resp.status_code in (429, 502, 503, 504):
        raise TransientOCRError(resp.status_code)
    resp.raise_for_status()
    return OcrResult.model_validate(resp.json())

Four attempts, capped at eight seconds of backoff, with a distinct exception type so that a 400 — a malformed PDF, which no number of retries will fix — is not retried at all. Every one of those is a design decision and none of them is a requirement. The requirement is that documents must not be silently lost. The attempt count, the backoff ceiling, the jitter that spreads retries so a fleet does not resend in lockstep, and the exception type that separates the retryable failures from the permanent ones are four choices somebody made in service of it, and each could have been made differently without touching the requirement. Keeping the two apart is what stops a conversation about a number from turning into a conversation about an architecture.

The volume that policy has to cover comes out of two figures and nothing else. A vendor returning an error on some proportion of requests, against a daily document count, gives a daily failure count as a plain product — and because the two are independent of everything else in the system, the product is complete. There is no third factor waiting to be discovered, so a failure count read off those two figures is the failure count, not an estimate of it. For this service, an error rate of about 0.8% of requests at 12,000 documents a day is roughly 96 failures. That decorator sits on a function inside a fixed pipeline and covers 96 documents a day at a cost of a few extra HTTP requests. Adding roughly 60 milliseconds of median overhead on the documents that need it, and none on the 99.2% that do not.

The complement is the part that makes the product useful for planning. The failing share and the non-failing share exhaust the traffic between them, so the overhead applies to one of them and provably not to the other. That licenses a prediction rather than a measurement: at any volume, on any day, the added latency of the retry policy is the per-failure overhead multiplied by the failing share, and the remaining traffic is untouched. Nobody has to instrument the happy path to know its cost went up by nothing.

What the same retry costs inside a loop#

When the retry decision is made by the model instead, the mechanics change in a way that is easy to underestimate until you look at the token accounting. The failure has to be turned into text, appended to the message list as a tool result, and sent back to the model along with everything else that has happened in the run so far. The model then produces a turn that requests the same call again. Compare the two paths for one 502 on one document:

That last row is the one that keeps people up, and it is the reason the table has to be read as a set of gaps rather than as a set of levels. Tokens, latency and cost are all differences of degree, and a reader who stops there concludes that the loop is the same design a few seconds slower. The reading that matters is the gap in the last two rows, between a guarantee and a tendency. The fixed policy retries, exactly four times, the identical call, because those are the only things the code can do. The loop retries usually, and usually the same call. Two of the words in that column are hedges, and they are hedges about behaviour rather than about measurement.

The cost row is worth pausing on, because it is a figure with nothing hiding in it. The whole of the added cost per occurrence is one model turn: an input side carrying the entire message list, an output side carrying the request to try again, and no second call, no retry of the retry, no background summarisation. Multiply the two token counts by the provider's per-token prices, add them, and the total is the figure in the table with no remainder. That is what makes it safe to scale — a cost built from a token count and a published rate moves predictably with both, whereas a cost with an unexplained component in it moves for reasons you will find out later.

On a sample of 200 injected 502s against the ingestion loop, the model re-issued the identical call 187 times, altered an argument 9 times — usually dropping the language_hint parameter, which changed the extraction result — and abandoned the document with a summary of the error 4 times, producing a "successful" run with an empty extraction. Two of those three behaviours were bugs, and none of them were configured; they were what the model happened to do. A fixed retry policy has one behaviour and you chose it.

Those three counts sum to exactly the sample, and that is not a decorative property. When a breakdown closes on its denominator, every case is accounted for, so a behaviour that does not appear in the list did not occur — as opposed to occurring and being filed under a category nobody wrote down. The absence of a fourth behaviour here is a measured absence. That is what licenses the word usually in the table's last two rows: it is a rate over a closed set of outcomes rather than an impression, and it means there is no fifth thing the model did that went unrecorded.

The cost per occurrence multiplied out is small here — 96 failures a day at two cents each is about two dollars a day, which is tens of dollars a month, and nobody will notice it. That is precisely why the argument has to be made on grounds other than money. The loop was not adopted for a sum that rounds to nothing on a cloud bill; it was adopted because it retries, which the decorator also does, and it brought four other things along with it.

The four things that arrive with model-controlled iteration#

There is state to persist between steps. A pipeline step holds its state in local variables and a row in a database; if the process dies, the row tells you where you were, and the shape of that row was designed once. A loop carries the whole message list — the user turn, the assistant turns holding tool calls, the result blocks matching each call id — and that structure has to survive a process restart if runs are long enough to span one, which means it has to be serialised, versioned, size-bounded and garbage-collected. The ingestion service's run store held a median of 34 KB per run and a p99 of 410 KB, and its eviction policy took two incidents to get right, the second of which involved a run resuming against a message list whose tool-result blocks had been truncated by a size guard.

There are trajectories to trace. In a fixed pipeline, "what happened to document 4471" is a row: four step names, four durations, one status. In a loop, it is a sequence of variable length, and the observability question changes from how long did step three take to how many iterations were there, which tools were called in what order, and where did the tokens go. Neither of those is hard, and they are different, and the dashboards, the alert thresholds and the on-call runbook all have to be rebuilt around a variable-length object. An alert of the form "step three p95 exceeded 2 s" has no direct translation.

The alerting rebuild is the part teams underestimate. The old pipeline had four alerts, one per step, each of the form "p95 above threshold for five minutes", and each mapped to a runbook entry naming the dependency to check. The loop's equivalents had to be invented: iterations per run above the 99th percentile, token spend per document above a ceiling, the rate of runs terminating at the step cap, and the rate of runs completing in one iteration, which turned out to be the most useful of the four because it caught a prompt regression that made the model skip classification entirely. None of those thresholds could be derived from the old ones, so the first six weeks after the rewrite were spent discovering what normal looked like.

There are budgets to enforce. A pipeline's worst case is the sum of its steps' timeouts, which you can compute by reading the code. A loop's worst case is whatever the iteration cap allows, and the cap only helps if something sensible happens when it is reached — topic 10.1 covered why an exit the model controls is not a termination condition, since the only exit it names belongs to a component you can influence and cannot constrain. The practical consequence here is that a loop needs at least a step cap, a token cap, a wall-clock deadline and a defined terminal result for each. Four knobs where the pipeline had none, and each one is a value somebody has to pick and later defend.

And there are failures that do not reproduce. This is the expensive one. The same PDF, submitted twenty times to the ingestion loop with identical inputs, produced three distinct tool-call sequences: fourteen runs did ocr → classify → extract, four did ocr → extract and skipped classification, two did ocr → classify → ocr → extract because the first OCR result contained a low-confidence region. All twenty produced acceptable output, and the failure a week later — an extraction that dropped the invoice total — came from the second sequence, and reproducing it required running the document repeatedly until the loop chose that path again.

That breakdown also closes on its sample, and it is worth saying what the closure buys, because it is the difference between two very unlike claims. Every one of the twenty runs is in one of the three sequences, so there is no fourth path that happened once and was not written down. The claim "all twenty produced acceptable output" is therefore a claim about the whole sample rather than about the runs somebody looked at, which is what makes the follow-up damning: the defect a week later came from a path that had already been observed passing. Acceptable output on every run of a path is not evidence that the path is correct. It is evidence that the grader agreed twenty times.

# What a workflow gives you for free, and what a loop asks you to build.
def reproduce(document_id: str):
    doc = store.get(document_id)
    return pipeline.run(doc)          # same input, same path, every time

def reproduce_agent(run_id: str):
    run = run_store.get(run_id)       # 34 KB of message list, if it was not evicted
    return replay(run.messages)       # same input, one of N paths

Idempotency stops being something you can reason about#

There is a fifth consequence that belongs to retries specifically, and it is the one that turns a latency discussion into an incident. A retry policy in code knows which calls are safe to repeat, because a human decided per call site: ocr is a pure read and can be repeated freely, ledger.write is not and gets one attempt with an idempotency key derived from the document hash. That distinction lives in the code next to each call, and reviewing it means reading four lines.

Inside a loop, the model sees a uniform surface. Every tool looks like a function it may call, the failure of any of them looks like a tool result containing an error, and the natural reaction to an error is to try again. When the failing call is a write whose response was lost — a gateway timeout after the write succeeded — retrying produces a duplicate, and nothing in the loop's structure prevents it.

That phrase, uniform surface, is the whole mechanism rather than a figure of speech, and it is worth taking apart. Every tool is presented to the model the same way — a name, a schema, a description — so a read and a write are indistinguishable objects in the same list. Every failure comes back the same way too, as a result block whose content happens to describe an error. There is no channel through which the difference between "this call is safe to send again" and "this call moved money" could reach the model, because that difference was never encoded anywhere the model can see.

Follow that through to the row it produces, one link at a time. A gateway timeout arrives on a write. A timeout is a statement about the response and not about the request, so the write may have succeeded and only the acknowledgement may have been lost. The loop cannot tell those two cases apart, because the evidence it holds is identical in both. The model reads a result describing a failure and re-issues the call. The second write lands. Each link is checkable on its own, and not one of them is a mistake — which is why no amount of care inside the loop closes the gap, and why the remedy has to be outside it.

The general remedy is worth stating as a rule, and it needs its conditions stated with it: tools exposed to a model-controlled loop should be idempotent or read-only, and anything else belongs on an edge in code. The condition is exposed to a model-controlled loop — the rule says nothing about writes in general, and it is not a claim that non-idempotent operations are dangerous by nature. It is a claim about who is choosing when to repeat them. That constraint is not a burden on the workflow design, because the workflow already had it implicitly: its writes were never candidates for arbitrary repetition, since nothing was choosing when to repeat them.

Where a model in the loop genuinely earns the retry#

There is a real capability hiding under the bad argument, and separating them is the point of the chapter. A fixed retry policy re-issues the same call; it cannot decide to do something different in response to a failure. When a search tool returns zero results, the useful reaction is usually to reformulate the query, broaden the date range, or try a different index, and no @retry decorator can do that because the decision requires reading the empty result and knowing what the goal was. A model can. That behaviour is genuinely model-controlled iteration, and it is worth having where it applies.

The test to apply to any "the agent handles failures better" claim is whether the failure carries information that changes what should be attempted next. Transport errors, rate limits and timeouts do not: everything they say is already in the status code, which is why the response to them can be written down in advance. Empty result sets, ambiguous matches and partial extractions might, because reading them requires knowing what was being looked for. A 502 from an OCR vendor carries no information that a model can act on, so the only sensible response is to try again later, which is a policy. The sentence in the design document did not describe the capability; it described a policy, and paid for the capability.

The failure that carries no status code#

The honest limitation is that a third category sits between the two the test separates. The OCR API sometimes returns HTTP 200 with output that is structurally valid and semantically wrong — a scanned page where a table has been read column-wise, producing plausible numbers in the wrong fields. No status code catches that, and a fixed policy has nothing to key on. Detecting it needs judgment, and judgment needs a model. What it does not need is for that model to hold the control flow: a validation node that reads the extraction and returns ok | retry_with_deskew | send_to_human is one model call and three enumerated outcomes, which is the shape chapter 1 already covered, where a model's output is a lookup key against a closed set of branches rather than a choice of what runs next.

The stumble this chapter exists to name is not carelessness, and it is worth stating in the form that makes it hardest to fall for. The tempting belief is that the loop is a defence against failure, and it is tempting because the loop does in fact retry — the property everyone was looking for is really there. What varies underneath the constant is what gets retried. The decorator's four attempts are four attempts at the identical call, by construction, on every occurrence forever. The loop's retry was the identical call most of the time, a call with an argument quietly removed some of the time, and an abandonment reported as a success some of the time, and no configuration selected any of that. A defence whose behaviour is a distribution rather than a guarantee is not the defence that was being bought.

One residual is worth naming rather than leaving as an implication. The run store's median and tail sizes are reported, and the fact that its eviction policy took two incidents to arrive at is reported, and what the policy finally became is not. That is a gap in the record and not a detail: a size-bounded store that truncates tool-result blocks is exactly what caused the second of those incidents, so "we fixed the eviction policy" is a claim that needs its mechanism attached before anyone treats it as settled. Where a chapter reports the cost of a component and not its resolution, the honest reading is that the cost is established and the fix is not.

There is also a scope on the whole argument that the chapter should not pretend away. Everything above compares a fixed pipeline against a loop on a task whose failure modes are known, because they were met and catalogued. That reasoning is straightforward once the catalogue exists. It is much harder before, and the strongest version of the case for the loop is not about any failure that has happened yet.

Retrying is not a reason to add a loop, because retrying is available in every design, at a cost of one decorator and a per-dependency policy that a reviewer can read in four lines. What a loop actually buys is model-chosen sequencing, and what it charges for that is a run store to maintain, an observability surface to rebuild, four budgets to defend, failures that cannot be reproduced from their inputs, and the loss of the one thing that made repeated writes safe — a human deciding, per call site, which operations may be sent twice.

The way to keep that trade honest is to name the property you are buying before you name the architecture. If the failure carries information that changes what should be attempted next, a model in the loop is the thing that reads it, and the cost is worth paying for the steps where that is true. If the failure is a status code, the answer was always a policy, and the argument for the loop has to be made on some other ground — usually a claim about inputs nobody has seen yet, which is a claim that can be measured rather than argued.