Built-in retries and backoff

Rate limiting is the provider capping how much of its capacity you may consume in a window — so many requests and so many tokens per minute, counted against your organisation rather than your process — and refusing the excess with a 429 rather than serving it slowly. What a client does with that 429 is the subject here. Both major Python clients wait and try again, on a schedule of their own, before returning anything to your code, so the number of HTTP calls your service makes is not the number of calls it appears to make.

Two pieces of vocabulary run through the argument. Backoff is the wait inserted between attempts, usually growing exponentially, and jitter is a random offset applied to it so that many callers waiting at once do not resume in lockstep. The measurements are latency percentiles: p50 is the median, and p99 is the latency that only the slowest one percent of requests exceed. The reason p99 is the interesting one is that retries are rare by construction, so they never move the median and they can dominate the tail entirely.

What the client does when the first attempt fails#

Both clients default to two retries, meaning up to three attempts per call, and both make the retry decision from the HTTP status of the response rather than from anything in the body.

The wait between attempts roughly doubles each time, starting around half a second and capped in the region of eight, and each wait carries jitter. Both clients prefer the server's own instruction when there is one: a retry-after header in seconds, or the millisecond-resolution variant some endpoints send, overrides the computed backoff. That is a good policy, and the reason it exists is that an unsynchronised fleet of workers all retrying after exactly two seconds re-creates the burst that caused the throttling in the first place.

Both parts of that schedule earn their place for reasons worth having explicitly, because a hand-rolled loop usually has one and not the other. Doubling the wait is what makes the strategy converge: if the cause has not cleared, each round halves the load you offer, so a client that keeps failing keeps stepping out of the way rather than leaning harder. A fixed wait does the opposite, holding your offered load constant against a service that is already refusing it. Jitter handles the fact that you are not one caller. Every worker that was throttled was throttled at roughly the same moment, so every worker computes roughly the same backoff, and without a random offset they all return together — reproducing the spike, getting throttled again, and synchronising a little more tightly each round. Doubling controls how hard you push; jitter controls whether you push in unison.

Notice which piece of information neither of them uses. The decision to retry at all was made from the status code, and nothing in the schedule consults why the status was returned. That is deliberate on the client's part: a status is the one part of a provider's error vocabulary that means the same thing everywhere, so a general-purpose library that wants to work against several providers has very little else it can safely read.

Work the arithmetic on a service where a successful call takes 1.4 s and, during the afternoon peak, roughly 7% of calls come back 429 with retry-after: 4. Each of those now costs 1.4 s for the rejected attempt, 4 s of waiting, and 1.4 s for the successful retry: about 6.8 s end to end.

Seven percent is well above the ninety-ninth percentile, so the entire tail is now made of retried calls, and the p99 number is essentially "how long a single retry takes". The requests being throttled are the same requests either way; only the place where the throttling appears in your telemetry has changed.

That sentence about the tail is doing real work, so it is worth walking. A percentile is a position in a sorted list of durations: the p99 is the value you reach after stepping past the fastest ninety-nine percent. Sort this afternoon's calls by duration and the retried ones sit at the top, because each of them carries a rejected attempt, a wait and a second generation. There are seven of them in every hundred. Stepping past ninety-nine of those hundred therefore lands you inside that group of seven rather than short of it, so the number reported is a retried call's duration — near the 6.8 s computed above — and not a generation's. Had the throttling rate been well under one percent, the same retries would have happened and the p99 would not have moved at all, which is why this defect is invisible on some services and dominant on others with identical code.

The other number the table quietly contains is the attempt ratio. Seven calls in every hundred are made twice and ninety-three are made once, so a hundred logical calls produce a hundred and seven HTTP calls, and the ratio is 1.07. That is the same fact as the p99 change, counted a different way, and it is the more useful of the two because it is a direct measurement rather than an order statistic — it moves proportionally as soon as anything starts retrying, instead of sitting still until the retried population crosses one percent.

The compounding case is worse and very common. Application-level retry decorators are usually already present on the function that calls the provider, so a wrapper configured for three attempts around a client configured for three attempts is nine HTTP calls for one logical request, and a job queue that retries the whole task three times makes it twenty-seven. Each of those attempts that reached the model before failing was a generation you paid for. The multiplication is invisible in code review because the two retry policies live in different files and neither one is wrong on its own.

Those two figures multiply rather than add, and the reason is worth naming because it is what makes the growth surprising. Each layer treats the layer below it as a single operation that either succeeds or fails. The decorator does not see three HTTP calls; it sees one function call that raised. So its three attempts are three attempts at something that is itself three attempts, which is nine. Add a queue that redelivers a failed task three times and it is three of those nines. Nothing in any of the three files is unreasonable, and no reviewer looking at one of them would object — which is exactly why the number that reaches the provider has to be measured rather than reasoned about.

Timeouts sit underneath all of this and are usually left at their defaults too. Both clients ship with a request timeout measured in minutes rather than seconds, which is a reasonable default for a library that cannot know whether you are asking for 50 tokens or 8,000, and an unreasonable one for a synchronous API serving a web request. The interaction with retries is what makes it sharp: a timeout is a retryable condition, so a call that hangs consumes the full timeout, waits, and hangs again, and the worst case for one logical call is the timeout multiplied by the attempt count plus the backoff. Setting the timeout from your own latency budget — a connect timeout of a few seconds and a read timeout sized to the largest generation you actually request — bounds that product to a number you chose.

Making each attempt visible#

The fix for the observability problem is not to remove the retries, it is to stop having a layer of the stack your metrics cannot see. The httpx event hook that exposes the outbound request also gives you the seam here: hooks fire per HTTP request, not per SDK call, so counting them counts attempts.

import httpx
from anthropic import Anthropic

ATTEMPTS = metrics.counter("provider_http_attempts_total", ["status", "retry_after"])

def on_response(response: httpx.Response) -> None:
    response.read()
    ATTEMPTS.labels(
        status=str(response.status_code),
        retry_after=response.headers.get("retry-after", ""),
    ).inc()
    log.info(
        "provider_attempt",
        status=response.status_code,
        request_id=response.headers.get("request-id"),
        tokens_remaining=response.headers.get("anthropic-ratelimit-tokens-remaining"),
    )

client = Anthropic(
    http_client=httpx.Client(event_hooks={"response": [on_response]}),
    max_retries=2,
)

With that in place, the ratio of provider_http_attempts_total to your own logical call counter is a direct measure of how much retrying is happening, and a rise in it is the earliest warning that you are approaching a quota. The rate-limit headers logged alongside are the other half: they tell you which budget you are consuming, since these endpoints meter requests per minute and tokens per minute separately, and a workload with modest request rates and very large prompts will exhaust the token budget while the request budget looks comfortable.

Two metrics are worth deriving from that stream of attempts. The attempts-per-call ratio is the direct measure of hidden work, and a value of 1.07 tells you seven percent of calls are being retried once whether or not anything else in your dashboards moved. The distribution of time spent waiting, as opposed to time spent generating, is the second, and it answers the question a p99 alert actually raises: whether the tail is slow because the model is slow or because your requests are queuing behind a quota.

There is also a cost dimension that no latency dashboard shows. A retried attempt that reached the model before failing has already consumed input tokens, and depending on where the failure happened it may have generated output too. A 429 costs nothing because the request was rejected at the edge, but a read timeout on a request the provider completed is a full generation you paid for and threw away — and the retry pays for it again.

The gap between those two is where the failure was decided, and it is the only thing that determines whether a retry is cheap. A throttled request never reaches a model: the quota check happens before any work is scheduled, so refusing it costs the provider nothing and costs you nothing. A read timeout is decided on your side, by your client giving up on a request that the provider is still happily serving — everything about that request is proceeding normally, the tokens are being read, and the reply will be produced and billed to an account that is no longer listening. Which is why "a retry is free" is safe to believe about exactly one of the statuses in the table above and unsafe about the rest, and why the retryable list is a list of things to wait for rather than a list of things that cost nothing.

On a workload with 40,000-token prompts, a two-percent timeout rate with two retries is 1.6 million wasted input tokens per hundred thousand calls, which is real money attached to a metric most teams do not have.

When to own the backoff yourself#

Turning the built-in retries off is a real option, and max_retries=0 at client construction is the whole of the change. It makes sense when you already have a retry policy that knows things the client does not: a queue that can defer a job by thirty seconds without holding a worker thread, a request-scoped latency budget that makes a four-second wait pointless because the caller will have given up, or a scheduler that needs to shed load rather than to queue it. It also makes sense when the calls are expensive enough that a blind repeat is a meaningful cost — a 40,000-token prompt retried twice is 120,000 input tokens for one answer.

Leaving them on is the right default for a service without any of that. The client's policy honours retry-after, jitters its backoff, and does not retry the statuses that cannot benefit from retrying, which is more than most hand-rolled loops do. What you should not do is leave them on and also wrap the call in your own retry decorator, because the product of two policies is a policy nobody designed. Pick the layer that owns backoff, and set the other to one attempt.

Two limits are worth stating plainly. Retries do not solve rate limiting; they smooth over brief bursts. If your sustained demand exceeds your tokens-per-minute allowance, every attempt is competing for a budget that is already spent, and the retries convert a throughput problem into a latency problem and then into a timeout problem — the fix there is concurrency control, batching, or a quota increase. And retries interact badly with streaming: once the provider has begun sending an event stream, a failure part-way through is not a clean transport failure the client can silently repeat, so the built-in policy generally does not cover it, and mid-stream errors remain yours to handle.

The habit this chapter argues for is narrow. Know your client's retry count and timeout as configured values rather than as defaults, count HTTP attempts separately from logical calls, and make sure exactly one layer of your stack decides when to wait and for how long. None of that removes retrying, which is worth having; it removes the situation where a fifth of your peak-hour latency is produced by a layer that emits no telemetry and appears in no dashboard.

One further limit remains, and it is the interesting one for what follows. The retry decision in the table above is made purely from the status code, which assumes that a status code identifies a condition well enough to act on. It does not, quite. A 429 that means "you sent too many requests this minute" and a 429 that means "your account has no credit left" are the same number to a client deciding whether to wait and try again, and only one of them will ever succeed on the second attempt. Providers disagree about which status each underlying condition gets, and some conditions do not get a status at all.