Retry-After and rate-limit headers

Provider accounts are metered on at least two axes at once. There is a requests-per-minute allowance, which counts calls regardless of size, and a tokens-per-minute allowance, which counts the tokens those calls consume and is frequently split further into separate input and output budgets. Both are usually implemented as continuously-refilling token buckets rather than as counters that reset on the minute, so capacity trickles back rather than arriving all at once. Exceeding either one produces the same HTTP 429, and the response carries headers reporting the state of every budget along with an explicit instruction about when to come back.

A token bucket is the standard name for that metering arrangement, and it is worth unpacking because every delay in this chapter is computed from it. The provider keeps one running number per budget — how much of the allowance is currently available — and adds to it continuously at a fixed rate, up to a ceiling which is the allowance itself. Each admitted call subtracts its own cost from that number. A call is admitted when the number is at least as large as what the call costs, and rejected when it is not. Nothing here resets: the number only ever moves by additions from the refill and subtractions from traffic.

The alternative arrangement, a counter that zeroes on the minute, is worth looking at once because it makes clear what the bucket is buying. A per-minute counter permits an account to spend its entire allowance in the first millisecond of the minute and then be refused for the remaining fifty-nine seconds, which is a shape of load the hardware cannot absorb. It also gives every client on the platform the same reset instant, so the boundary of each minute becomes a scheduled stampede, for exactly the reason the previous chapter gave: identical deadlines produce identical wake-ups. A continuous refill has neither property. Capacity returns in proportion to time elapsed, which is what makes "wait and try again" a repair, and it makes the amount of waiting computable rather than a guess.

Those headers are the subject of this chapter, and the reason they matter more than they look is that the trickle rate relative to what a single call costs is the whole story. A retry schedule invented from an attempt counter — one second, then two, then four — knows nothing about which budget was exhausted or how much of it one call consumes, which is how a single policy manages to be too aggressive and too slow on the same account at the same moment.

One status code, two budgets#

Take an account rated at 4,000 requests per minute and 400,000 input tokens per minute. The request bucket refills at 66.7 requests per second and one call costs one request, so from a completely empty bucket another call can be admitted after 15 milliseconds. The token bucket refills at 6,667 tokens per second, so a 60,000-token document-ingest call must wait nine seconds from empty before it can be admitted, while a 2,000-token chat call needs 0.3 seconds of the same refill.

Every figure in that paragraph comes from two divisions, and both are worth doing rather than reading. The refill rate is the allowance spread over the window it is quoted for, so 4,000 requests a minute is 4,000 divided by the sixty seconds in a minute, which is 66.7 a second, and 400,000 input tokens a minute is 400,000 divided by sixty, which is 6,667 a second. The wait from empty is then the cost of the call divided by that refill rate, because the bucket has to accumulate the whole cost before the call can be admitted and it accumulates at a fixed rate. One request divided by 66.7 a second is 0.015 seconds. Sixty thousand tokens divided by 6,667 a second is nine seconds. Two thousand tokens divided by the same 6,667 is 0.3 seconds.

The intuition to take from those three sums is that the budgets are not refilling at different speeds relative to their own sizes — each one refills its whole allowance in a minute, by construction. What differs is the size of one call measured against its budget's refill. A request costs one sixty-seventh of a second's worth of request refill. An ingest call costs nine seconds' worth of token refill, six hundred times as much in units of waiting. One status code covers both, and the number underneath it differs by nearly three orders of magnitude.

The fifth row is where the failure is visible, and it is a comparison of two numbers that were computed in different places. A schedule of one, then two, then four seconds spends its whole budget after one plus two plus four, which is seven seconds of waiting. The ingest call needs nine. So the last attempt is made two seconds before the bucket can possibly hold enough, and it is made against a bucket the client has just been told is empty. Nothing about that outcome is probabilistic: the schedule expires while the condition it is waiting for is still, arithmetically, in the future.

The last row is the useful one for anyone tempted to fix this by turning the attempt count up. The large-prompt path does not need more attempts; it needs one attempt placed after nine seconds instead of five attempts crowded into seven, and those are different configurations even though both take about the same wall-clock time. Nine seconds of sleeping followed by one call costs one request against the budget. Five attempts within nine seconds cost five, four of which are guaranteed rejections that make the bucket's recovery marginally worse for everyone else on the account.

A large prompt against a token bucket has an awkward property that a small one does not: it can be starved indefinitely if smaller calls keep draining the bucket as it refills. An ingest call needs the bucket to reach 60,000 tokens, and every chat call arriving in the meantime takes 2,000 off the top. With enough chat traffic the bucket never gets there, so the large call fails because it can never accumulate enough at one instant while staying well inside its allowance in aggregate. No retry schedule solves that. It wants a reservation, a separate account or key for bulk work, or a scheduler that admits the large call when the reported headroom is already sufficient.

The starvation is a consequence of the bucket being one number rather than a queue, and following it through is what makes the case convincing. The level rises at the refill rate and drops by the cost of each admitted call. An ingest call is admitted only at an instant when the level has reached sixty thousand. A chat call is admitted whenever the level has reached two thousand, which is almost always, and takes that much back off the top. So the level is being harvested near the bottom of its range while the ingest call is waiting for the top of it. Put a chat call every three hundred milliseconds against this account: that is a little over six thousand tokens a second consumed, against a refill of 6,667 a second, so the level climbs by a few hundred tokens a second and would need minutes of uninterrupted quiet to reach sixty thousand. There is no queue to join and no position to hold, because the bucket has no memory of who has been waiting. The ingest call is not being treated unfairly by a scheduler; there is no scheduler.

What the headers actually say#

None of that arithmetic has to be inferred. A rate-limit rejection reports the state of every budget the provider is metering, and a token-exhausted rejection is the mirror image of a request-exhausted one:

HTTP/1.1 429 Too Many Requests
retry-after: 9
anthropic-ratelimit-requests-limit: 4000
anthropic-ratelimit-requests-remaining: 3891
anthropic-ratelimit-requests-reset: 2025-03-11T14:22:41Z
anthropic-ratelimit-input-tokens-limit: 400000
anthropic-ratelimit-input-tokens-remaining: 0
anthropic-ratelimit-input-tokens-reset: 2025-03-11T14:22:50Z

Read as a diagnosis, this response says the request budget is essentially untouched with 3,891 of 4,000 remaining, the input-token budget is at zero, and the provider expects it to be usable again in nine seconds. Every part of the earlier arithmetic is stated outright.

That last equality is the one to notice, because it is a check rather than a coincidence. The nine in retry-after is the same nine that came out of sixty thousand tokens divided by a refill of 6,667 a second. The provider is computing the wait from the state of its own bucket, which is the calculation the previous section did from the outside with published numbers. When a reported value and your own arithmetic disagree, the reported value wins — it knows about traffic from every other process on the account, and the arithmetic does not — but agreeing is the ordinary case, and it is what justifies treating these headers as a measurement rather than as advice.

Retry-After is defined by HTTP as either a number of seconds or an absolute HTTP-date, and providers use both forms, so a parser that assumes an integer will throw on Wed, 11 Mar 2025 14:22:50 GMT at the least convenient moment. The reset headers are typically RFC 3339 timestamps, which introduces the other parsing hazard: comparing a provider timestamp against your own clock imports whatever skew exists between the two machines. A server whose clock is four seconds fast computes a negative delay and retries immediately. Clamping the computed delay to a sane range, and preferring the relative retry-after over an absolute timestamp when both are present, removes an entire category of incident that is very hard to reproduce.

The skew failure is one subtraction, and seeing it makes clear why the relative form is safer rather than merely more convenient. Turning an absolute timestamp into a delay means computing the reset time minus the current time, where the reset comes from the provider's clock and the current time comes from yours. If your clock runs ahead by four seconds, every delay computed this way is four seconds shorter than the truth. That is the whole of the fault, and it explains its shape: the claim above holds for any wait shorter than the skew, so a nine-second reset still leaves five seconds of sleep and looks fine, while the fifteen-millisecond request-budget waits go negative and the client hammers. The bug therefore appears only on the budget that recovers fastest, which is the one nobody is investigating.

The relative form has no such term in it. retry-after: 9 is a duration, and a duration does not need the two machines to agree on what time it is — only on how long a second lasts, which they do to a far better tolerance than they agree on the offset. Clamping is the belt to that brace. Forcing the computed delay into a sane range means a badly wrong clock produces an immediate retry rather than an exception or a sleep measured in hours, and it puts a ceiling on what a malformed header can talk a worker into doing. Both bounds in that line are doing separate jobs: the lower one is for clocks, and the upper one is for values nobody meant to send.

from datetime import datetime, timezone
import email.utils, random

def reset_delay(headers: dict, fallback: float) -> float:
    raw = headers.get("retry-after")
    if raw:
        try:
            delay = float(raw)                       # seconds form
        except ValueError:
            dt = email.utils.parsedate_to_datetime(raw)   # HTTP-date form
            delay = (dt - datetime.now(timezone.utc)).total_seconds()
    else:
        # No retry-after: fall back to the exhausted budget's reset.
        resets = [headers.get(f"anthropic-ratelimit-{b}-reset")
                  for b in ("input-tokens", "output-tokens", "requests")
                  if headers.get(f"anthropic-ratelimit-{b}-remaining") == "0"]
        stamps = [datetime.fromisoformat(r) for r in resets if r]
        if not stamps:
            return fallback                          # our own schedule
        delay = (max(stamps) - datetime.now(timezone.utc)).total_seconds()
    delay = min(max(delay, 0.0), 120.0)              # clamp skew and absurdity
    return delay + random.uniform(0.0, 0.25 * max(delay, 1.0))

The final line is the part most implementations leave out, and it looks like it contradicts the rest of the chapter. If four hundred workers are all told retry-after: 9, they will all sleep exactly nine seconds and all return in the same instant, which is the lockstep from the previous chapter reconstructed out of the provider's own advice. A reported reset is authoritative about the earliest moment capacity exists and says nothing about how many other callers were given the same number. The correct treatment is to use it as a floor and add a positive random offset sized as a fraction of the delay, with a uniform draw across the first quarter of the wait as a reasonable default. Subtracting is the mistake to avoid, because a jitter that can pull the retry earlier than the reported reset guarantees a rejection and wastes a request against a budget that is, by construction, empty.

The reason the provider hands the same number to everyone is not carelessness, and seeing why settles the apparent contradiction. The value describes the bucket, and the bucket is one object shared by every process using that account. "Capacity exists again at 14:22:50" is a fact about a resource, not an appointment made with a particular caller, and it would be false to give different callers different times because there is only one level rising at one rate. Everything the provider knows is in that number. Everything it does not know — how many of its callers are yours, and whether they are about to arrive in a single instant — is on your side of the connection, which is where the correction has to be applied.

The direction of the correction follows from what the number means. The reported reset is the earliest moment the bucket can satisfy the call, so any draw that lands before it is asking for a certain rejection: a round trip, a connection, and a place in the provider's admission path spent to be told something the header already said. A draw that lands after it might be unnecessary and might not, depending on who else arrived, which makes the two directions asymmetric in a way symmetric jitter cannot express. Adding a positive offset therefore risks a little latency, and subtracting one guarantees a wasted attempt.

These headers arrive on successful responses too, which is the cheaper way to use them. A 200 reporting 8,000 input tokens remaining out of 400,000 is saying a rejection is about a second away, and a client that reads the remaining counts on every response can slow itself down before the provider has to. Feeding *-remaining into an admission gate — hold large calls when the reported headroom is below what they cost, release them when it recovers — converts a reactive retry loop into pacing, and pacing does not consume request budget to discover it has none. The state is slightly stale, since other processes on the same account are spending the budget concurrently, so treat it as a signal for a soft threshold rather than as an exact ledger.

Where the "about a second away" comes from is the refill rate again, used in the other direction. Eight thousand tokens remaining out of 400,000 is two per cent of the budget, and at a refill of 6,667 a second that headroom is a little over a second's worth of refill. A caller consuming faster than the bucket refills — which is the only way it got down to two per cent — closes that gap in about a second. The same reading also settles what can be admitted right now: eight thousand tokens is four chat calls at two thousand each, and it is not an ingest call at sixty thousand, which is a decision the client can make without asking. That is the difference between pacing and retrying. One reads a number it already has; the other spends a request to find out.

When honouring the reset is the wrong move#

A reported reset does not obligate anyone to wait for it. Nine seconds is a fine thing to sleep in a batch worker and an unacceptable thing to sleep inside an HTTP request that a browser is holding open with a four-second patience budget. The rule that falls out is to compare the delay against the deadline remaining for the call, and where the delay exceeds it, decline to retry at all — return the failure, shed the request, or hand the item to a delayed queue. Sleeping for nine seconds and then discovering the caller has gone spends nine seconds of a connection, a worker and a rate-limit slot to produce nothing.

The comparison is between two numbers the client already holds, which is what makes the rule cheap enough to apply on every rejection. One is the delay the provider reported. The other is the time left on this call's own deadline — what remains of the four seconds after however long the first attempt took. If the reported nine is larger than what remains, no arrangement of attempts fits, and that verdict is available before any sleeping happens rather than after. The decision it enables is not "retry or fail" but "retry here or somewhere else": an item handed to a delayed queue is still going to be processed, by a worker with no browser waiting on it and no deadline to breach.

The header is also a statement about a budget rather than a promise about the outcome. Waiting the reported nine seconds means the budget will have refilled; it does not mean the request succeeds, because every other caller on the account is also arriving at that moment and the bucket is shared. Treating the reset as a guarantee produces a policy with one attempt and no fallback, which fails badly on a busy account.

Then there is the case the whole mechanism does not cover. 529 overload responses generally arrive with no retry-after and no budget headers at all, and there is a good reason: the provider does not know when capacity returns, because that depends on aggregate demand across every tenant rather than on a bucket with your name on it. The absence is itself informative. A rate limit hands you a number because the constraint is a quantity the provider tracks on your behalf; a 529 hands you nothing because the constraint is a condition nobody can schedule.

So the two cannot share one policy. The rate-limit branch honours a reported reset, needs almost no invented backoff, and knows in advance whether the wait fits inside the caller's deadline. The overload branch has nothing to honour, needs the full jittered exponential schedule with a hard attempt cap, and cannot predict its own duration. The argument from the first chapter of this topic about which failure deserves a page follows from the same asymmetry: the one that reports a number is the one where the number is yours to change.

Both branches assume a call that fails cleanly with a status code and a set of headers to read. The next chapter deals with the calls that do not fail at all in any way the client can see: the request that is still open, still connected, and no longer producing anything.