Fixed backoff returns every throttled client at the same instant

At 14:31:07 a provider region degrades. Nothing is down; median completion latency moves from 900 ms to 4.2 seconds and stays there for ninety seconds. A service running at a steady 200 requests per second now has about 840 requests in flight instead of 180, because in-flight count is arrival rate times latency and the latency quadrupled. At 14:31:19 the provider's own protection engages and it starts rejecting, and the 840 requests that were open all fail inside the same few hundred milliseconds.

Every client in that fleet retries up to three times with a flat 100 ms backoff, the backoff being the pause a client waits after a failed call before attempting it again. The interval is the same for all of them and the failures arrived together, so the retries arrive together too. A 100 ms bucket that normally carries 20 requests carries 860 at t+100 ms, another 860 at t+200 ms, and another at t+300 ms — three spikes of roughly 43 times normal offered load into a provider that is already shedding. Each spike is rejected, and each rejection is another synchronised cohort. By 14:32:37 the provider has recovered and the service has not, because the retry waves have kept the account above its ceiling continuously and every new arrival is now landing on top of a rejection backlog it did not create. The slowdown lasted ninety seconds. The incident channel closed at 15:04.

The mechanism has nothing to do with language models. It is the collision problem that Ethernet solved in 1976 with truncated binary exponential backoff: several senders detect the same contention at the same moment, and if they all wait the same interval they collide again, deterministically, forever. The fix there was randomness, and the fix here is the same.

Randomness spreads the load; a budget bounds it#

Exponential backoff alone does not solve the synchronisation problem, which is the part most implementations get wrong. Doubling the interval — 100 ms, 200 ms, 400 ms — still gives every client in the cohort the same interval, so the waves get further apart and stay just as tall. What breaks the phase lock is sampling the delay from a range rather than computing it. Marc Brooker's "Exponential Backoff And Jitter" on the AWS Architecture Blog is the standard reference, and its conclusion is that full jitter — sleeping a uniform random duration between zero and the exponential cap — beats both no jitter and the half-and-half variants that were common at the time.

import random, asyncio

async def call_with_retry(fn, *, attempts=3, base=0.5, cap=8.0, budget):
    for attempt in range(attempts + 1):
        try:
            return await fn()
        except RateLimited as e:
            if attempt == attempts or not budget.try_spend():
                raise
            if e.retry_after is not None:
                delay = e.retry_after + random.uniform(0, 0.5)
            else:
                ceiling = min(cap, base * (2 ** attempt))
                delay = random.uniform(0, ceiling)   # full jitter
            await asyncio.sleep(delay)
        except NonRetryable:
            raise                                     # 400s, context length

Applied to the incident above, full jitter with a 1-second ceiling spreads 840 simultaneous retries across ten 100 ms buckets, so each bucket carries about 84 retries plus the 20 baseline arrivals. That converts a 43-fold spike into a 5-fold one, which the provider may still reject, and which will still keep the account over its ceiling for as long as the retries continue. Jitter changes the shape of the load and not its volume; over the ninety seconds, the same 2.5 times as many requests are sent either way.

There is a variant worth knowing about because it behaves better when the outage is long. Decorrelated jitter draws the next delay from a range whose upper bound is three times the previous delay rather than from a fixed exponential schedule, so a client that has already waited a long time tends to keep waiting a long time, and the population's delays spread out further with each round instead of all collapsing back toward the same ceiling. In Brooker's simulations it completed the same work in fewer total calls than full jitter for contention that persisted, at the cost of a longer tail on individual requests. For a ninety-second event either choice is fine; the choice that is never fine is a delay every client computes to the same value.

The reservation ledger from chapter 2 adds its own twist to this incident. When latency quadruples, holds are outstanding four times as long, so the ledger's held figure climbs even though nothing about the workload changed, and local admission starts refusing requests that the account could actually have served. That is the correct behaviour — the tokens genuinely are committed — but it means the slowdown produces local sheds and provider 429s at the same time, and the two look different in the logs and need to be counted separately. A dashboard that sums them reports a single rejection rate that cannot be attributed to a cause, which is a bad thing to be looking at during the eleven minutes when someone is deciding whether the provider or your own limiter is the problem.

Bounding the volume is what a retry budget does. The rule is that retries may consume only a small fraction of the traffic that is succeeding — gRPC's retry throttling keeps a token bucket that gains a token on each success and loses a configurable fraction on each failure, suppressing retries whenever the bucket falls below half, and Envoy exposes the same idea as a percentage of active requests. During a healthy minute the budget is full and a handful of unlucky requests retry freely, which is exactly the case retries were designed for. During the incident, successes stop, the bucket empties within about two seconds, and retries stop with it. The offered load returns to 200 requests per second, the provider's recovery is not fought, and the service fails honestly for ninety seconds instead of dishonestly for thirty minutes.

Every layer that retries multiplies every other layer#

The counts above assume one retry loop. Real stacks usually have three, and nobody chose the product. The OpenAI Python SDK defaults max_retries to 2, so every call already makes up to three attempts before your code sees an exception; your own wrapper adds three; and a service mesh or API gateway in front commonly retries idempotent-looking requests twice more. A single user action can therefore generate up to eighteen provider requests during an incident, and the team that wrote the middle layer believes the amplification factor is four.

Auditing this is a matter of reading configuration rather than reasoning about it. Set the SDK's retry count to zero wherever you have implemented your own policy, so the jitter and the budget you control are the only ones in play, and disable retries at the gateway for the model path specifically — a gateway cannot see a token budget, cannot distinguish a per-minute 429 from an exhausted monthly quota, and cannot know that a partially streamed response has already cost you the tokens.

The other multiplier is retrying failures that cannot succeed. A request rejected for exceeding the model's context window will be rejected identically on every attempt, as will a malformed tool schema, an invalid model name, or an authentication failure. These are cheap to classify and expensive to ignore, because they are perfectly correlated: a deployment that introduces a prompt-length regression turns a fixed fraction of traffic into an unbounded retry source that fires continuously rather than only during incidents. The classification worth encoding is that 429 with a Retry-After is a wait instruction, 429 without one is a throttle you should back off from with jitter, 5xx is retryable once or twice, and everything in the 4xx family other than 429 is final.

Retry-After deserves the deference. When a provider tells you how long to wait, waiting that long and no less is the only cooperative behaviour available, and adding a small random offset on top keeps the cohort from re-synchronising at the moment the header specifies. Ignoring the header and applying your own shorter backoff is the most direct way to convert a temporary throttle into a sustained one.

A breaker decides when to stop asking#

Backoff governs a single request's behaviour. A circuit breaker — named after the electrical fitting, and working the same way, in that it watches the recent rejection rate and cuts the caller off from the provider entirely once that rate crosses a threshold — governs the caller's behaviour as a whole, which is the thing that needs to change when the rejection rate says the account is over its ceiling rather than that one request was unlucky.

While the breaker is open, requests are shed at the edge in microseconds using the machinery from the previous chapter, so no reservation is taken, no queue accumulates, and the user is shown the busy state rather than a spinner that resolves into a timeout. The half-open probe is what distinguishes this from simply giving up: a small number of requests are allowed through to discover whether the ceiling has cleared, and the probe count has to be small enough that a failed probe does not itself constitute a wave. Five probes against a fleet of forty pods means the breaker must be coordinated or the probes must be sampled, since forty independent breakers half-opening at the same second reproduce the synchronisation problem at a smaller scale.

None of this is a capacity mechanism. Retries, jitter, budgets and breakers together do not serve a single additional token; they only stop the client from making a shortage worse, and they are irrelevant to the case in the previous chapter where offered load genuinely exceeds the ceiling for forty minutes. In that situation a perfectly tuned retry policy sheds promptly and correctly and the requests are still not served. A breaker also cannot distinguish a provider problem from a self-inflicted one, so an account driven over its ceiling by a runaway batch job will trip the breaker for the interactive traffic too, which is an argument for separate breakers and separate budgets per route rather than one global one.

Every chapter so far has treated the ceiling as immovable and asked how to behave underneath it. Ceilings can be bought, at a price that is charged whether or not the capacity is used, and the arithmetic of that purchase is where the next chapter goes.