A token bucket alone still draws 429s from a request ceiling

A month after the token bucket shipped, a customer moved their content-moderation workload onto the platform. Each call sends a 60-token comment with a 40-token instruction and asks for a one-word verdict, so max_tokens is 12 and a full generation costs about 112 tokens. They ran it at 8,000 requests per minute, which their contract allowed, and which came to 896,000 tokens per minute against a 2,000,000-token account ceiling. The tenant dashboard showed them at 24% of their token allocation. The gateway spent that afternoon returning 429s to them and to everybody else, and the header that explained it was anthropic-ratelimit-requests-remaining: 0 against a 4,000-per-minute account limit.

The token bucket was not wrong about anything. It was measuring the resource that varies most and it was measuring it correctly, and there was a second scarce resource, counted in requests, that it had never been asked to watch. Two chapters of request counts failing to see output length make it tempting to conclude that the request count has no job left, and the conclusion is the wrong lesson from the right evidence: what those chapters established is that requests are a bad proxy for token consumption, which is a narrower claim than requests being a bad thing to limit.

Everything downstream of admission is priced per request#

The provider's RPM ceiling is the most visible of these, and it exists because a request costs the serving fleet something before it generates a single token — a slot in the scheduler, a routing decision, a validation pass over the payload, an entry in the batch. Anthropic, OpenAI and the major hosted-inference vendors all publish a request ceiling alongside their token ceilings for that reason, and the ceilings are set independently, so it is entirely ordinary for a workload to be at a quarter of one and at the whole of the other.

Your own infrastructure is priced the same way, and usually more tightly. The list is short and each item has a number attached, which is what makes it worth walking rather than asserting.

An outbound connection pool is sized in concurrent streams. A pool of 512 with a mean streaming response time of 32 seconds sustains 512 ÷ 32 = 16 requests per second, or 960 per minute, and that figure does not move when completions get shorter in tokens, because a short generation with a long time-to-first-token holds a connection just as long as a verbose one. Worker slots behave the same way: each in-flight request holds a coroutine, a response buffer and a partially assembled SSE stream — server-sent events, the long-lived HTTP connection over which tokens are pushed to the client as they are generated — so a gateway node with 4 GiB of headroom and 2 MiB of per-request state tops out around 2,000 in flight regardless of the token arithmetic.

The fan-out is where the number gets large. A RAG request — retrieval-augmented generation, where the gateway fetches supporting passages and pastes them into the prompt before calling the model — makes one embedding call, one vector search, and up to three tool invocations before the generation starts, so 8,000 chat requests per minute become as many as 40,000 downstream calls per minute against a retriever provisioned for 12,000. The retrieval side has no idea how many tokens the eventual completion will run to, and would not care if it did.

Admission itself has a per-request cost that the previous chapter introduced without pricing. Reserve and refund are two Redis round trips, each about 0.35 ms at p50 and 4 ms at p99 on a same-region cluster, plus a usage row written per request and an audit log line. That is 16,000 Redis operations per minute at 8,000 requests per minute, which is nothing, and 400,000 per minute at 200,000 requests per minute, which is a shard. The cost per request is flat in tokens, so the workload that stresses it hardest is precisely the one the token bucket reports as cheap.

The third row is a control the topic has not needed until now and is worth separating from the other two. A rate says how much may start per unit time; a concurrency cap says how much may be unfinished at once, which is what actually governs queueing delay. It is usually a semaphore: a counter of permits taken on admission and returned when the response completes, blocking or refusing while none are free. A tenant running 40 requests a minute of 8,000-token generations is trivially inside both rate limits and can still have 30 requests in flight simultaneously, and it is the 30 that other tenants feel.

Composing them without unwinding work you have already committed#

Having three checks means deciding what happens when they disagree, and the ordering follows from what each one costs to undo. The request check and the concurrency check are scalar: they increment a counter, and reversing them is another increment in the other direction with no fairness consequence. The token check is a reservation that holds capacity nobody else can use until it is refunded, so a token reservation taken and then discarded because a later check failed has denied a competing request for the duration of the round trip. Take the cheap-to-reverse checks first, and take the reservation last, at the point where the decision is final.

class Admission:
    async def acquire(self, tenant: str, route: Route, messages) -> Lease:
        # 1. request rate — a counter, reversible with a decrement
        ok, retry = await self.requests.take(tenant, 1)
        if not ok:
            raise RateLimited(kind="requests", retry_after=retry)

        # 2. concurrency — a semaphore with a hard ceiling, no queueing here
        if not await self.inflight.try_acquire(tenant):
            await self.requests.give_back(tenant, 1)
            raise RateLimited(kind="concurrency", retry_after=1)

        # 3. tokens — the expensive reservation, taken last and only once
        reserved = count_input_tokens(route.model, messages) + route.reserve_output()
        if reserved > self.tokens.capacity_for(tenant):
            await self._unwind(tenant)
            raise RequestTooLarge(reserved=reserved,
                                  capacity=self.tokens.capacity_for(tenant))
        ok, retry = await self.tokens.reserve(tenant, reserved)
        if not ok:
            await self._unwind(tenant)
            raise RateLimited(kind="tokens", retry_after=retry)

        return Lease(tenant=tenant, reserved=reserved, inflight=True)

    async def _unwind(self, tenant: str) -> None:
        await self.inflight.release(tenant)
        await self.requests.give_back(tenant, 1)

Two details in that listing carry more weight than their line count suggests. The RequestTooLarge branch catches the case where a single request's reservation exceeds the tenant's whole bucket capacity, which no amount of waiting will fix; returning a retryable 429 there produces a client that retries forever with a growing backoff and a support ticket at the end of it, whereas a 4xx that names the ceiling and the request's size tells the caller to lower max_tokens or split the input. The other is give_back: a token bucket refund is a genuine credit, but returning a request-counter increment is only approximately correct in a sliding-window implementation, since the timestamp you recorded has already been counted. In practice the drift is a few requests a minute and not worth a second data structure — noting it stops someone from discovering the imprecision later and rewriting the limiter to fix it.

If your limiters live in Redis, all three checks can be one Lua script over three keys, provided the keys share a hash tag — the braced substring Redis hashes instead of the whole key name — so they land in the same slot — {acme}:req, {acme}:inflight, {acme}:tok. That collapses three round trips into one, keeps the composite decision atomic, and removes the unwind path entirely, because a script that fails at step three simply never performs steps one and two.

The refusal a client receives should say which ceiling it hit, since the remedy differs by an order of magnitude in wait time and by kind in code changes.

{
  "type": "error",
  "error": {
    "type": "rate_limit_error",
    "limit": "requests_per_minute",
    "message": "Tenant acme is at 4000/4000 requests per minute. Token budget is at 24% and is not the constraint; batch multiple items per request or retry in 180ms."
  }
}

The remedy suggested in that message is worth thinking through, because it moves load between the two limiters rather than removing it. Batching sixteen comments into one moderation call takes 8,000 requests per minute down to 500, which clears the RPM ceiling with room to spare, and it raises tokens per request from 112 to roughly 1,100 — the sixteen comments plus one shared instruction block instead of sixteen copies of it — so token consumption actually falls to about 550,000 per minute from 896,000. Batching wins on both axes here because the per-call overhead was mostly the repeated instruction. It does not always: batching a route whose instruction block is short and whose inputs are long saves almost nothing in tokens while making each failure take sixteen items with it, and it raises the reservation per request, which chapter 4 shows has its own cost. Recommending it in an error message is fine; assuming it as the general answer to a request-limit refusal is not.

The load test needs a second axis#

The sweep from chapter 1 held the request rate at 500 and varied the completion length, which was enough to prove that the request limiter could not see tokens. Proving that both limiters are necessary requires varying both, and the useful output is a grid whose cells name the binding constraint. Each cell is one run of the harness with the shadow limiters recording verdicts, and the interesting cells are the corners.

The bottom-left and top-right corners are the argument. At 200 requests per minute of 3,000-token drafts the token bucket refuses work while the request counter reads 5%, and at 8,000 requests per minute of one-word verdicts the request counter refuses work while the token bucket reads 24%. Delete either limiter and one of those corners becomes an outage that the remaining dashboard describes as healthy. The middle of the grid, where most traffic actually lives, is the region where neither binds and both look like overhead, which is why the corners are the cells to keep in the test suite.

What the request limit is not for#

Keeping a request limit is not a licence to tune it as though it were a spend control. Its number should come from the smallest per-request capacity in the path — the provider's RPM ceiling, the connection pool's sustainable rate, the retriever's ceiling divided by the fan-out factor — and not from a view about how much a tenant ought to be spending, which is the token bucket's question and, over a longer horizon, the budget's. A request limit set from a cost intuition will be tight enough to refuse cheap traffic that the platform could serve comfortably, and it will be the wrong control to reach for when a customer's bill is the actual problem.

It is also worth resisting the symmetry that suggests every unit deserves its own bucket. Some teams add a bucket per model, per route and per region on top of these three, and each one adds a round trip to the hot path and a way for a request to be refused for a reason nobody can explain to a customer at 3 a.m. Two rate limiters and a concurrency cap, all keyed the same way, cover the resources that are genuinely independent in an inference path.

That leaves the estimate the token reservation is built on, which chapter 2 filled in with a route's max_tokens and a promise to return to. Routes that set max_tokens are the easy case; the ones that leave it open, and the streaming ones where the client hangs up in the middle, decide whether a reservation is merely safe or actually useful.