Capacity planning for model traffic

Read these first

• Rate limiting and quotas for LLM traffic
• Cloud model platforms

What you will know

• Plan capacity against TPM and RPM rather than QPS

  A request with a long context and large completion can consume far more quota than a short request despite counting as one request.

• Handle a burst without dropping requests

  Admission control must reserve for prompt and worst-case completion tokens, or synchronized retries can exhaust the budget faster than steady traffic.

• Decide whether provisioned throughput is worth it, with numbers

  Compare reserved capacity cost with measured demand across average, peak, and idle periods, including the cost of unused allocation and overflow.

• Plan multi-region for latency or for residency, and say which

  Latency routing permits broader failover choices, while residency constraints can rule out otherwise useful regions and complicate capacity during an outage.

• Reproduce quota-ceiling behavior and verify the fallback path

  Providers may reject requests with rate-limit errors or delayed recovery signals, and indiscriminate retries can turn a limit event into an outage.

Chapters

1. 1

   A request is one unit of RPM and thousands of units of TPM

   A summariser runs happily at 300 requests/min against a 500 RPM and 300k TPM account, then starts returning 429s at 380 because a launch brought longer documents and the binding counter was never the request one. Size the workload in tokens per minute, count the completion against the same window as the prompt, and be able to say in one sentence why '50 QPS' is not a capacity plan.

   Read chapter
2. 2

   Admission control must reserve the completion nobody has seen yet

   Reserving prompt tokens before dispatch looks like admission control and still yields provider 429s under burst, because the completion is charged against the same per-minute window and its length is unknown at the moment you must decide. Reserve the worst case the request could produce rather than the typical output, then reconcile against actual usage once the response lands.

   Read chapter
3. 3

   Queueing a request the user has abandoned is not serving it

   A promotion brings ten times normal traffic for forty minutes against a quota that cannot grow, and 'queue everything, drop nothing' satisfies the requirement while breaking the product: the queue grows for the whole window, drains long after it, and late arrivals wait minutes for an answer to a question they left. Covers deadline-aware admission, deliberate shedding, and what the user is actually shown when it fires.

   Read chapter
4. 4

   Fixed backoff returns every throttled client at the same instant

   A ninety-second provider slowdown pushes you over quota and every client retries three times at a flat 100ms interval. Everything throttled in the same moment waits the same interval, so offered load multiplies exactly while capacity is scarce and each round lands louder than the one before. Jitter, retry budgets, and a breaker are what stop the limit event outliving the slowdown that started it.

   Read chapter
5. 5

   Reserved throughput is billed on allocation, not on usage

   Units at $2,000 a month per 100k TPM against $3 per million on demand: a unit held for a month is roughly 4.3 billion tokens of capacity, so it pays for itself somewhere above 15% average utilisation. Buying nine units to blanket a p99 you touch rarely idles most of them overnight and converts a variable bill into a fixed one. Work the break-even, then decide what serves the overflow.

   Read chapter
6. 6

   Latency routing and data residency imply different failover rules

   'Three regions for latency and for GDPR residency' states two goals whose failover behaviour conflicts: when the EU region degrades, latency routing sends its traffic to the nearest healthy region, which residency rules out entirely. Decide which goal each region serves, agree the degraded mode in the contract before the incident call, and know what you say when the CEO asks why headroom is sitting unused.

   Read chapter
7. 7

   Crossing the ceiling on purpose is the only way to know what happens

   A fallback that has never carried production traffic is an untested branch of the capacity plan. Drive load at a staging account whose quota is deliberately small until you genuinely cross the limit, then assert the whole path: the rejection shape the provider returns, what happens to requests already streaming, and whether the second provider's own limits can absorb what you divert into it.

   Read chapter