Tuning the concurrency bound

The bound derived from a published limit and an observed latency is a safe starting point rather than a measurement of what the account will serve. Providers over-provision, enforce over windows they do not document, and change what they serve by time of day, so the real ceiling is usually somewhat above the arithmetic and occasionally below it. Finding where it sits means running the job at several settings and reading what comes out.

The procedure that suggests itself is to raise concurrency one step at a time until the first 429 — the status a provider returns when it refuses a request for exceeding an allowance — and then back off one step. That terminates, it produces a number, and the number is close to worthless. This chapter is about why, what a rejection actually costs, and what to measure instead.

The first rejection is an artifact of one unlucky second#

Providers meter over short windows, and traffic is never perfectly smooth even from a batch job with a fixed concurrency bound, because latency varies per call and completions therefore cluster. A run at 420 requests per minute against a 500-per-minute limit will, somewhere in a five-hour window, produce a second in which nine calls complete and nine new ones are issued instead of the expected seven. If the provider's window is short enough to see that, one request gets refused.

Stopping at the setting just below the one that produced that single rejection means the configuration is determined by the worst-behaved second in five hours rather than by anything about sustained capacity. Run the same procedure on a different night and it hands you a different answer, because a different second will have been the unlucky one. The number it lands on is typically 20–30% below what the account will serve continuously, which is the same waste the procedure was meant to eliminate, arrived at with more ceremony.

The deeper problem is the variable being optimised. Errors are being minimised; the thing anyone actually wants is items completed per minute, and those two move together only up to a point. Past that point the error rate keeps climbing while completed work flattens and then falls, and a procedure watching errors cannot distinguish the two regimes.

Zero rejections is the belief this chapter is written against, and it is tempting for a reason that is sound almost everywhere else in engineering: an error counter sitting at zero usually means a system is behaving. What makes provider rejections different is that the counter is not measuring failures of your system at all. It is measuring a negotiation with somebody else's admission control, and the only way to hold it at zero across a five-hour run is to stay far enough below the ceiling that no unlucky second can reach it. The reading everybody wants is therefore the reading that guarantees the account is being under-used, and the number itself says nothing about by how much.

What a rejection actually costs#

The reason a small rejection rate is affordable is that a refused request is a fast negative acknowledgement rather than a failed unit of work. The provider refuses it at the edge before any model is invoked, so it returns in something like 30–50 ms instead of 800 ms, it consumes no input tokens, it produces no output tokens, and it is not billed in currency on any major provider.

That changes the arithmetic of a slot. A successful call occupies a concurrency slot for 800 ms; a rejected one occupies it for 35 ms, plus whatever backoff you serve — and the backoff belongs outside the slot, so the slot is free during it. Ten per cent of requests being rejected therefore costs about 0.4% of the stage's service capacity, not 10%. What a rejection does cost is a retry, which means the item finishes later, and that shows up in per-item p99 rather than in throughput.

That figure comes from pricing a slot in time rather than in requests. Take a hundred calls at a 10% rejection rate: ninety occupy a slot for 800 ms each and ten occupy one for 35 ms each. The ninety account for seventy-two seconds of slot time and the ten for about a third of a second, so the refused calls consume well under half a per cent of the slot time the stage spends — the 0.4% above — rather than the 10% the rejection rate itself suggests. The backoff does not enter that calculation, and the reason is a property of the code rather than of the provider: the sleep in the loop below is served after the async with CALL_GATE block has exited, so a coroutine waiting out a backoff is holding nothing. Move that sleep inside the gate and a 10% rejection rate does cost something like 10%, because each refused call now parks a slot for the whole backoff.

The one currency a rejection does spend is the request allowance itself, since the provider counts a refused request against your rate budget. At a 3% rejection rate that is 3% of the quota spent on nothing, which is affordable; at 50% it means half the account's serving capacity is being consumed by requests that produce no work, and that is the mechanism behind the regression at the bottom of the sweep below rather than anything subtle.

Measuring the right variable takes one counter that increments on a durable result:

COMPLETED = Counter("items_completed_total", "Items written to the sink")
REJECTED = Counter("provider_429_total", "Requests refused by the provider")
ATTEMPTS = Counter("provider_attempts_total", "Requests issued, including retries")


async def process(item: Item) -> None:
    for attempt in range(5):
        async with CALL_GATE:
            ATTEMPTS.inc()
            response = await client.post(PROVIDER, json=prompt_for(item))
        if response.status_code == 429:
            REJECTED.inc()
            await asyncio.sleep(backoff(attempt, response.headers))   # slot released
            continue
        response.raise_for_status()
        await sink.write(item.id, parse(response))
        COMPLETED.inc()
        return
    await sink.mark_failed(item.id, "rate limited after 5 attempts")

items_completed_total is the rate to maximise. provider_attempts_total / items_completed_total is the waste ratio, which at 1.0 means no retries and at 2.0 means half of all requests are being refused. Both are needed, because the first alone will happily accept a configuration that doubles the request volume for a 2% throughput gain, and on an account shared with other traffic that is a bad trade even though it is billed at zero.

The sweep, and where the curve goes flat#

Running the job for ten minutes at each setting on a representative night produces a shape rather than a threshold. Ten minutes matters because it spans many metering windows; a two-minute sample can miss the burst behaviour entirely and will report zero rejections at settings that reject steadily.

Every column in that table is derived from the two to its left, which is what makes it possible to check a sweep somebody else ran. Offered requests per minute is the in-flight bound divided by the 800 ms service time and multiplied by 60, so four in flight offers 300 a minute and fourteen offers 1,050. Completed per minute is the offered rate less the fraction refused: 600 offered with 12% refused completes 528, and 750 offered with 29% refused completes 533. Attempts per item is the reciprocal of the success rate, because a refused item tries again and can be refused again, and the expected number of tries is a geometric series summing to one over the chance of succeeding on any single try — at a 51% rejection rate barely half of attempts succeed, and the reciprocal of that is 2.05. Wall clock is then 100k items divided by the completed rate: 300 a minute gives 5 h 33 m, and 528 a minute gives 3 h 09 m.

Three things in that table are worth naming. The account serves about 530 requests per minute sustained, slightly above its published 500, which is common and is exactly why the published number is a floor for planning rather than a measurement. Completed work saturates between eight and ten in flight, and every setting past that buys nothing while doubling the traffic. And the last row is a genuine regression: at fourteen in flight the retry volume is displacing first attempts, so the job completes fewer items per minute than at eight while issuing twice as many requests.

Running the sweep without breaking anything#

Ten minutes per step and six steps is an hour of a nightly window spent measuring rather than working, which is affordable once and annoying to repeat. Two practical constraints shape when it can happen at all.

The first is that the ceiling being measured is not a property of your account alone. Providers over-provision during quiet hours and tighten when their own fleet is loaded, so a sweep run at 03:00 on a Sunday can land 30% above what the same account serves at 14:00 on a Tuesday. A number measured on the quietest night of the week and then applied on a Monday morning produces a job that is 30% over the real ceiling, spends a third of its requests on rejections, and completes more slowly than the conservative setting it replaced. Measuring in the window the job actually runs in is the only version of this that transfers.

The second is that the job under test shares an account with production traffic, and a sweep that pushes to the ceiling is deliberately generating rejections that an interactive endpoint may absorb instead. Running the sweep against a separate key sized to a known fraction of the account is the clean approach where the provider will issue one; where it will not, the sweep belongs in the window with the least interactive traffic, with the interactive latency condition below treated as a hard abort rather than as a stopping rule. A useful check on the resulting number is that it should be reproducible on a second night to within roughly one step. A sweep that says eight one night and thirteen the next is measuring noise, usually because the steps were too short to span enough metering windows, and the honest response is to lengthen the step rather than to average the two answers.

The procedure is worth committing to text, in the runbook next to the job, because the person raising concurrency in eight months will not be the person who ran the sweep:

Raise in-flight by 1. Hold for 10 minutes. Continue only while ALL of:
  - items_completed_per_minute rose by more than 2% over the previous step
  - attempts_per_item < 1.3
  - retry_exhausted_total == 0
  - interactive p95 on the same account < 1.5 s
Stop at the first step that fails any condition; keep the previous step's value.
Re-run when the prompt template, the model, or the account quota changes.

The interactive condition is the one that gets left out and matters most. A batch and a product's chat endpoint draw on the same account allowance, so pushing the batch to the ceiling means every rejection the batch does not receive is one a user receives instead. Where both classes of traffic share a key, the batch is the one that should yield, and the mechanics of making it yield are chapter 7's subject.

An adaptive controller can run this loop continuously instead of once, which is how to handle an account whose real ceiling moves through the day.

Additive increase with multiplicative decrease is the same control law TCP congestion control uses, and it has the same property: it never knows the ceiling, it oscillates around it, and the oscillation is small enough to be cheaper than being permanently wrong.

Why that particular pair of moves converges is worth one step, because the name is usually offered in place of the argument. Increasing by one after each clean window means the controller approaches an unknown ceiling in small steps, so the overshoot when it finally crosses is small whatever the ceiling turns out to be. Cutting by a constant fraction on a rejection means the retreat is proportional to how far it had climbed, so a controller that has wandered a long way above the ceiling returns quickly while one that has only just crossed barely moves. The two together produce a sawtooth sitting just under a number nobody measured, and its average is close to that number precisely because the climbs are slow and the drops are sharp. Reverse the pair — multiply on the way up, subtract on the way down — and the same loop overshoots hard and recovers slowly, which is the shape of a retry storm rather than of a controller.

The cost of running that loop continuously is a system whose concurrency is not a number anyone can read out of a config file, which makes an incident harder to reason about, so it earns its place on long-running shared workloads and not on a job that runs once a night against a dedicated key.

Where a small rejection rate is the wrong target#

The case for tolerating rejections rests on their being cheap, and that premise is worth checking rather than assuming. Some gateways bill refused requests, some enterprise contracts treat sustained rejection as a violation with account-level consequences, and providers metering a shared organisation pool may penalise the whole organisation for one team's aggression. A rejection that arrives after the request body has been uploaded also costs real bandwidth and time, which is negligible for a 250-token prompt and much less so for a request carrying a 2 MB PDF.

Interactive traffic changes the calculation completely. A user waiting on a chat response experiences a rejection plus a two-second backoff as a two-second stall, so the operating point for a request path is the one where rejections are genuinely rare, and the idle quota that leaves is the price of the latency guarantee. The argument in this chapter is specifically about bulk work, where an item finishing four seconds later than it might have is invisible.

The operating point for a batch, then, is the setting at which completed work per minute stops improving, and finding it means accepting a rejection rate that a zero-error target would have called a failure. Both numbers have to move together: the bound and the alert that watches it, because the second is what decides whether the first survives its first night on call.

Everything so far has assumed that a shortfall between offered and achieved throughput is the provider's doing. The next chapter takes a batch that runs slowly with no rejections at all, where the provider is not the constraint and the extra 250 milliseconds per item is spent before the request ever leaves the machine.