Concurrency, throughput and tail latency

The concurrency cap is the ceiling a serving engine puts on how many sequences it will keep in flight at once, set as a configuration value and enforced by making further requests wait. It is the practical control over batch size, because in production nobody sets a batch size directly: traffic arrives at whatever rate it arrives, and the cap decides how much of it the scheduler is allowed to admit before the rest queues. Raising it is the standard response to a cost target, since a deeper batch amortises the same weight read over more sequences and lowers cost per million output tokens.

What the cap also does is move the server's operating point along a curve where waiting time rises with utilisation, and rises fastest at the end. The numbers below come from an 8B model on one H100 with 2,000-token prompts and 300-token outputs, driven by the open-loop generator of chapter 4.

The same traffic at two concurrency levels#

Hold everything else fixed — same model, same GPU, the same Poisson arrival process warmed to steady state — and run the generator against two configurations.

Every number in that table is a consequence of the mechanism in chapter 6, where one weight read serves every sequence in a forward pass. Requests wait longer for admission into the running batch, because the scheduler has more work competing for each iteration and more prefills to interleave, which is the queue row and most of the TTFT row. Once admitted they share decode capacity with 88 neighbours instead of 31, so each forward pass takes 16.8 ms instead of 10.3 ms and every sequence in the batch receives its next token at that slower rate. The card is doing more total work per second, and each individual is getting a smaller share of it.

For the reader, 60 tokens per second is about 45 words per second, which is still comfortably above reading speed, so the sustained rate is not the part that hurts. The 1,890 ms p95 TTFT is. Nearly one interaction in twenty now spends close to two seconds showing nothing after the user hits enter, against a third of a second before, and that is the interval a person experiences as the product thinking about whether to answer.

Why the utilisation curve has a knee#

The tail moved far more than the median, and that ratio is not specific to inference. Any queueing system where arrivals are irregular has a waiting time that rises with utilisation and rises without bound as utilisation approaches one. Utilisation here is the fraction of the server's capacity the offered traffic is consuming, so 0.9 means the work arriving takes nine tenths of the time available to do it, and the remaining tenth is the only slack the system has for absorbing a burst. Pushing a server from 60% busy to 90% busy roughly quadruples the expected wait, and from 90% to 97% roughly triples it again. Raising the concurrency cap moved the operating point along that curve, so the throughput improvement and the tail-latency degradation are two readings of the same movement.

Why the wait rises at all, and why it rises so much faster near the end, is worth building rather than citing. Arrivals are irregular: at a mean of fourteen requests a second, some seconds bring more than fourteen and some bring fewer, and a second that brings more leaves a backlog behind it. That backlog drains at whatever rate the server has left over once it has served the steady flow, and the leftover is precisely the unused fraction of capacity. At sixty per cent utilisation, four tenths of the machine is free to drain a burst and the burst clears quickly. At ninety-seven per cent, three hundredths of the machine is available for the same job, so the same burst takes many times longer to clear — and while it is clearing, new arrivals land on top of it. As utilisation approaches one the spare capacity approaches zero and the drain time grows without limit. Nothing in that argument mentions language models. It is the same argument that governs a checkout queue.

The same reasoning says why the tail moved further than the median. At a moderate rate most requests arrive when no backlog exists, so they are admitted immediately and the median is describing a server that happened to be free. The unlucky requests are the ones that arrive while a burst is draining, and each inherits whatever is left of the drain. Raising the cap increased both how often a backlog exists and how long each one lasts, but the median only responds once backlogs become common, while the tail responds as soon as they become long. A median moving by two and a half against a p95 moving by five and a half is that difference, measured.

Sweeping the offered rate against a fixed cap of 96 shows the knee explicitly:

Between 6 and 13 requests per second, throughput more than doubles for an extra 99 ms of median TTFT, which is the region where batching is close to free and where a deployment should run. Past 15 the achieved rate stops following the offered rate, which means the queue is growing without bound and the only thing still rising is how long people wait. The last row's aggregate throughput is 0.7% better than the row above it, and its p95 TTFT is 3.4× worse.

Little's law makes the relationship between the columns concrete enough to plan with: the mean number of requests in the system equals the arrival rate multiplied by the mean time in the system. At 14.5 requests per second and 5.29 seconds each, 77 requests are resident on average, which is consistent with the measured mean batch of 88 once queued-but-not-admitted requests are included. Running that calculation before a config change tells you roughly what batch size a given traffic level will produce, and therefore which row of chapter 6's sweep you are about to land on.

The law is an accounting identity rather than a model of anything, which is why it can be applied to a system nobody has characterised. Watch the server over a long stretch. Every request passing through contributes its own residency — the seconds between arriving and leaving — to the total time-in-system accumulated across that stretch. Requests arrive at some rate, and each one contributes its own residency, so for every second of wall clock the system accumulates the arrival rate multiplied by the mean residency, in units of request-seconds per second. Request-seconds per second is requests, which is the average number resident. No assumption about the arrival distribution, the service order, or what the server does internally appears anywhere in that argument, which is why the identity survives on a queue you have not modelled.

Both lines of the code below are that identity used in opposite directions. Forward: 14.5 requests a second, each resident for 5.29 seconds, gives 76.7 resident on average, which reconciles with the measured mean batch of 88 once the requests that are queued but not yet admitted are counted in. Backward: fix the residency and the batch depth you are willing to run at, and the arrival rate one replica can absorb is the second divided by the first.

# what concurrency will this traffic actually produce?
arrival_rate = 14.5          # requests/s
mean_latency = 5.29          # s, measured
print(arrival_rate * mean_latency)      # 76.7 requests resident

# and what does the cap have to be to keep decode at the batch-32 rate?
target_batch = 32
print(target_batch / mean_latency)      # 6.0 req/s per replica

The second calculation is the one that turns a latency target into a capacity number. Holding the batch near 32 at this request shape supports about six requests per second per replica, so 14.5 requests per second with the better tail needs three replicas rather than one — which is the real price of the latency users want, and a number that a cost model optimising throughput per GPU in isolation never computes.

One assumption inside that second calculation is worth pulling out, because it makes the answer conservative rather than exact. The 5.29 seconds being divided into was measured at cap 96, where each request shares every decode step with 88 neighbours. Held near a batch of 32, the same request finishes sooner: the table's own cap-32 column puts total p50 at 3.19 s, and 32 divided by 3.19 lands close to the 10.3 completed requests a second that column actually measured. Which divisor is right depends on the latency that the target batch would itself produce, so the honest reading of the calculation is a range — somewhere between two and three replicas for this traffic — narrowing as soon as you measure at the setting you intend to run. It is still worth doing before measuring anything, because it puts the answer in the right order of magnitude before anyone buys hardware, and because the direction of its error is known rather than guessed.

Why a mean hides what a p95 shows#

The dashboards that showed no regression were not lying about their own contents. They were reporting quantities that do not describe the experience, and three properties of those quantities are worth being explicit about.

Aggregate tokens per second is a server-side quantity that no user has. It rises when more people are served concurrently, which is exactly the condition under which each of them is served more slowly, so it moves in the opposite direction to per-request experience across most of its useful range. It belongs on a capacity dashboard and is actively misleading on a latency one.

A mean over TTFT absorbs a tail that a p95 exposes. The TTFT distribution at cap 96 is heavily skewed — most requests are admitted immediately at 130 ms, and a minority wait more than a second behind a run of prefills — so the mean lands around 390 ms while the p95 is 1,890 ms. A dashboard showing the mean records the change as "TTFT rose 280 ms", which sounds survivable, when the actual experience for one interaction in twenty got five times worse.

Total latency conflates speed with length, which chapter 3 established when it separated the prompt and output slopes, and which becomes dangerous when a metric is used as a target rather than a description. The p50 total in the table rose, so this particular change would have been caught by watching it; a change that also shortened answers, or that increased the share of traffic taking a short fast path, could raise TTFT and lower p50 total at the same time. That combination is common enough that the next chapter treats it as the default case rather than an edge one.

Batch 1, admission control, and prompt length#

None of this is an argument for running at batch 1. The first two rows of the sweep in chapter 6 gave a 7× throughput improvement for an 11% cost in step time, and refusing that trade would multiply infrastructure cost by roughly seven for a latency difference no user could detect. Almost all of batching's value is available in the flat part of the curve, and the engineering question is where to stop rather than whether to start.

Bigger is therefore the right default and unbounded is not, which sounds obvious and is precisely the decision the cost model got wrong. A model asked to minimise dollars per million tokens contains no term for a person waiting, so its optimum sits at the far end of the curve, where the last few per cent of throughput are bought with whole seconds of tail. Any parameter with a different quantity written on each side needs both quantities in the objective it is optimised against, and the second one exists only if somebody has written it down.

The cap is also not the only control, and usually not the best one. Admission control is the better instrument: rather than letting the queue absorb overload, reject or shed requests beyond a defined concurrency, so that the requests you accept keep their latency contract and the ones you refuse fail fast enough to retry elsewhere. Separating traffic classes helps for the same reason — a bulk summarisation job and an interactive chat share nothing but the model, and giving them separate queues and separate caps lets the bulk queue run deep while the interactive one stays shallow.

And the curve is not a fixed property of the workload. Every row of both tables assumed 2,000-token prompts, and prompt length is one of the strongest levers on how much prefill work the scheduler has to interleave. Trimming a template from 3,400 tokens to 1,200, or getting a prefix cache hit on the stable part of it, reduces per-request prefill by two thirds and pushes the knee of the curve to the right — the same hardware then sustains a materially higher request rate at the same tail. Chapter 3 showed prompt length barely touches an idle server's latency; under load it is one of the things that touches it most.

Concurrency is a dial with two labels, and turning it always turns both. On one side it reads cost per million tokens, on the other it reads how long one person waits before anything appears, and the second one degrades at the tail long before it degrades in the middle. The instrument that makes the exchange visible is a load test at two settings with percentiles rather than means, and the arithmetic that makes it plannable is Little's law, which converts a target batch depth into a request rate per replica and therefore into a replica count.

What the load test cannot do is decide the trade. Somebody has to state what the product owes its users, in numbers, so that the next config change can be evaluated against something firmer than a support-ticket count nine days later. Writing that objective — which metric, at which percentile, at which value, and why — is the next chapter.