Continuous batching

A decode step runs the model once to produce one token for one sequence. Doing that costs roughly 16 GFLOP of arithmetic on an 8-billion-parameter model and a read of all 16 GB of weights out of the GPU's high-bandwidth memory — HBM, the pool of memory attached to the card — which on an H100 takes about 38 microseconds and about 4.8 milliseconds respectively: more than 99% of the step is spent waiting on memory with the arithmetic units idle. Those weights are the same weights for every sequence in flight. If sixty-four requests are all sitting mid-generation, the server can stack their sixty-four single-token queries into one forward pass, read the 16 GB once, and produce sixty-four tokens from it. The memory traffic barely changes and the arithmetic — sixty-four times 16 GFLOP, so about 1 TFLOP — is still small relative to what the card can do in that window.

This chapter is about how far that goes and what it costs. The first half sweeps batch size against step time and finds where the sharing stops being nearly free, which is a specific point determined by when the batch's cached keys and values grow to rival the weights. The second half is about the scheduling design that delivers it in production, since the word "batch" carries an intuition from other systems that is wrong here in two ways that both show up in latency data.

The sweep, and where it stops paying#

The step time under batching has two terms that grow differently. Weight traffic is fixed at 16 GB per step. KV traffic scales with the batch, because every sequence's cached keys and values — the per-position state prefill left behind, which each decode step must read for its own attention — have to be read separately, at 128 KiB per token for this model, or 287 MiB for a sequence 2,300 tokens deep. Arithmetic scales with the batch too, from a base so low that it stays irrelevant for a long time.

From batch 1 to batch 128 the aggregate rises 41-fold while each individual request's generation rate falls by a factor of 3.1. That asymmetry is the entire economics of inference serving, and it is also the source of the confusion this topic exists to clear up, because both columns are legitimately called "tokens per second".

Every number in that table comes from two divisions, and doing them once makes the table something you can rebuild rather than consult. Take the batch-8 row. The bytes moved are the fixed 16 GB of weights plus 2.3 GB of cached keys and values — eight sequences at 287 MiB each — which is 18.3 GB. Step time follows from bytes because the step is memory-bound, and 7.8 ms is what this card takes to move that many. Per-request rate is then a second of milliseconds divided by the step time, since each sequence gets exactly one token per step: divide by 7.8 and the answer is 128 tokens a second. Aggregate is that rate times the number of sequences sharing the step, so eight at 128 is 1,026.

The memory-bound claim can be checked against the table rather than taken on trust, which is worth doing because it is the load-bearing assumption of the whole chapter. Divide any row's total bytes by its step time. Every row returns nearly the same answer, and that answer is the bandwidth this card actually achieves. A step whose duration is proportional to the bytes it moves, across a range where the arithmetic grows by a factor of 128, is a step whose duration is set by the bytes.

The relationship between the two extreme columns is one more division, and it dissolves the apparent paradox of a number rising forty-fold while another falls. Aggregate throughput is the batch size multiplied by the per-request rate. Over the sweep the batch rises by a factor of 128 and the per-request rate falls by a factor of 3.1, so the aggregate rises by 128 divided by 3.1, which is the 41-fold figure. Nothing was gained from nowhere. A large multiplier was bought at the price of a small divisor, and the trade is favourable only while the divisor stays small — which is exactly what the flattening at the bottom of the table is.

The gain flattens, and the table shows exactly where and why. Between batch 1 and 8, aggregate throughput rises 7.2× for an 11% increase in step time — nearly free, because the extra KV traffic is a rounding error against the fixed weight read. Between 64 and 128 it rises only 31%, because by then KV traffic is larger than the weight traffic and doubling the batch genuinely doubles most of the bytes moved. The regime where batching is nearly free lasts until the batch's KV reads approach the size of the weights, which for this model and this sequence length is around batch 60.

Two hard ceilings sit just past the bottom row. Memory is one: 16 GB of weights plus 36.8 GB of cache at batch 128 is 53 GB of an 80 GB card, so around 200 concurrent sequences of this depth is the physical limit, and longer contexts lower it proportionally. The other is that at batch 128 each reader is receiving 46 tokens per second, which is around 35 words per second — still faster than reading, but the margin is gone, and the next doubling would put the product below the rate at which people read.

Quantisation moves the whole table, and moves it in the direction the bottleneck points. Serving the same model with 8-bit weights halves the fixed per-step read from 16 GB to 8 GB, which nearly halves step time at small batch and, just as importantly, frees 8 GB for KV cache and so raises the batch ceiling. The gain is large precisely because the constraint is bytes moved rather than operations performed; the same change applied to a compute-bound prefill returns much less.

Sequence length interacts with all of this. Every number above assumed 2,300-token sequences; at 16,000 tokens each the KV read per sequence is 2 GB, so batch 32 alone moves 64 GB of cache per step and the whole picture shifts left by a factor of seven. Long-context workloads run at much smaller batches for the same hardware, and their aggregate throughput is correspondingly lower.

Continuous batching does not wait for a batch to fill#

The word "batch" carries the wrong intuition from every other system it appears in. Early inference servers did static batching: hold arriving requests until N of them accumulate or a timeout expires, run them together from prefill to the last token, then start the next batch. That design has two bad properties — an arriving request waits for the batch to fill before anything happens, and the whole batch runs until its longest member finishes, so a request generating 40 tokens occupies a slot for the 900 tokens its neighbour needs.

Decode is what makes that design unnecessary, and the reason is a property it has that other batched systems do not. In a training step, or a message consumer, the members of a batch are processed together because separating them would mean repeating fixed work, so the batch is a unit with a beginning and an end and anything joining it late has to wait for the next one. A decode batch is not a unit of that kind. Its members share exactly one forward pass, they share nothing whatsoever across passes, and the membership of the next pass is unconstrained by the membership of this one. Whether a particular sequence participates is decided afresh every step. Once that is clear, holding a request back until a batch fills stops looking like a scheduling policy and starts looking like a delay with nothing on the other side of it, which is why the design was abandoned.

Serving engines abandoned that for continuous batching, also called iteration-level scheduling, in which the scheduler makes a fresh decision before every single forward pass. A sequence that emitted its stop token at step 411 leaves the batch immediately and its slot is filled at step 412. A request that arrives at step 412 has its prefill scheduled at step 413. Nothing waits for a batch boundary, because there are no batch boundaries.

Two consequences of that design are the ones that show up in latency data.

An arriving request waits behind other people's prefill work for its first token. Request C above sits queued from t=12 because the scheduler is mid-iteration and because request D's prefill, once admitted, occupies iterations that C also needs. No batch is waiting to fill. A 2,000-token prefill costs about 80 ms of GPU time, so a server admitting twelve new requests per second is spending roughly a second of every second on prefill, and any request arriving into that has to be slotted between other prefills. This is why time to first token under load is dominated by queueing rather than by the request's own prefill.

Everyone's inter-token rate stretches as the batch grows, and it stretches for requests already in flight. A request admitted when the batch held eight sequences was receiving a token every 7.8 ms; when the batch reaches sixty-four it receives one every 14.1 ms, mid-response, with no change on its side. The text visibly slows down in the middle of the answer, which chapter 5 established is now something users can see.

Prefill also stalls decode when it is not chunked. A full 2,000-token prefill run as one iteration means every decoding sequence waits 80 ms for its next token instead of 14 ms — a visible stutter for everyone currently reading. Chunked prefill, introduced in chapter 3 as a setting that bends the TTFT slope, exists for exactly this: splitting that prefill into four 512-token pieces that each share an iteration with the decode work keeps the worst-case stall near 25 ms at the cost of a slightly later first token for the arriving request. The knobs that control this on a vLLM-style server are worth naming, since they are what a load test is actually varying:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --max-num-seqs 128 \              # ceiling on concurrent sequences in a batch
  --max-num-batched-tokens 2048 \   # tokens per iteration; caps prefill chunk size
  --enable-chunked-prefill \        # interleave prefill pieces with decode
  --gpu-memory-utilization 0.90     # how much of the card the KV cache may claim

Per-request latency, prefill, and adding replicas#

Batching does not speed up any individual request. Every row of the sweep after the first is slower per request than the row above it, and the fastest possible single response comes from a completely idle server. Batching converts spare memory bandwidth into aggregate capacity, and it pays for that capacity with per-request latency.

It does not help prefill in the same way. A 2,000-token prefill already presents the hardware with a large matrix and already saturates the arithmetic units, so combining two prefills into one iteration mostly just makes a bigger matrix multiply that takes proportionally longer. The scheduler batches prefills for efficiency of dispatch, and the gain is a few percent rather than a few hundred.

It is also a different mechanism from running more replicas. Adding a second GPU with its own copy of the weights doubles aggregate throughput and leaves per-request rate untouched, at twice the cost; raising the batch on one GPU raises aggregate throughput for free and degrades per-request rate. The two are complementary, and the reason batching comes first is that it is the one that costs nothing.

The frontier here is to stop making prefill and decode share a machine at all. Because one phase is compute-bound and the other bandwidth-bound, running them on separate pools — prefill on one set of GPUs, the KV cache shipped across the interconnect, which is the high-speed link between cards in a machine or a rack, decode on another — lets each pool be sized and tuned for its own bottleneck and removes prefill stalls from decode entirely. Disaggregated serving of this kind is in production at large providers and is described in Meta's Scaling LLM Inference write-up; it is out of scope here beyond knowing that the phase split is load-bearing enough that people build hardware topologies around it.

One weight read serving many sequences is what makes serving a language model affordable, and the price of that arrangement is paid in per-request rate rather than in money. The sweep says how much: nearly nothing up to the point where cached keys and values rival the weights in bytes moved, then a steepening cost per additional sequence, then a memory wall. Continuous batching is what turns that arithmetic into a running system, and its two side effects — a first token that waits behind strangers' prefill, and a generation rate that changes mid-response — are not defects to be fixed but the shape of the trade.

The sweep in this chapter was measured by controlling the batch size directly. In production you do not set the batch size; you set a concurrency limit and traffic decides. The next chapter runs that experiment — two concurrency levels, everything else fixed — and looks at what happens to throughput, queue time, first token and inter-token latency at the same time.