Mid · Core path · 9 chapters
Prefill vs decode
TTFT vs inter-token latency; throughput vs latency
Ignoring the split produces misleading latency wins: a system can improve throughput while making interactive responses feel slower or hiding queueing behind a good average.
Read these first
What you will know
Run a benchmark that separates prefill time from decode time for fixed prompt and output lengths
Warmup, scheduler queues, batching, and cold-start effects can otherwise be mistaken for model execution cost.
Run the same generation with and without streaming, and measure TTFT, total latency, and inter-token latency
Streaming changes when useful output becomes visible, while buffering, network flushes, and client rendering can still delay what the user actually sees.
Measure latency while independently doubling prompt length and output length, then account for the observed changes
Prompt tokens primarily increase one-time prefill work; generated tokens extend the serial decode path, though cache state and batching can alter the slope.
Load-test two concurrency levels and compare throughput, queue time, TTFT, and inter-token latency
Higher utilization may raise tokens per second while worsening tail latency through queueing, larger batches, or decode contention.
Choose which metric to put in an SLO for a chat feature, and defend it
For chat, time to first token is what users judge. For a batch job, it is irrelevant and throughput is everything.
Chapters
- 1
Prefill and decode
Serving one request to a language model is two operations rather than one. Prefill pushes every prompt position through the model in a single largely parallel pass and runs out of arithmetic throughput; decode then runs one forward pass per generated token, each waiting on the last, and runs out of memory bandwidth. This chapter derives both bottlenecks from the FLOPs and the bytes, names the two latency metrics each phase produces, and explains why the same GPU can serve 21,000 tokens a second in one phase and 143 in the other.
- 2
The KV cache and prefix caching
Prefill leaves per-position keys and values behind, so each decode step is one incremental pass for the newest token against that stored state rather than a fresh run over the entire prefix. This chapter computes the cache's size and shows why it, rather than the weights, sets a GPU's concurrency limit; explains why doubling a prompt raises time to first token while leaving inter-token latency flat; covers prefix caching and the exact-prefix rule that decides whether you get a hit; and explains why a first measurement against a fresh server is unrepresentative in both directions at once.
- 3
Attributing latency to a phase
Doubling a prompt with output length fixed raises time to first token and leaves inter-token latency flat; doubling the output does the reverse. This chapter builds the four-cell grid that measures both slopes, covers the harness details that quietly invalidate it — unpinned output length, repetitive filler, warmup — and reads off the exchange rate between a prompt token and an output token. It then shows where prefix caching, long context, load and chunked prefill bend the slopes enough that the grid has to be re-run.
- 4
Benchmarking a serving endpoint
A single timed request runs against a cold cache, an empty scheduler queue and no batch neighbours, which is the one condition production never runs in. This chapter breaks a stopwatch reading into its five components, shows where queue time and model execution can be read separately from a serving engine's own metrics, and sets out the harness a quotable result needs: warmup, pinned prompt and output lengths, per-request records, and an open-loop generator that does not hide the queueing it was built to find.
- 5
Streaming responses and buffering
Run the same generation with and without streaming and the first useful output moves from 2,190 ms to 96 ms while total time gets 25 ms worse. Streaming changes where in a pipeline a token becomes visible, not how fast the model produces one, so every component between the model and the eye — a buffering reverse proxy, a compression layer, a rendering loop — can defeat it silently. This chapter measures the trade, traces the five places a token can be held, and shows what to record at the client, including the stall metric that percentiles smooth away.
- 6
Continuous batching
A decode step spends more than 99% of its time moving weights out of memory, and those weights are the same for every sequence in flight, so one read can serve sixty-four requests at once. This chapter sweeps batch size against step time to show where that stops being nearly free, then explains iteration-level scheduling: modern engines admit requests before every forward pass rather than waiting for a batch to fill, which is why an arriving request queues behind other people's prefill work and why everyone's token rate stretches mid-response as the batch grows.
- 7
Concurrency, throughput and tail latency
Raising a serving concurrency cap from 32 to 96 buys 40% aggregate throughput and costs a p95 time to first token that goes from 340 ms to 1,890 ms. This chapter runs that load test, sweeps the offered rate to locate the knee where queueing stops being free, applies Little's law to turn a latency target into a replica count, and shows why aggregate tokens per second and a mean latency both report the change as an improvement.
- 8
Chat latency SLOs
Total response time on a streaming endpoint tracks answer length as much as speed, so it can be met while users complain, it can fall because answers got shorter, and it can be improved by making answers worse. This chapter states the objective a chat feature should carry instead — a p95 on time to first token, a floor on sustained inter-token rate, and a separate stall metric — defends each number against human-factors evidence and against what the hardware can reach, and shows which remedy each kind of breach points at.
- 9
Choosing an endpoint by workload
One endpoint returns its first token in 180 ms and generates at 45 tokens per second; another takes 600 ms and generates at 120. The crossover is at 31 output tokens, so the second wins nearly every single response and loses an agent loop of eleven short calls. This chapter works both sides — a 400,000-document nightly job where startup cost is 1.3% of runtime, and a voice agent where 420 ms of extra silence lands on every conversational turn — and ends on the practice the whole topic reduces to.
External resources
- Meta-Metrics and Best Practices for System-Level Inference Performance Benchmarkingarxiv.org
Defines TTFT, ITL and throughput so you benchmark the right thing.
- Prefill vs Decode: LLM Inference Optimizationoutcomeschool.com
This is a substantive tutorial explaining prefill and decode mechanics, KV-cache flow, compute versus memory bottlenecks, latency metrics, and phase-specific optimization techniques.
- Prefill vs Decode in LLM Inference: How They Work & Why They Matterweka.io
Compute-bound prefill vs memory-bound decode, and what each costs.
- Scaling LLM Inference: Innovations in Tensor Parallelism, Context Parallelism, and Expert Parallelismengineering.fb.com
How tensor, context and expert parallelism address prefill/decode bottlenecks and latency tradeoffs.
- KV Cache Explained: Speed Up LLM Inference with Prefill and DecodeReady Tensor
Visual split of the two phases and where TTFT comes from.
- LLM Inference Engines: vLLM, KV Cache, Paged attention and Continuous Batching.The Cef Experience
Shows how batching trades TTFT against tokens per second.