Attributing latency to a phase

Two quantities describe how a generation feels. Time to first token (TTFT) is the interval from sending a request to receiving the first piece of output, and it is dominated by prefill — the single pass in which the model processes every position of the prompt at once. Inter-token latency (ITL) is the gap between consecutive output tokens during decode, the strictly serial phase that runs one forward pass per generated token. Total response time is a composition of the two, TTFT + (output_tokens − 1) × ITL, so any change in the total can come from either speed term or from the token count, and the total on its own never says which.

This chapter measures both slopes on one machine, reads off the exchange rate between a prompt token and an output token, and names the four conditions under which that rate stops transferring.

The grid that attributes a regression#

The general form of the measurement is a grid: hold one length fixed and vary the other, then swap. Four points is enough to read the slopes, and a fifth doubling in each direction makes the linearity visible.

Double the prompt, hold the output fixed, and whatever moves belongs to prefill. That is the whole idea, and the reason it is legitimate rather than merely convenient sits in the composition formula. That formula has three quantities in it, and each of the two lengths touches a different one. TTFT is the prefill term, and it moves with prompt length. The output count multiplies the decode term, and it moves with output length. The per-token gap is decode speed, and it is the single quantity that could in principle answer to both, because a decode step reads a KV cache whose size grew with the prompt. Where that dependence is negligible the two effects simply add, each can be measured with the other held still, and the attribution is exact. Where it is not negligible the two effects interact, and no amount of careful measurement of one at a time will predict what they do together. The flat ITL column in the results below is what tells you which of those two situations you are in, which is why the column that never changes is still worth recording.

import time, httpx

def measure(prompt_tokens: int, max_tokens: int) -> dict:
    body = {
        "model": "llama-3.1-8b-instruct",
        "messages": [{"role": "user", "content": filler(prompt_tokens)}],
        "max_tokens": max_tokens,
        "min_tokens": max_tokens,   # force a fixed output length
        "stream": True,
        "temperature": 0,
    }
    stamps = []
    t0 = time.perf_counter()
    with httpx.stream("POST", URL, json=body, headers=HEAD, timeout=120) as r:
        for line in r.iter_lines():
            if line.startswith("data: ") and '"content"' in line:
                stamps.append(time.perf_counter())
    ttft = stamps[0] - t0
    gaps = [b - a for a, b in zip(stamps, stamps[1:])]
    return {
        "ttft_ms": ttft * 1e3,
        "itl_ms_p50": sorted(gaps)[len(gaps) // 2] * 1e3,
        "total_s": stamps[-1] - t0,
        "output_tokens": len(stamps),
    }

Three details in that harness matter more than they look. Output length is pinned rather than left to the model, because a sampler that decides to stop early turns a latency experiment into a verbosity experiment — min_tokens where the server supports it, otherwise a prompt that reliably produces a fixed-length answer, and always recording the actual token count so you can check. Temperature is zero so that the same prompt produces the same length and the same work every run. And the per-token gaps are summarised by their median — the p50, the value half the samples fall below, in the percentile notation the rest of this topic uses — rather than their mean, because a single scheduler hiccup in the middle of a generation drags a mean far more than it drags the experience.

The filler that pads a prompt to a target length is the part most likely to invalidate the whole grid. Repeating one paragraph a hundred times is the obvious way to build a 4,000-token prompt and the worst one, because identical blocks are exactly what a prefix cache is built to collapse, and because a model given a highly repetitive prompt often produces a much shorter answer. Sampling real documents from a body of text you actually serve and truncating them to the target token count avoids both problems and has the additional benefit of exercising the tokeniser on text of the shape your users send. Whatever you use, count the tokens rather than the characters, since the mapping between them varies enough by language and content that a "4,000-token" prompt built by character count can easily be 5,200.

Each cell should be a distribution rather than a reading. Twenty runs per cell, with the first five discarded as warmup, takes a few minutes at these sizes and gives you enough to see that the variance is small — on an idle server, TTFT at a fixed prompt length typically holds within ±3 ms, and a cell whose spread is much wider than that is telling you something is wrong with the setup before it tells you anything about the model. Record the raw per-run values rather than only their summary, because the question "did this cell get slower, or did one run in it stall for 400 ms" comes up constantly and cannot be answered from a median.

Run against an otherwise idle server, warmed with a few dozen requests of the same shape so that its caches have reached steady state, the grid comes back like this.

Reading down the first three rows: prompt tokens quadruple, TTFT goes up 3.6×, inter-token latency is unchanged to within a tenth of a millisecond, and total time rises by 14%. Reading rows one, four and five: prompt is constant, TTFT is constant, and total time tracks output tokens almost exactly at 7 ms each. Each row's total also reconstructs from its own parts — 52 ms plus 599 × 7.1 ms is 4.30 s — which is a useful consistency check, since a total that does not equal TTFT + (n−1) × ITL means something is happening between the server and your stopwatch, and that is the next chapter's subject.

The slope in the first column is 45 microseconds per prompt token, or about 22,000 prompt tokens per second, which is the prefill rate from chapter 1, where one pass over the whole prompt saturates the arithmetic units, arriving by a completely different route. The slope in the last column is 7 ms per output token. The ratio between them is 155:1, and that number is the practical content of this topic: one output token costs as much wall clock as 155 prompt tokens. A prompt template that seems bloated at 3,400 tokens costs less time than 22 tokens of extra answer.

Both of those slopes come off the table by subtraction and division, and reading them out by hand takes about a minute. Take the first and third rows: prompt tokens go from 1,000 to 4,000, and TTFT goes from 52 ms to 186 ms. Divide the rise in TTFT by the three thousand extra prompt tokens and each prompt token is worth about 45 microseconds. Then check that figure against a row it was not computed from — rows one and two differ by a thousand prompt tokens and by a little over forty milliseconds of TTFT, which gives nearly the same per-token number — and the agreement is what licenses calling it a slope rather than a line drawn through two points. The decode side is shorter still, because rows one, four and five share a prompt: total time rises in step with the output count, at 7 ms a token, and TTFT does not move at all.

The division behind that ratio repays doing slowly, because the units are where people slip. The prefill slope is 45 microseconds for each prompt token. The decode slope is 7 ms for each output token. Those two are not comparable until they are in the same unit, and a millisecond is a thousand microseconds, so the decode cost of a token is several thousand microseconds set against the prefill cost of forty-five. Dividing one by the other gives about 155.

That lands as a surprise because of the intuition it displaces, and the intuition is worth naming rather than stepping over. The prompt is the visible bulk of a request. It is what fills the screen in the code, what the template work goes into, and what a reviewer sees when a request looks heavy; the answer is a few sentences. Everything about the shape of the text says the prompt is the expensive half. The arithmetic in chapter 1 says the reverse, and the reason is that the prompt's thousands of tokens cross the machine in one pass while the answer's few hundred cross it one at a time. What costs is not the amount of text. It is how much of it has to wait for the rest.

Where the slopes bend#

Both slopes are stable within a range and unreliable outside it, and the honest version of the experiment is to run it at the shapes you actually serve rather than to extrapolate.

Prefix caching — the reuse of stored keys and values across requests that share an opening, covered in chapter 2 — can flatten the TTFT slope to nothing. If the 2,000-token prompt in row two is the 1,000-token prompt of row one with 1,000 tokens appended, and the server caches prefixes, the second row will read 61 ms instead of 95: the added tokens are prefill work, but the original thousand are not. The same experiment with the 1,000 new tokens prepended instead shows the full 95 ms, because the cached prefix no longer starts at position 0. If your grid produces TTFT numbers that do not rise with prompt length, that is the first thing to check, and the check is to look at the cached-token count the response reports rather than to guess.

At long context the TTFT slope bends the other way, because attention's quadratic term stops being negligible: measured out to 32,000 prompt tokens, prefill time on this setup runs about 35% above the linear projection from the 4,000-token point. Inter-token latency stops being flat in the same region, since each decode step must also read a KV cache that has grown with the sequence, so at 32,000 tokens of context the ITL column would read closer to 13 ms than 7.

Load bends both, and it is the bend that matters most, because it is the one your users are inside. Every number in the grid came from an idle server. On a busy one, TTFT includes time spent waiting for the scheduler, and inter-token latency includes sharing decode capacity with everyone else being served in the same batch. The 45 microseconds per prompt token becomes an underestimate of what an extra thousand tokens of template costs, because that extra prefill work also delays every other request's tokens. Those effects are large enough that the grid should be run twice — once idle to isolate the model's cost, once under representative load to get the number a user experiences — and chapters 6 and 7, on batching and on concurrency, are about the difference between the two runs.

Server configuration bends them too, and one setting is worth knowing by name because it changes the shape of the TTFT column directly. Chunked prefill splits a long prefill into fixed-size pieces — 512 or 2,048 tokens at a time — and runs each piece in a slot that would otherwise have been a decode step. A 32,000-token prompt then arrives as sixteen chunks spread across sixteen scheduler iterations rather than one long occupation of the GPU. The request's own TTFT gets slightly worse, since it now competes with interleaved decode work, while every other in-flight request stops suffering a multi-second stall. Measuring the grid on a server with chunked prefill enabled and deploying with it disabled, or the reverse, produces numbers that will not reproduce.

Quality, hardware, and the cost a flat column hides#

The grid attributes latency to a phase, and that is all it does. It says nothing about whether the answers got better, which matters here because the two changes in the scenario were both quality changes: retrieval was added to reduce fabricated answers, and the token cap was raised because truncated answers were useless. A measurement showing that output length is responsible for 96% of a regression is an input to a product decision about how long answers should be, and it does not make that decision.

It also does not generalise across models or hardware. The 155:1 ratio is specific to an 8B model on an H100; on a 70B model served with tensor parallelism — every layer's weight matrices split across four GPUs that each compute a slice of it and then exchange results — prefill has more arithmetic to do while decode has more bandwidth available and an added inter-GPU synchronisation per step, and the ratio typically lands nearer 60:1. Re-run the grid when you change either the model or the machine, because the number is a property of the pair rather than of transformers in general.

The most common misreading is to treat the flat ITL column as proof that prompt length is free. It is free in inter-token latency, which is a different claim from being free in cost, in queueing or in context budget. Those 2,500 extra prompt tokens are billed on every request, they occupy 305 MiB of KV cache while the request is alive, and their prefill work occupies the GPU in a way that pushes other requests' first tokens later.

Two lengths, two slopes, one grid. The value of running it is that it converts "it got slower" into a statement about which phase moved, and the phases have different remedies: prefill work responds to shorter prompts and to prefix reuse, decode work responds to fewer output tokens, faster memory or a smaller model, and nothing that helps one does much for the other. The exchange rate between the two is what makes the attribution actionable rather than merely tidy, because at 155:1 the intuition that a long prompt is the expensive part is wrong by two orders of magnitude on an idle server.

Everything measured here came from a server with no other traffic, which was a deliberate simplification and is now the limiting one. The next chapter is about what a single stopwatch reading actually captures, and why a benchmark that does not separate queue time from execution time will misattribute a scheduling problem to the model.