Measuring TTFT and TPOT separately

Prefix caching is sold on latency more often than on cost, and the latency claim is the one most often misquoted, because the number that gets quoted averages two phases that behave completely differently. Prefill processes every token of the prompt in parallel before a single output token exists; its cost grows with prompt length, and it is exactly the work a cache hit skips. Decode then emits tokens one at a time, each step a full forward pass over one position, and its cost per step is set by how fast the model's weights can be read out of memory — which happens on every step whether or not the prefix was cached.

This chapter is about measuring the two separately and reporting them honestly. You will get the decomposition that explains why the same cache hit can look like a 6.9% win on one endpoint and a 54% win on another; a streaming measurement harness that records TTFT and TPOT from the same request; a sweep that shows what happens to each as prefix length grows from 512 to 32,768 tokens; and the memory column that belongs in the same sweep, because the reuse that keeps warm TTFT flat is paid for in resident bytes.

Decompose the number before you celebrate it#

End-to-end latency for a streaming generation is two quantities glued together:

e2e ≈ TTFT + (n_out − 1) × TPOT

The − 1 there is not a fudge factor. The first token's arrival is exactly what TTFT already measures, so charging a full inter-token gap for it as well would count it twice. What is left after the first token is the gaps between tokens, and eight hundred tokens have seven hundred and ninety-nine gaps between them. Multiply those gaps by the inter-token interval, add the head start, and the total is what a stopwatch running over the whole request would read.

Time to first token is dominated by prefill, plus queueing and tokenizer overhead. Time per output token is the steady-state inter-token interval during decode. A cache hit removes prefill work for the shared prefix, so it moves the first term. It does not touch the second, because decode was never doing the work that got skipped — decode's per-step cost is dominated by streaming the model weights out of HBM — the high-bandwidth memory stacked next to the GPU die, which holds the weights and the cache — for every token, and those weights are read whether or not the prefix was cached.

Why the two phases have different bottlenecks is the mechanism this chapter rests on, and it comes down to how much arithmetic each one gets per byte it reads. A GPU has two capacities that run out separately: how many floating-point operations it can perform per second, and how many bytes it can pull out of memory per second. Whichever of the two runs out first is what sets the time.

Take a decode step. It processes exactly one position, so every weight in the model is read out of memory, used for a couple of operations against that single position's vector, and dropped. The ratio of arithmetic to bytes read is therefore about as low as it can go, and the arithmetic finishes long before the next stretch of weights has arrived. The step's duration is the time to stream the weights, which depends on how large the model is and not at all on how the prompt was assembled.

Now take prefill over a few thousand tokens. The same weights are read once, and each of them is used against every one of those thousands of positions before being dropped. The arithmetic per byte read has gone up by a factor of the token count, which is enough to cross the balance over: the memory system now delivers weights faster than the arithmetic units can consume them, so the time is set by the operations rather than by the bytes. That is why prefill scales with prompt length while decode does not, and why an intervention that removes prompt tokens moves one of them and leaves the other precisely where it was.

Here is that decomposition for one prompt shape on one machine: a 9,000-token stable prefix plus a 200-token question, Llama 3 8B in bf16 on a single A100 80GB, batch size 1, measured over 50 runs.

Every cell in that table follows from the two phase figures and the output length, which is why it is worth reconstructing rather than reading. The warm 800-token row is 0.21 seconds of first-token delay plus seven hundred and ninety-nine gaps of 18 ms each, which is the 14.59 shown; the cold row is the same sum with 1.30 in front of it instead. The improvement column is then one subtraction and one division: 15.68 minus 14.59 leaves 1.09 seconds saved, and dividing 1.09 by the 15.68 it started from gives 6.9%. On the 40-token rows the saving is the identical 1.09 seconds, and dividing it by a starting point of 2.00 gives 54.5%. Same numerator, different denominator, and the denominator is chosen by the product rather than by anything the cache did.

The same 1.09 seconds is saved in every row. Whether that reads as "barely moved" or "more than twice as fast" is decided entirely by how many tokens the response contains, which is a property of the product and not of the cache. A classifier returning a single label sees the full benefit. A report generator writing two thousand tokens sees almost none of it in an end-to-end average, while still seeing its first token arrive six times sooner — which is the thing the user actually perceives in a streaming UI.

That is why a single latency figure is the wrong instrument. It is a weighted average of a phase that improved by 84% and a phase that did not improve at all, with weights set by output length, and it will report a different verdict for the same optimisation on two different endpoints. That 84% is the same division run on the phase in isolation: TTFT went from 1.30 s to 0.21 s, and the difference over the starting figure is where it comes from. The reason it never appears in the end-to-end column is that the phase it describes made up a small share of the request.

Measuring the two phases#

The measurement is straightforward if you consume the stream rather than the completed response, because the first chunk arriving is the first token.

import time, statistics, requests, json

def measure(prompt, max_tokens, url="http://localhost:8000/v1/completions"):
    t0 = time.perf_counter()
    r = requests.post(url, json={
        "model": "meta-llama/Meta-Llama-3-8B-Instruct",
        "prompt": prompt,
        "max_tokens": max_tokens,
        "temperature": 0.0,
        "stream": True,
    }, stream=True)

    stamps = []
    for line in r.iter_lines():
        if not line or not line.startswith(b"data: "):
            continue
        payload = line[6:]
        if payload == b"[DONE]":
            break
        if json.loads(payload)["choices"][0]["text"]:
            stamps.append(time.perf_counter())

    ttft = stamps[0] - t0
    gaps = [b - a for a, b in zip(stamps, stamps[1:])]
    return {
        "ttft_ms": ttft * 1000,
        "tpot_ms": statistics.median(gaps) * 1000,     # median, not mean
        "n_out": len(stamps),
        "e2e_ms": (stamps[-1] - t0) * 1000,
    }

Two details in there are load-bearing. Skipping chunks with empty text matters because engines emit role and usage frames that are not tokens, and counting them as the first token understates TTFT by however long the connection setup took. Taking the median gap rather than the mean matters because continuous batching — the scheduler admitting arriving sequences into the running batch and retiring finished ones between steps, rather than running fixed groups start to finish — produces occasional multi-hundred-millisecond stalls when a new sequence joins the batch and a prefill runs alongside your decode; a mean absorbs those into the steady-state figure and makes decode look worse than it is, and the median tells you the rate the user experiences between hiccups.

Then sweep, holding output length fixed and varying prefix length, and run each length twice — the first call fills, the second reads:

for prefix_len in (512, 2_048, 8_192, 32_768):
    prompt = build_prompt(prefix_len)
    cold = measure(prompt + unique_suffix(), max_tokens=64)   # forced miss
    _ = measure(prompt + "\n\nQ: warmup", max_tokens=1)       # fill
    warm = measure(prompt + "\n\nQ: warmup", max_tokens=64)   # read
    print(prefix_len, round(cold["ttft_ms"]), round(warm["ttft_ms"]),
          round(cold["tpot_ms"], 1), round(warm["tpot_ms"], 1))

prefix   ttft_cold   ttft_warm   tpot_cold   tpot_warm
   512       118 ms       92 ms     17.8 ms     17.8 ms
 2 048       281 ms       97 ms     18.0 ms     18.1 ms
 8 192       1 168 ms     104 ms    18.6 ms     18.7 ms
32 768       4 640 ms     131 ms    21.9 ms     22.0 ms

Read the columns rather than the rows. Cold TTFT grows roughly linearly with prefix length, which is prefill doing work proportional to tokens. Warm TTFT is nearly flat, growing only slightly because the engine still has to hash the blocks, walk the index, and prefill the unmatched suffix. And TPOT is identical between cold and warm at every prefix length, to within measurement noise, which is the whole claim of this chapter stated as data.

The TPOT column also does something that surprises people: it rises with prefix length, from 17.8 ms to 21.9 ms. Every decode step attends over the full resident KV, so a 32,768-token context means reading 4 GiB of key and value tensors out of HBM per token on top of the 16 GB of weights. At roughly 2 TB/s that is about 2 ms of extra memory traffic per step, and it is present whether the prefix was cached or recomputed. A long cached prefix makes your first token much faster and your subsequent tokens slightly slower, and both effects are consequences of the same tensors being resident.

Both figures in that paragraph derive from what earlier chapters supply, and they mark the only place in this topic where the cache costs time instead of saving it. A token of key and value on this model occupies 128 KiB, so 32,768 tokens occupy the 4 GiB quoted — the same multiplication chapter 1 ran on a shorter prefix. Dividing 4 GiB by a memory bandwidth of roughly 2 TB/s leaves a time, and that time is the couple of milliseconds by which TPOT rises between the shortest and the longest rows of the sweep. It stays small because 4 GiB is small next to the 16 GB of weights being read on the same step. It is not zero, and on a workload that keeps very long contexts resident it is the term that eventually stops being small.

Sample memory in the same sweep#

Latency is half of what that loop should record. The other half is what the reuse is costing in residency, and it is cheap to capture in the same pass — sample the device before the warm call and after it, and you have the trade on one row.

import torch, subprocess

def gpu_bytes():                       # in-process view
    return torch.cuda.memory_allocated()

def gpu_used_mb():                     # whole-device view, incl. the engine
    out = subprocess.check_output(
        ["nvidia-smi", "--query-gpu=memory.used", "--format=csv,noheader,nounits"])
    return int(out.split()[0])

prefix   ttft_warm   tpot_warm   kv_resident   device_used
   512        92 ms     17.8 ms       64 MiB      18 042 MiB
 2 048        97 ms     18.1 ms      256 MiB      18 235 MiB
 8 192       104 ms     18.7 ms     1 024 MiB     19 010 MiB
32 768       131 ms     22.0 ms     4 096 MiB     22 088 MiB

The two columns move in opposite directions and at different rates, which is the whole shape of the decision. Warm TTFT is essentially flat across a sixty-four-fold increase in prefix length — the reuse is doing its job. Resident KV is exactly linear in prefix length, because it is 128 KiB × tokens and nothing else. So the marginal cached token is nearly free in time and precisely priced in bytes, and the question of how long a prefix is worth retaining is a memory question rather than a latency one.

Record the device-wide figure alongside the in-process one. A serving engine preallocates its KV pool at startup, so torch.cuda.memory_allocated() inside your own process will not show the engine's blocks at all, and a benchmark that reports only the in-process number will show a flat memory line while the device fills up underneath it.

Reporting the result without overstating it#

Which number to publish depends on what the endpoint is for, and the honest version is usually two numbers rather than one.

The belief that produces the overstatement is worth writing out as somebody would hold it: the prompt got six times cheaper to process, so the endpoint is six times faster. It is tempting because the prefill figure really did move by that much, because latency is habitually reported as a single number, and because nobody naturally thinks of a response as having two halves running on different physics. What is true instead is that a request is a short phase whose cost you changed followed by a long phase whose cost you did not, and the ratio between them is fixed by how many tokens the model was asked to write. A team that has the decomposition can predict the end-to-end figure before shipping, from the median output length alone, which is a better position to argue from than measuring it afterwards and being surprised.

For anything with a person waiting on a stream — chat, a coding assistant, an interactive agent — the metric that matches perceived responsiveness is TTFT, and it is the one to attach an SLO to, meaning the service level objective you commit to and are paged about. A p95 TTFT target of 500 ms is a statement about prefill, queueing, and cache hit rate, all of which you control; a p95 end-to-end target on the same endpoint is largely a statement about how long the answers are, which the product decides. Teams that put their SLO on end-to-end latency find it regressing whenever a prompt change makes responses more thorough, which is a quality improvement showing up as a reliability breach.

For batch and non-streaming work there is no first token to perceive, so end-to-end is the right metric — and prefix caching should be sold to that audience on throughput and cost rather than on latency, because the latency effect will be small and the cost effect will not be.

Speculative decoding, output-bound work and continuous batching#

Prefix caching does not increase the generation rate. With prefix caching working well, prefill only gets faster: TTFT drops while the per-token rate is unchanged. If a stakeholder needs tokens per second to improve, prefix caching is the wrong lever and the right ones are elsewhere — speculative decoding, a smaller or quantised model, tensor parallelism (splitting each layer's weights across several cards so their memory bandwidth adds up), or simply generating fewer tokens.

It also does not help a workload that is output-bound in aggregate. A batch pipeline that sends 200-token prompts and receives 3,000-token completions spends 98% of its GPU time in decode, and a perfect hit rate on those 200 tokens is a rounding error. The workloads where prefix caching is transformative are the ones with the opposite ratio — long stable instructions, tool schemas, retrieved documents, multi-turn history — and short answers.

There is one honest complication in the other direction. Everything above is batch size 1, where the two phases are cleanly separable and nothing competes. Under continuous batching they compete for the same GPU: a prefill for an arriving request occupies the device in the same window your sequence wants a decode step, which is where those multi-hundred-millisecond gaps come from. Cutting prefill work across the fleet therefore does improve decode latency at the system level, because there is less prefill to be interleaved with the decode steps. No individual decode step got cheaper. On a loaded server you will often see p99 TPOT improve after enabling prefix caching while p50 TPOT stays flat, and that is a queueing effect rather than a per-token one. Report it as such, because the alternative is a claim that will not reproduce on an idle machine.

The instrument this chapter argues for is therefore two series rather than one, recorded from the same streaming request: the interval before the first token, which prefix caching moves by roughly the ratio of cached to total prompt tokens, and the median interval between subsequent tokens, which it does not move at all and which rises slightly as more cache goes resident. Sweeping them together across prefix length shows both effects at once and takes an afternoon. Reporting them together, with the median output length attached, is what keeps the claim true when somebody re-runs it on an endpoint that answers in forty tokens instead of eight hundred.

Which points at the constraint the next chapter takes up. The reason those flat warm-TTFT numbers hold is that the prefix blocks were resident when the second request arrived. Under real traffic they compete for the same pool as every in-flight sequence's own cache, and the sweep above — 4 GiB of resident KV for a single 32,768-token sequence — is a hint about how quickly that pool runs out. The failure when it does is not a slower response.