Autoregressive generation

Read these first

• Tokenization

What you will know

• Build a tiny greedy decoder trace that logs one forward pass and one selected token per step, then verify no step can select token n without the prior generated prefix.

  The dependency is causal: changing an earlier token changes the state from which every later choice is made, and there is no revision pass.

• Predict which of two requests will take longer from output length alone

  This is a useful first estimate only when model, hardware, batching, input size, and decoding settings are otherwise comparable.

• Measure time to first token and inter-token latency while varying prompt tokens and requested output tokens independently.

  Prompt processing mainly affects startup, while serial decoding affects the response tail; batching and cache state can blur that split.

• Compare a long-prompt, short-completion request with a short-prompt, long-completion request at fixed model, hardware, and concurrency, and report which component dominates.

  Prefill can be parallelized but still consumes memory bandwidth and compute, whereas decode repeatedly pays for each additional output token.

• Run a streaming request, cancel it before a target suffix, and verify the service cannot return that suffix without decoding the intervening tokens.

  Cancellation prevents future work, but a separate request that asks directly for the suffix is a different computation and may produce different content.

Chapters

1. 1

   Logits and the decoding policy

   A forward pass does not hand back a word. It hands back one score — a logit — for every entry in the vocabulary, tens or hundreds of thousands of numbers, normalised by softmax into a probability distribution over all of them. Choosing is a separate step performed by ordinary code above the model, and greedy decoding simply takes the highest-probability entry. Keep the halves apart: the network produces the distribution, the decoding policy picks from it, and every sampling parameter you can tune acts only on the second half.

   Read chapter
2. 2

   The decoding loop

   Trace a greedy decoder by hand: each step runs the model over the prefix so far, selects one token, appends it, and starts again. Nothing revisits an earlier choice, so changing one word near the beginning of an answer changes the state every later token was conditioned on, and the rest diverges completely — measurably, in 92% of cases within twelve tokens. The exchange rate that fixes everything later in this topic is set here: one token of output costs one full invocation of the model.

   Read chapter
3. 3

   Prefill and decode

   The prompt is known in full before generation starts, so all of its positions are computed in a single largely parallel pass under a causal mask. The per-position key and value state from that pass is kept, so each generated token costs one incremental step rather than a fresh run over the whole prefix. Prefill is compute-bound and amortises one weight read across every prompt token; decode is bandwidth-bound and can amortise it across nothing, which is why an output token costs roughly a hundred prompt tokens on the same hardware.

   Read chapter
4. 4

   Output length and latency

   A 20,000-token prompt with a 50-token answer finishes about ten times sooner than a 200-token prompt with a 2,000-token answer, because the prompt is one largely parallel pass while every generated token costs its own sequential pass. Ranking two requests by total tokens gets this backwards, and on the same hardware one output token is worth roughly 140 prompt tokens. Compare the two shapes at fixed model, hardware and concurrency, and report the split — how much of the wall clock fell before the first token and how much after it.

   Read chapter
5. 5

   Comparing latency across deployments

   Output length ranks two requests reliably on one model, one machine, one concurrency level and comparable prompts, and inside that fence it explains almost all the variance in request latency. Across deployments it compares two exchange rates as though they were one: a 70B model on four A100s decodes at 20 tokens a second where an 8B on an H100 manages 90, so 500 tokens can lose to 2,000. Measure decode rate per deployment, compare tokens divided by rate, and carry the conditions with the prediction.

   Read chapter
6. 6

   Streaming and time to first token

   A request has a startup clock and a per-token clock. Streaming collapses the first and leaves the second exactly where it was, because the same prefill and the same number of decode steps still have to run. Measure them separately with a two-armed sweep: hold the output fixed and push the prompt to watch startup move, hold the prompt fixed and push the output to watch the tail move. Batching, cache hits, chunked prefill and buffering proxies all blur the split, which is why one aggregate latency number hides what users feel.

   Read chapter
7. 7

   Cancelling a stream

   Billing follows compute performed. Cancel after 300 of an expected 800 tokens and you pay for 300 forward passes that cannot be un-run, while saving the 500 that never happened and the nine seconds of accelerator time they would have taken. Cancellation cannot skip ahead either — no path through the network produces token 750 without tokens 1 through 749 — and the usage event never arrives on an aborted stream, so the tokens have to be counted as they land.

   Read chapter
8. 8

   Trimming the output schema

   Not asking for a field is the largest latency lever on a generation endpoint, because those tokens are never generated; generating them and stripping them server-side saves the response body and nothing else. The catch is that the skipped tokens were part of the computation that produced the fields after them, so the cheaper request can return a different and sometimes worse answer — and a schema that declares its score before its reasoning gets the cost of both with the quality of neither.

   Read chapter