Latency optimization

Read these first

• Metrics and SLOs

What you will know

• Reduce perceived latency without reducing total latency

  A fast first token can still conceal a long wait for useful or complete output, so evaluate multiple user-facing milestones.

• Parallelise independent calls in a pipeline

  Fan-out lowers wall-clock time but raises concurrency, rate-limit exposure, and the chance that one slow branch delays the response.

• Use a small model for a first response and a larger one behind it

  The handoff can produce contradictory or duplicated content unless the provisional response is clearly bounded and safely replaced.

• Measure what users actually feel rather than what the timer says

  Track time to first token, time to useful content, completion time, and tail percentiles separately from server-side duration.

• Compare streaming, predicted outputs, prefetching, and skeleton-then-fill on latency, cost, and quality using a fixed workload

  Predictions fail on mismatches, prefetches can waste capacity, and skeletons can add rework when the final content does not fit.

Chapters

1. 1

   First token, first useful content, and last token are three different waits

   Streaming can cut time to first token from 2.1s to 300ms while the wait users complain about does not move at all, because the eligibility verdict they came for is the last sentence of the answer. Separate three milestones — first token, first content worth acting on, and completion — and decide which one the task actually depends on before optimising anything.

   Read chapter
2. 2

   A faster first token can still finish three times later

   Model A gives 200ms to first token at 30 tokens/sec; Model B gives 900ms at 110. On a 600-token answer A finishes near 20 seconds and B near 6.4, so ranking by first-token time picks the model that is three times slower to a complete answer. How the output is consumed — read along with the stream, or parsed whole as a JSON payload — decides which number should win.

   Read chapter
3. 3

   The server timer starts after admission and stops at the last byte

   Server-side p50 and p95 stay flat for a month while sales and support insist the assistant feels slower than it used to. The server clock never saw the queueing, the network, or the render. Instrument at the client instead: time to first token and time to the first content a user can act on, at p95 and above, because the averages hide precisely the requests that generate the complaints.

   Read chapter
4. 4

   Fan-out turns a sum of latencies into a maximum

   Five sequential enrichment calls at 4.0s become 0.9s when fired concurrently, and p99 gets worse than it ever was sequentially: the response now waits on the slowest of five branches rather than the sum of five typical ones, so one fat-tailed straggler anywhere sets the tail. Covers per-branch deadlines, what the request returns when one fires, and why retries inside a fan-out extend the barrier instead of protecting it.

   Read chapter
5. 5

   A provisional answer may only say what the real one cannot contradict

   Returning a small model's answer at 300ms and replacing it at 3s works only if the provisional text is bounded to content the large model cannot retract — restating the question, naming the sources — and lands in a single atomic swap. On a legal-research tool a retracted claim has already been acted on, and a 'still checking, this may update' label moves responsibility without changing behaviour.

   Read chapter
6. 6

   Predicted outputs and prefetches bill for every miss

   Tokens that diverge from a prediction are still charged as output, so a restructure pays for the mispredicted file and gets no speedup while a one-line edit accepts nearly all of it. Prefetching a summary on every document open buys near-zero latency on the click and pays for every open that never clicks. Compare streaming, prediction, prefetch, and skeleton-then-fill on one fixed workload, pricing hit rate beside the milliseconds.

   Read chapter
7. 7

   A latency win that moves no user milestone is a cost increase

   A quarter ends with first-token time down 70%, p95 completion time up 15%, cost per request up 20%, and an unchanged thumbs-down rate offered as proof that nothing broke. Thumbs are far too sparse and self-selected to detect a 15% shift in anything. Learn to declare the target milestone and its budget before the work starts, so that keeping everything and fixing completion time later is not the only option left.

   Read chapter