Chat latency SLOs

Total response time is the natural quantity to promise, because it is the one a client can measure without any cooperation from the server, and it is the one every non-streaming service is judged on. On a streaming endpoint it decomposes into three terms — TTFT + (output_tokens − 1) × ITL, the time to first token plus one inter-token gap for every token after the first — of which two are speed and one is length. A number built from three inputs cannot say which one moved, and one of those inputs is under the product's direct control through a system prompt.

What follows is the objective a chat feature should carry instead, with every number in it defended from two directions at once: what a person actually perceives, and what the hardware in chapters 4 and 7 was measured to deliver.

Total time on a streaming endpoint is two variables#

Take the concurrency change from chapter 7, where a cap raised from 32 to 96 deepened the batch, and suppose that in the same release a prompt tweak asked the model to be more concise, cutting median answers from 300 tokens to 120.

before:  TTFT 108 ms + 299 × 10.3 ms  =  3.19 s
after:   TTFT 272 ms + 119 × 16.8 ms  =  2.27 s     (−29%)

Median total latency improved by 29% while every speed term got worse: the first word arrives 164 ms later and the text then crawls at 60 tokens per second instead of 97. The dashboard reports a 30% win. The user waits noticeably longer to see anything, reads more slowly, and gets a shorter answer. Nothing about that is exotic — it follows from the arithmetic whenever length and speed move at the same time, and they very often move in the same release.

The arithmetic is worth pulling apart, because the trap lives in a product rather than in a sum. Total time is the first-token wait, plus the number of gaps multiplied by the size of a gap. Between those two lines the number of gaps fell from 299 to 119, which is less than half, while the size of each gap rose from 10.3 ms to 16.8 ms, which is an increase of about three-fifths. A product in which one factor more than halves and the other rises by three-fifths comes out smaller. It comes out smaller regardless of how much worse the machine became, and it would come out smaller still if the machine had become worse again. That is the entire mechanism of the false improvement: a dashboard watching a product, and a release note attributing the movement to the factor somebody can name.

The same effect appears without any deliberate change, through traffic mix. If a routing layer starts sending 30% of questions to a short-answer fast path — a cached FAQ, a small model, a tool result rendered directly — the aggregate total-latency distribution shifts down even if every request served by the main model got slower. Percentiles computed over a mixed population move when the mixture moves, and there is no way to tell that from a single line on a chart.

Where a stricter target is met by shorter answers#

The natural response to complaints against a green dashboard is to make the target stricter: p95 under 2 seconds instead of 3. Follow the incentive through and the outcome is predictable, because engineering will hit the target using the cheapest available lever, and the cheapest lever on total response time is answer length. Cutting median output from 300 tokens to 180 removes 1.2 seconds at 10.3 ms per token, requires no infrastructure work and no cost increase, and can be done in an afternoon by editing a system prompt. Any real latency work — more replicas to lower the batch, a faster GPU, speculative decoding — costs money and weeks and delivers less.

So the metric gets greener while answers get terser and less useful, which is a worse product than the one that started the process. The failure is structural: a target on a quantity that does not describe the experience will be met by moving whichever of its inputs is easiest to move, and here the easiest input is the one you least want changed.

A p95 on the first token, and a floor on the rest#

The objective a chat feature should carry is a p95 on time to first token together with a floor on the sustained rate at which text then appears, both measured at the client where the promise is made.

slo:
  ttft:
    metric: client_first_visible_token_ms   # first chunk with non-empty content
    p95: 800
    p99: 2000
  sustained_rate:
    metric: per_response_median_itl_ms
    p95: 50                                  # ≥ 20 tokens/s sustained
  stall:
    metric: per_response_max_gap_ms
    p99: 600
  window: 28d
  diagnostics_only: [total_response_ms, output_tokens, queue_time_ms]

The 800 ms on first token comes from two directions that happen to agree. Human-factors work on interactive response times, going back to Miller's 1968 thresholds and repeated in every usability text since, puts roughly 100 ms at the boundary of "instantaneous", about one second at the limit for keeping a user's flow of thought uninterrupted without a progress indicator, and about ten seconds at the point where attention leaves entirely. A first token inside 800 ms therefore stays under the flow threshold with a little room for the network. From the other direction, chapter 4 measured idle TTFT at roughly 130 ms and chapter 7 measured 340 ms at p95 under moderate load, so 800 ms is achievable on current hardware and leaves about 670 ms of budget for queueing, the gateway and the client's own network. A target should be both meaningful to a person and reachable by the system, and a number that satisfies only one of those is either theatre or an unfunded mandate.

The p99 at 2 seconds exists because the p95 alone lets one interaction in twenty be arbitrarily bad, and chapter 7's sweep showed that queue-driven tails do not degrade gently — the difference between a healthy server and an overloaded one appears at p99 long before it appears at p95.

"One interaction in twenty" is what a percentile means once it is unpacked, and unpacking it matters because the phrase sounds far more forgiving in the abstract than it is in a session. A p95 is obtained by sorting every measured value and reading off the one below which ninety-five per cent of them fall, so a p95 of 800 ms says that nineteen of every twenty first tokens arrived inside 800 ms, and says nothing at all about how bad the twentieth was. Now count in sessions rather than in requests. Somebody asking twenty questions in one sitting meets that target twenty times, and the chance that all twenty land inside it is nineteen-twentieths multiplied by itself twenty times, which comes to a little over a third. A comfortably met p95 therefore means most engaged users meet the tail during an ordinary session. That is why a clause bounding the twentieth case is a separate promise rather than a refinement of the first one.

The 50 ms floor on inter-token latency comes from reading speed. Adults read continuous English prose silently at around 240 words per minute, which is 4 words per second, or roughly 5.5 tokens per second at typical English tokenisation. A floor of 20 tokens per second is about 3.6× that, which keeps generated text comfortably ahead of the eye and leaves enough margin that a brief slowdown does not let the reader catch up with the cursor. Setting the floor at reading speed itself would be a design that fails as soon as anything goes slightly wrong, and setting it much higher buys nothing a reader can perceive — which is why chapter 7's drop from 97 to 60 tokens per second was not what generated tickets.

Each step of that conversion is worth seeing, because a floor stated in tokens has to be defended in words. 240 words a minute, divided by the sixty seconds in a minute, is 4 words a second. English tokenisation cuts a typical word into a little more than one token, so a reading rate of 4 words a second is a somewhat larger number of tokens a second, which is where 5.5 comes from. Dividing the floor of 20 tokens a second by that reading rate gives the 3.6× of headroom. The headroom is the whole object of the exercise. At parity the reader sits exactly on the cursor, so any hesitation anywhere in the path puts them ahead of the text, and waiting for the next word is a much worse experience than reading words that are already on the screen.

Aggregating inter-token latency per response before taking a percentile matters more than it sounds. A percentile over all token gaps pooled across all responses is dominated by whichever responses were longest, and it treats a token as the unit of experience when the unit is a response. Taking each response's median gap, then the p95 across responses, answers the question the SLO is actually asking: in what fraction of conversations did the text come out too slowly.

The weighting is easy to make concrete. A thousand-token response contributes ten times as many gap samples to a pool as a hundred-token response does, so a percentile taken over the pool is weighted by response length whether or not anybody intended that. What it reports is the fraction of tokens that were slow. Nobody experiences a token. A reader experiences a response, so the aggregation that matches the experience gives every response one vote — its own median gap — and then asks what fraction of the votes were slow. Two aggregations of the same measurements, answering two different questions, and only one of the questions has a person in it.

The separate stall metric exists because a single long gap is what makes a stream feel broken, as chapter 5 established, and because such a gap is invisible in any per-token percentile. One 900 ms pause in a 300-token response is one sample in three hundred, so it does not reach a p99 computed over gaps, while it is the single most noticeable thing about that response.

What goes on the dashboard, and what stays a diagnostic#

Total response time keeps a place on the chart, because it remains the right diagnostic for asking whether a change was speed or length — read alongside the output-token distribution it becomes informative again, since a fall in total with flat token counts is a genuine speed win and a fall in both is a verbosity change. What it must not be is a target, because a target is a thing people optimise, and this is not a quantity anyone should optimise directly.

Writing the objective this way also changes what a breach means operationally, because the clauses point at different remedies. A TTFT breach with server queue time high says the deployment is past the knee of chapter 7's utilisation curve, and the responses available are more replicas, a lower concurrency cap, admission control, or shorter prompts. A TTFT breach with queue time flat says the delay is outside the engine — a cold replica, a buffering proxy, the client's own network. An inter-token breach with TTFT healthy says the batch is too deep for the decode capacity, which is a different fix from the first case even though both are triggered by load. A single total-latency objective supports none of those inferences, since every cause lands on the same number.

Two practical points about measurement. Histogram bucket boundaries have to include the SLO threshold, or the percentile you report is an interpolation across a bucket that straddles it; a histogram with edges at 0.5 and 1.0 seconds cannot tell you your p95 against an 800 ms objective to better than ±300 ms. And the metric has to be emitted from the client, per chapter 5, since a server-side TTFT of 96 ms and a client-side one of 4.2 seconds are both true and only one of them is a promise to a user.

The first of those looks like a rounding detail and is not, so the mechanism is worth having. A histogram does not store values. It stores how many observations fell into each bucket, and nothing about where inside the bucket they fell. Asking it for a percentile therefore means finding the bucket that the percentile falls inside and then guessing a position within it, conventionally by assuming the observations are spread evenly across the bucket. With edges at half a second and one second, every observation between those two is recorded only as "somewhere in there", so a p95 reported as 800 ms is an assumption about the shape of a distribution inside a bucket rather than a measurement of anything, and the true value can sit anywhere across the range. Putting a bucket edge at the threshold you promised removes the guess completely: the count at that edge is the number of requests that met the objective, which is the quantity the objective is written about.

Workloads with nobody waiting#

For a workload with no human waiting on the output, every number above is irrelevant. A nightly job that summarises a document archive, an offline classification pass, an embedding backfill — none of them has a reader, so first-token latency measures nothing anybody experiences and a floor on inter-token rate constrains nothing worth constraining. The objective there is aggregate throughput and cost per million tokens, with a completion deadline as the only latency-shaped requirement, and the correct engineering move is the opposite of this chapter's: raise concurrency until the tail degrades, because nothing is on the other end of the tail.

An objective is a statement about what a service owes somebody, and it only works if the quantity it names is the one that person experiences. On a streaming chat endpoint that quantity is not how long the response took, because the response taking longer is often what a good answer looks like; it is how long the user stared at nothing, how steadily the text then arrived, and whether it ever stopped. Three clauses, measured at the client, cover those three things, and each of them points at a different part of the system when it breaks — which is the practical reason to prefer them over one number that is easier to compute and answers nothing.

That inversion is complete enough that the same two endpoints can rank in opposite orders for the two workloads, and the final chapter works through exactly that choice with numbers on both sides.