A first token on time says nothing about the last one

The latency panel for the support assistant has been green since January. The objective is p95 time-to-first-token under 500 ms on the streaming chat endpoint, and the quarterly figures read 310 ms in March, 344 ms in May, 380 ms in June. Every one of those is comfortably inside target, the graph slopes gently upward in a way that nobody would call an incident, and the on-call rotation has not been woken by it once. Over the same three months, support tickets tagged "assistant is slow" went from 9 in March to 31 in June. The traces from the previous topic are clean: no retries, no tool loops that ran three times, no provider errors, no span with a suspicious gap in it.

The tickets are correct and the dashboard is correct, because they are describing different halves of the same request. A representative answer from that assistant is around 600 output tokens. In March the model emitted those tokens at roughly 150 per second, so the user watched the answer finish about 4.3 seconds after they hit enter. In June the same 600 tokens came out at about 55 per second, which puts the last token at 11.3 seconds. Nothing in the metric set recorded that, because the only clock anyone had started stopped when the first token arrived.

Both of those figures are service level indicators, or SLIs: measurements taken from live traffic that stand in for something a user experiences, here how long they waited. The 500 ms line drawn across the first of them is a service level objective, an SLO, which is a target agreed in advance and written in the form this indicator, over this window, across these requests, stays inside this bound. Writing it down is what turns an argument about whether the assistant feels slow into a number a team either met or missed, and the shortfall the target tolerates — its error budget — is what says whether there is room to keep shipping features or reason to stop and repair the service. You meet them on the latency dashboard, in the threshold that decides whether an alert wakes someone at three in the morning, and on the weekly slide where a platform reports whether it held its targets. The contractual relative is the SLA, a service level agreement, which is the version promised to a customer with money behind it and is normally set looser than the internal objective, so that missing an SLO is a warning rather than a refund.

The indicators this topic is about are the ones an LLM service needs and a conventional web service never had to think about: time to first token (TTFT), time to last token, output tokens per second, error rate, cost per request, and a target on whether the answers were any good. The support assistant above has an objective on the first of those and nothing on any of the others, which is the whole of its problem.

The objective stops at the boundary between the two phases#

Prefill and decode were covered earlier as a fact about how inference works; what matters here is that they fail independently and that TTFT is measured exactly at the seam between them. Everything that determines TTFT — queue wait, prompt tokenisation, the single forward pass over the whole prompt — happens before the seam. Everything that determines how long the user actually waits for a usable answer happens after it, one token at a time, and a TTFT objective is structurally incapable of seeing any of it.

The two halves also degrade for different reasons, which is why the June regression could leave TTFT almost untouched. Prefill is compute-bound and decode is memory-bandwidth-bound, so when a provider raises the maximum batch size to push more aggregate tokens per second through the same GPUs, per-request decode slows down while prefill barely notices. That is a rational change for the operator and an invisible one to a TTFT dashboard. The same asymmetry shows up with speculative decoding being disabled after a model update, with a KV-cache eviction policy that starts recomputing under load, and with a switch from one serving tier to another. In each case, the number you were watching is the one that did not move.

Four clocks are worth running on a streaming request, and they answer four different complaints.

ITL and output TPS are two views of the same quantity and you generally instrument one of them; ITL is more sensitive to the stutter caused by a single slow batch, output TPS reads better on a dashboard because bigger is better. NVIDIA's NIM LLM Benchmarking: Metrics page is worth reading once for the precise definitions, because teams routinely disagree about whether the first token counts in the ITL denominator, and a 600-token answer scores differently depending on the answer.

Instrumenting the whole request, not the handshake#

The instrumentation is not complicated once you decide to record the end of the stream as deliberately as the beginning. The pattern is to stamp a monotonic clock at request start, at first content token, and at stream close, and to carry the output token count out of the final usage event so that throughput is computed rather than estimated from character counts.

import time
from prometheus_client import Histogram

TTFT = Histogram(
    "llm_ttft_seconds", "Time to first content token",
    ["route", "model", "prompt_version"],
    buckets=(0.1, 0.2, 0.3, 0.5, 0.8, 1.2, 2.0, 5.0),
)
TTLT = Histogram(
    "llm_ttlt_seconds", "Time to last token",
    ["route", "model", "prompt_version"],
    buckets=(0.5, 1, 2, 4, 6, 8, 12, 20, 40),
)
OUT_TPS = Histogram(
    "llm_output_tokens_per_second", "Per-request decode throughput",
    ["route", "model", "prompt_version"],
    buckets=(10, 20, 40, 60, 90, 120, 160, 220),
)

async def stream_answer(client, req, labels):
    start = time.perf_counter()
    first = None
    out_tokens = 0
    async with client.messages.stream(**req) as stream:
        async for event in stream:
            if event.type == "content_block_delta":
                if first is None:
                    first = time.perf_counter()
                    TTFT.labels(**labels).observe(first - start)
                yield event.delta.text
        final = await stream.get_final_message()
        out_tokens = final.usage.output_tokens
    end = time.perf_counter()
    TTLT.labels(**labels).observe(end - start)
    if first is not None and end > first:
        OUT_TPS.labels(**labels).observe(out_tokens / (end - first))

Two details in that snippet do real work. The first content token is not the first event — the stream opens with message and content-block starts that carry no text, and timing TTFT from those will report a number 40 to 80 ms better than the user experiences. The second is that a stream the client abandons never reaches the TTLT observation, so an endpoint where users habitually cancel long answers will quietly drop its worst requests out of the histogram, which is the population problem in miniature.

Histogram buckets deserve a moment of thought too, because a percentile computed from a histogram is interpolated within a bucket. If the June regression had pushed TTLT from 4.3 s to 11.3 s and the top bucket boundary was 10 s, the reported p95 would have pinned at the bucket edge and understated the damage. Buckets that stop where you expect trouble to start are buckets that cannot report trouble.

Answer length moves end-to-end latency, and the objective still holds#

The obvious objection to an end-to-end objective is that it depends on something the model chooses. A prompt tweak that makes answers more thorough pushes mean output length from 600 to 900 tokens; at an unchanged 150 tok/s the p95 of end-to-end latency rises by roughly two seconds, and the objective can breach without any component of the system having got slower. The mechanism is real and the arithmetic is right.

It does not make the objective wrong, because the user is waiting for the answer and not for the tokens. A summary that takes eleven seconds to finish is eleven seconds of waiting whether the cause is a slow GPU or a chatty prompt, and if verbosity has pushed you outside the latency budget then verbosity is a thing you now have to manage — with max_tokens, with a prompt that asks for brevity, or by renegotiating the budget on evidence. The failure mode to avoid is treating a breach caused by output length as a measurement artefact and excusing it, because the excuse works equally well for a genuine slowdown that happens to coincide with longer answers.

What length-dependence does require is a companion metric that is normalised, which is what per-request output TPS is for. When end-to-end latency rises, output TPS says immediately whether the tokens got slower or there were simply more of them, and the two numbers together decompose the regression in one glance. Stratifying by request class helps further where classes differ by an order of magnitude in length: a routing classification that returns 20 tokens and a document summary that returns 900 do not belong in the same percentile, and a single p95 over both is dominated by whichever class happens to be busiest that week.

Which percentile belongs to which experience#

An objective also has to choose a percentile, and the three that get quoted describe different things about the same distribution. The median is the experience of a typical request and is the right figure for a claim about how the product feels; it is also nearly useless as an objective, because a serving problem that affects one request in twenty will not move it at all. The 95th percentile is where most latency objectives live because it corresponds to roughly one request in twenty being worse than the stated bound, which is frequent enough that a user with a daily habit meets it within a fortnight. The 99th percentile is dominated by the tail causes — a cold model replica, a request that queued behind a very long prompt, a retry after a provider hiccup — and it is the figure that governs anything with fan-out.

That last point is the one that decides the choice on an agent. A single chat turn that makes one model call sees the p95 of that call. A turn that makes eight sequential tool-augmented calls sees at least one p99-ish call with probability around 8%, because the chance that none of eight independent calls lands in the slowest 1% is 0.99 to the eighth power, or about 92%. The user-visible latency of a multi-step agent is therefore governed by the tail of the per-call distribution rather than by its middle, which is why an agent product with a healthy p95 on every individual call can still feel unreliable. ClickHouse's write-up on inference latency makes the same argument in more detail and is worth reading if you are choosing percentiles for the first time.

Percentiles on end-to-end latency also need the output-length caveat applied twice, since the tail of the latency distribution and the tail of the length distribution are the same requests. A p99 end-to-end objective on a route where the longest 1% of answers run 2,400 tokens is, in effect, an objective about max_tokens, and it is more honest to cap the output explicitly than to discover the cap by breaching a latency target.

A percentile with no window and no population is an average over whatever arrived#

Neither "p95 TTFT under 500 ms" nor "p95 end-to-end under 8 s" is a complete objective, because a percentile is a function of a set of events and neither statement says which events. Written out fully, the first one reads: over a rolling 28-day window, the 95th percentile of time-to-first-token across streaming chat completions from authenticated end users, excluding synthetic monitors, internal load tests, replayed evaluation traffic, and requests that fail before the first token, shall be under 500 ms.

Leaving the window unstated means the number is not comparable to itself. A p95 over the last hour is dominated by whatever is happening right now and will swing by hundreds of milliseconds between refreshes; a p95 over 90 days will absorb an entire bad week without visibly moving. Two people looking at the same dashboard with different range pickers will read different numbers and both will believe they are quoting the SLO. A fixed window is also what makes error-budget arithmetic possible at all, since a budget is a quantity of bad events permitted per window and there is no budget without one.

Leaving the population unstated is worse because it is silent. On the assistant above, synthetic uptime probes fired every 15 seconds with a 40-token prompt and made up 8% of request volume; they are fast by construction, and removing them from the population moved reported p95 TTFT from 380 ms to 455 ms. Nothing about the service changed. Health checks, warm-up calls, the eval harness replaying yesterday's traffic, and retries counted as fresh requests all bias a percentile in whatever direction their latency profile happens to run, and none of them are the user the objective exists to protect. Aggregation is the other trap: percentiles do not average, so a "global p95" computed by taking the mean of five regional p95 values is not a percentile of anything and can sit well below the true figure.

What these four numbers do not tell you#

None of them says the answer was any good. A model that has started replying "I don't have enough information to answer that" in 40 tokens will improve TTLT, improve output TPS by shortening the decode tail, and reduce cost per request, and every panel on the latency dashboard will look like a win. Truncation at max_tokens does the same thing: a cut-off answer finishes early and reports beautifully.

Per-request output TPS is also not system throughput, and the two move against each other. Raising batch size increases total tokens per second across all concurrent requests — the number a capacity plan cares about — while decreasing the rate any single user sees, which is the tradeoff DigitalOcean's LLM Inference Trilemma piece lays out between throughput, latency and cost. Reporting one figure called "tokens per second" without saying whether it is per request or per cluster will eventually cause an argument in which both sides are quoting real measurements.

The four latency figures share a property that makes them easy to reason about: every request emits its own timestamps, so the SLI is computed from complete data and the only judgement calls are the window and the population. Quality has no such property. There is no timestamp for correctness, nothing in the response says whether the answer was right, and any number you put on it comes from a separate system that looks at a subset of requests and forms an opinion. That subset, and the thing forming the opinion, are what the next chapter has to pin down before a quality objective means anything at all.