Benchmarking a serving endpoint

The interval a client measures — from sending a request to receiving the first visible output — is a sum of several things, only one of which is the model doing arithmetic. Network and TLS are in it, the gateway is in it, and so is queue time: the interval between a request being accepted by the server process and the scheduler actually giving it a slot on the GPU. On an idle server queue time is zero and the reading is nearly pure model cost, which is why an idle measurement looks so clean and why it answers a different question from the one most benchmarks are trying to answer.

This chapter is about closing the distance between those two questions. It takes a stopwatch reading apart into the five components a request passes through, shows where a serving engine reports queue time and execution time as separate series, and then builds the harness that makes a number quotable: warmup, pinned lengths, per-request records, and an open-loop load generator — one that sends at a target arrival rate regardless of how fast responses come back, rather than waiting for each response before sending the next. It closes on what an idle single-request measurement is still the right instrument for, which is more than the framing so far suggests.

What a stopwatch reading is made of#

The scheduler in that diagram is the component inside the serving engine that decides which requests the GPU works on next; a request that has been accepted by the gateway is not yet running, and the interval between those two states is queue time. Under load it is often the largest single term in time to first token, and it is invisible from the client, which sees only a total that got bigger.

The three vantage points available to you measure overlapping but different intervals, and quoting a number without saying which one you used is the root of most benchmark disagreements.

A serving engine exposes the split directly. On a vLLM-style server — vLLM being the widely used open-source inference engine whose scheduler design most current stacks follow — the Prometheus endpoint, an HTTP path that dumps the process's own counters and histograms as plain text for a metrics scraper to collect, carries queue time and time to first token as separate series. Reading both is what turns "TTFT went up" into "TTFT went up because queue time went up, and execution time is unchanged":

curl -s http://serving-host:8000/metrics | grep -E 'queue_time|time_to_first_token' 
# vllm:request_queue_time_seconds_sum        41.83
# vllm:request_queue_time_seconds_count     512
# vllm:time_to_first_token_seconds_bucket{le="0.1"}   38
# vllm:time_to_first_token_seconds_bucket{le="0.25"} 214
# vllm:time_to_first_token_seconds_bucket{le="0.5"}  467
# vllm:time_to_first_token_seconds_bucket{le="1.0"}  508

Mean queue time there is 41.83 / 512 = 82 ms, while the TTFT histogram says well over half of requests took longer than 250 ms and roughly one in eleven took longer than half a second. Those two facts together say the queue is doing most of the damage at the tail, which points at admission and concurrency rather than at the model. Behind a hosted API you do not get this endpoint, and the substitute is to run your load generator at two arrival rates and attribute the difference, which chapter 7 does with a concurrency cap.

Both of those readings rest on two conventions, and these are the shapes almost every serving metric arrives in. A _sum and a _count pair is a running total of observed values and a running number of observations, so dividing the first by the second gives a mean — 41.83 seconds of accumulated waiting spread over 512 requests is 82 ms each. Notice what that mean cannot do. It is cumulative since the process started, so a bad minute an hour ago is still inside it and will stay there; a recent value has to be obtained by differencing both series between two scrapes and dividing the differences. A _bucket series is cumulative in a different sense. Each line counts the requests whose value was at most its le threshold, so the buckets are nested rather than exclusive. The count falling between two thresholds is the difference of their two numbers, and the count above a threshold is the total minus that bucket's number, which is the arithmetic behind both of the statements above.

A benchmark you would be willing to quote#

The fixes are not subtle, and none of them requires infrastructure beyond the harness from chapter 3, where a grid of fixed prompt and output lengths separated the two phases.

Warm the server before recording anything. The operational form is a few dozen requests of the same shape as the measurement, with their results thrown away, and a check that the numbers have stopped drifting downward before the recorded run starts. The size of the effect is easy to underestimate. On a freshly started 8B server, a sequence of identical 2,000-token requests measured 2,410 ms, 640 ms, 310 ms, 190 ms and then settled between 95 and 105 ms from about the twentieth request onward — a factor of 25 between the first reading and the steady state, with CUDA graph capture and allocator warmup accounting for most of the early drop and the prefix cache for the rest. A benchmark that took three samples and averaged them would have reported 1.1 seconds.

That last sentence should be checked rather than taken, because it is the most common way this goes wrong in practice. Add the first three readings — 2,410 ms, 640 ms and 310 ms — and divide by three, and the mean is about 1.1 seconds. The steady state is between 95 and 105 ms. So a careful-looking procedure, three samples and an average, reports a number more than ten times the truth, and reports it with the reassuring air of something that was measured repeatedly. The averaging is what does the damage: the first reading is an outlier by a factor of 25, and a mean over three values cannot suppress an outlier that large, since one value in three carries a third of the weight no matter how extreme it is. Discarding a fixed number of early samples works where averaging does not, because it acts on the reason those samples are wrong rather than on their values.

The same effect operates in production on any autoscaled deployment, where it is not a measurement artefact but a real user experience. A pod that has just started serves its first requests at multi-second TTFT, so a scale-up event triggered by a traffic spike delivers its worst latency to exactly the traffic that caused it. The mitigations are to warm new replicas with synthetic traffic before putting them in the load balancer's rotation, and to scale on a signal that leads demand rather than one that lags it.

Pin both lengths, with the actual counts recorded per request, because otherwise you are comparing verbosity rather than speed. A benchmark result that does not state the prompt length and output length it used is not reproducible and not comparable to anything. Record TTFT, inter-token latency and total separately, per request, with timestamps rather than aggregates: percentiles computed from stored per-request values can be recomputed later at a different cut, while a stored mean cannot be turned back into a p95.

Generate load open-loop. This is the one people get wrong most often, and the failure mode is specific enough to have a name. A closed-loop generator runs N workers that each send a request, wait for the response and send the next; when the server slows down, the generator sends fewer requests, so the offered load falls exactly when the queue would have formed. The measurement hides the queueing it was built to find, which is Gil Tene's coordinated omission. An open-loop generator sends at a target arrival rate regardless of how fast responses come back, so a slow server accumulates the backlog it would accumulate in production.

The mechanism has two parts, and the second is where the real damage is done. The first is a feedback loop. With a fixed number of workers, each calling send() and then blocking until its response arrives, the number of requests in flight can never exceed the worker count. When the server slows, every worker waits longer, so the generator issues fewer requests per second. Offered load is therefore a function of the server's own response time, and it falls at exactly the moment a queue would otherwise have built.

The second part is a sampling bias, and it survives even when the average rate comes out looking correct. Suppose the server stalls for a second. A closed-loop generator with a handful of workers records that stall in a handful of samples, because a handful of requests is all it had outstanding. The requests that would have arrived during that second — at any realistic arrival rate, many more than a handful — are never sent, so they never enter the distribution as the very slow requests they would have been. The stall is recorded once per worker instead of once per affected user. A percentile computed from that sample is computed over a population from which the worst-affected requests have been removed, and removed in proportion to how bad the stall was. That is what coordinated omission means: not a tail that is measured noisily, but a tail that has been deleted by the measurement process itself.

import asyncio, random, time

async def open_loop(rate_per_s: float, duration_s: float) -> list[dict]:
    results, tasks = [], []
    deadline = time.perf_counter() + duration_s
    while time.perf_counter() < deadline:
        tasks.append(asyncio.create_task(one_request(results)))
        # Poisson arrivals: exponential gaps, not a fixed tick
        await asyncio.sleep(random.expovariate(rate_per_s))
    await asyncio.gather(*tasks)
    return results

async def one_request(results: list[dict]) -> None:
    t_send = time.perf_counter()
    r = await measure_streaming(PROMPT_2000, max_tokens=300)
    results.append({**r, "sent_at": t_send})

Exponential gaps rather than a fixed tick matter because real arrivals clump, and a perfectly even arrival process understates queueing at any given mean rate. Sending one request every 40 ms and sending an average of 25 per second with exponential spacing produce noticeably different tails on the same server.

Finally, run long enough to reach steady state and report the run's own health. A 30-second run at 25 requests per second is 750 samples, which is enough for a p95 and thin for a p99. Record whether the generator kept up with its target rate — if it fell behind, the number you have is not the load you configured, and the result is closer to a closed-loop measurement than an open-loop one. A result that satisfies all of that is compact enough to paste into a ticket, and it is worth writing down in a fixed form so that two runs a month apart are comparable:

model=llama-3.1-8b-instruct  engine=vllm-0.6.3  gpu=1xH100-80GB
prompt=2000 tok (fixed)  output=300 tok (fixed, min_tokens set)
warmup=40 req discarded   run=60 s open-loop, Poisson, target 20 req/s
achieved 19.8 req/s       5,940 output tok/s aggregate

              p50      p95      p99
queue          31 ms   210 ms   480 ms
TTFT          128 ms   334 ms   611 ms
ITL             7.9 ms  11.2 ms  26.0 ms
total          2.49 s   3.41 s   4.20 s

Every quantity a later argument might need is on that block: the two lengths, the arrival process, the achieved rate against the target, and the queue split that says how much of the 334 ms p95 TTFT was waiting rather than computing.

What an idle measurement is still good for#

None of this makes single-request timing useless. It measures model execution cost with everything else removed, which is exactly what you want for a regression check: run the chapter 3 grid on every model or engine upgrade, on an idle box, and a 15% rise in the prefill slope is a clean signal that something in the model or the kernels changed. It is a controlled experiment, and its control is the reason it cannot answer questions about production.

The distinction worth holding is between a benchmark that measures a component and one that predicts an experience. An idle single-request number is the former, and calling it unrepresentative is a statement about which of those two jobs it is being asked to do rather than a criticism of the number. It tells you what the machine can do when nothing is in the way, which bounds what your service can do and is the right input to a capacity model. It does not tell you what your users will see, because your users arrive concurrently, at uneven rates, with prompts of varying length, into a queue.

Two further limits are worth stating plainly. A benchmark of your provider from your laptop measures your laptop's network as much as anything else, and the fix is to run it from the environment your service runs in — a 40 ms round trip from a home connection is nearly half of an idle TTFT and will swamp exactly the differences you are trying to detect. And a benchmark against a hosted API measures that provider's current load, not your traffic; run it at several times of day before quoting a number, because the tail on a shared endpoint moves with other people's traffic and you have no visibility into it at all.

The through-line is that a latency number is only interpretable together with the conditions that produced it. Warmup state, prompt and output length, arrival process, achieved rate and vantage point are not caveats attached to a measurement; they are part of what the measurement means, and a number quoted without them can be reproduced only by accident. Writing them down converts a benchmark from an argument into evidence, and the fixed-form block above exists so that the writing down takes a minute rather than an afternoon.

Every measurement in this chapter has been of when tokens leave the server. The client's experience begins when tokens arrive and are shown, and between those two points sit a proxy, a compression layer and a rendering loop, any of which can hold a token that the server considers delivered. That gap is where streaming either pays off or quietly fails to, and it is the next chapter.