Variant selection and tail latency

A variant decision usually arrives as a comparison of two model names and ends as a preference. It has six dimensions, three of which are close to binary and therefore disqualify rather than trade, and all six are measurable on your own task in an afternoon. Modality support is one: a model either accepts the input you have, through the content-part path a multimodal checkpoint expects, or it is not a candidate. Structured-output support is another, and what it means here is schema-constrained decoding, where the server restricts each sampled token to those that keep the response valid against a JSON schema. Latency against a stated budget is the third, and it is the one people treat as tradeable when it is not.

This chapter is about measuring the trade rather than arguing it. p50, p95 and p99 are the 50th, 95th and 99th percentiles of a measured time — the duration below which that share of requests finished — and a requirement written at p95 is a requirement about the tail, which is exactly where a reasoning variant's behaviour differs most from an instruct variant's. What follows is the arithmetic that makes the ordering unavoidable, the harness that produces the numbers, the table that records the decision, and the conditions under which routing between two variants beats choosing one.

Why the thinking arrives first#

Take a multimodal request carrying one photographed document: a 1,176-token image plus a short instruction. Prefill — the single pass over the whole prompt that must complete before any output token can be sampled — takes about 0.4 seconds on a current inference GPU. Decoding then runs at roughly 55 tokens per second per request under production concurrency. A reasoning trace of 640 tokens therefore puts the first visible character about 12 seconds after the request left the device, and no amount of streaming, spinner design or connection pooling changes that ordering, because the thinking tokens are not overhead around the answer — they are the mechanism that produces it.

The gap between those two numbers is worth deriving rather than quoting, because the derivation is what makes the ordering structural instead of merely unfortunate. Every token in the prompt already exists when the request arrives, so the hardware can work on all of them together and finish in a single pass. That is prefill, and it is why a 1,176-token image costs a fraction of a second. A generated token does not exist until the token before it has been chosen, because it is scored from a sequence that includes its predecessor. Generation therefore cannot be collapsed into one pass; it is one pass per token, in order, with no way around it. A reasoning trace is generated text like any other. So it pays the per-token rate, and it pays it before the answer begins, because the answer is scored from a sequence that has the trace inside it.

Measured on 200 real delivery notes at a concurrency of 8, a fast multimodal instruct model and a multimodal reasoning model of similar size produced these distributions:

The p50 numbers are bad and the p99 is the one that decides. Thinking length is data-dependent and its distribution has a long right tail: on the delivery notes that were creased, partially in shadow or handwritten, the reasoning model did not fail — it thought harder, and 3% of the sample ran into the 4,096-token reasoning cap, which at 55 tokens per second is 74 seconds of generation for a request that then returned a truncated answer. An instruct variant's output length is bounded by the schema it is filling, so its distribution is narrow by construction: 38 tokens at p50 and 61 at p99, which is why its p95 sits close to its median. Two systems can have similar means and completely different tails.

"Bounded by the schema" and "decided by the model" are two different kinds of limit, and the distance between them is the whole of the tail. An instruct variant filling three named fields writes three named fields. The shape of its output is settled before generation starts, so the only thing left to vary is how long each value happens to be, and values on a delivery note are short. A reasoning trace has no shape imposed on it from outside. It ends when the model samples the token that ends it, and how soon that token becomes likely depends on what the trace has said so far, which depends on the document. One distribution is bounded by something you wrote down and the other by something the input decides, and averaging does not turn the second into the first.

Four points of exact match is a real difference and deserves a real answer rather than a dismissal. The question is what those four points are worth against a fifteen-fold latency increase and a fivefold cost increase, and that is a product question with a computable input: at 5,000 documents a day, 200 of them would be extracted wrongly by the fast model and correctly by the slow one, each costing perhaps 40 seconds of manual correction, against 5,000 people waiting an extra ten seconds every time.

The harness that ends the argument#

None of those numbers come from a public benchmark, and none of them could. Benchmarks measure a different task on different inputs at a concurrency of one, and what decides a choice like this is the interaction between one document distribution, one schema and one load. The harness is about eighty lines.

import asyncio, json, time, statistics
from openai import AsyncOpenAI

client = AsyncOpenAI(base_url=BASE_URL, api_key=KEY)
sem = asyncio.Semaphore(8)            # match production concurrency

async def run_one(model: str, item: dict, extra: dict) -> dict:
    async with sem:
        t0, ttft, text, usage = time.perf_counter(), None, "", None
        stream = await client.chat.completions.create(
            model=model,
            messages=[{"role": "user", "content": [
                {"type": "text", "text": SCHEMA_INSTRUCTION},
                {"type": "image_url",
                 "image_url": {"url": f"data:image/jpeg;base64,{item['b64']}"}},
            ]}],
            max_tokens=4096, temperature=0,
            stream=True, stream_options={"include_usage": True},
            **extra,
        )
        async for chunk in stream:
            if chunk.usage:
                usage = chunk.usage
            if chunk.choices and chunk.choices[0].delta.content:
                if ttft is None:
                    ttft = time.perf_counter() - t0
                text += chunk.choices[0].delta.content
        return {
            "ttft": ttft, "e2e": time.perf_counter() - t0,
            "out": usage.completion_tokens,
            "hidden": getattr(usage.completion_tokens_details, "reasoning_tokens", 0),
            "ok": parse_and_compare(text, item["truth"]),
        }

def pct(xs, p):
    xs = sorted(xs)
    return xs[min(len(xs) - 1, int(round(p / 100 * len(xs))) - 1)]

async def evaluate(model: str, items: list[dict], extra: dict | None = None) -> dict:
    rows = await asyncio.gather(*(run_one(model, i, extra or {}) for i in items))
    e2e = [r["e2e"] for r in rows]
    return {
        "model": model,
        "exact_match": sum(r["ok"] for r in rows) / len(rows),
        "ttft_p50": round(pct([r["ttft"] for r in rows], 50), 2),
        "e2e_p50": round(pct(e2e, 50), 2),
        "e2e_p95": round(pct(e2e, 95), 2),
        "e2e_p99": round(pct(e2e, 99), 2),
        "median_out": statistics.median(r["out"] for r in rows),
        "median_hidden": statistics.median(r["hidden"] for r in rows),
    }

Three details in that code are what make the output trustworthy. The semaphore is set to production concurrency, because a model measured one request at a time on an idle GPU reports a latency it never achieves under batching, and the gap is usually a factor of two or more. stream=True with stream_options={"include_usage": True} is how you get both the time to first visible token and the usage record in one pass; the reasoning-token count arrives only in the final chunk, and without it you are measuring cost blind, since hidden tokens are billed and do not appear in the text you received. And parse_and_compare does an exact comparison against ground truth rather than a similarity score, for the same reason a serialisation bug is best caught by exact match: the failures here produce confidently wrong values, not near misses.

The concurrency detail deserves its own step, because "measure under load" reads as advice when it is really arithmetic. A serving stack fills each batch from whatever requests are waiting, and every request in a batch advances by one token per pass. A batch of eight therefore produces eight tokens per pass rather than one, and each individual request waits for the whole batch to step. What goes up is throughput; per-request latency goes up too, because your tokens now share every pass with seven other people's. A model measured alone on an idle GPU is reporting the latency of a batch of one, which is a figure no production request will ever experience.

Run it once per candidate, and once more per candidate with the knobs that change the trade — reasoning_effort="low" where the API exposes it, or an explicit thinking-token budget. On the delivery notes, capping the budget at 256 tokens brought p95 to 6.1 seconds and exact match to 0.93, better than the default configuration in both respects.

Writing the decision down#

Written this way the decision takes a minute and the record survives the next person who asks why the more capable model was not chosen. It also makes the conditions for revisiting explicit: if the latency budget moves to ten seconds, or a thinking-budget parameter brings p95 under two, the disqualification lapses and the remaining dimensions become a straightforward cost-for-quality trade somebody can price.

The word "disqualifies" in that last column is doing something different from "trades", and the difference is structural rather than a matter of emphasis. A trade is an exchange between two quantities that have been priced against each other, so four points of accuracy against five times the cost is a conversation somebody can actually hold. A requirement is not a term in that sum. It is a condition on which options enter the sum at all, and a candidate that fails one has not scored badly — it has been disqualified, and there is nothing left to weigh it against. Treating a p95 budget as tradeable is how a design review reaches agreement on a candidate that cannot do the job, because a requirement written as a number looks exactly like every other number on the page.

Routing rather than choosing#

The interesting design is not one model but two, and its viability is decided by arithmetic people usually get backwards. Send everything to the fast instruct variant, validate the result — schema conformance, a checksum, a confidence signal — and escalate the failures to the reasoning model. The fast path serves the common case inside its budget and the slow path cleans up the hard documents.

Whether that satisfies a p95 requirement depends entirely on the escalation rate, because escalated requests pay both latencies. If 8% of requests escalate and escalation costs 11 seconds, then 8% of the distribution sits above 12 seconds, and the 95th percentile lands inside that group: the combined p95 is roughly 12.3 seconds, which fails. The escalation rate has to stay below 5% for the p95 to remain on the fast path at all, and comfortably below it — around 2% — for the number to be stable as the input mix drifts. Measuring that rate is the first thing to do, before building any of the routing, because it decides whether the design exists.

The step carrying that argument is what a percentile actually is. Sort every request in a run by how long it took, and the 95th percentile is the request standing nineteen-twentieths of the way up the list — the one with exactly one request in twenty slower than it. Escalated requests are the slowest group by a wide margin, so they sit at the top of that sorted list together. If they make up more than one request in twenty, then the request nineteen-twentieths of the way up is still inside that group, and the reported p95 is an escalated request's latency. Below one in twenty, the boundary falls back into the fast group and the p95 is a fast request's latency. Nothing smooth connects those two states. The number jumps as the escalation rate crosses the threshold, which is why the rate has to sit comfortably under it rather than near it.

The escalation signal has to be cheap and honest, which rules out most of what people reach for first. Asking the fast model how confident it is costs another round trip and correlates weakly with correctness. Sampling the fast model three times and escalating on disagreement triples the cost of the common path to detect a case that occurs twice in a hundred. What worked on the delivery notes was structural validation with no model involved: does the output parse against the schema, does the consignment number match the format and its check digit, is the item count an integer within range, and did the vision path report a low-resolution or heavily rotated image. Those four checks caught 71% of the fast model's errors at a cost of under a millisecond, and flagged 3.4% of requests — inside the 5% ceiling, and close enough to it that the escalation rate became something to monitor rather than assume.

Two properties earned those checks more trust than their crudeness suggests. They are computed from the output itself rather than from the model's opinion of the output, so there is nothing in them that can be talked round. And they run on every request, which makes the escalation rate a measurement taken continuously rather than an estimate taken once. On a depot handling tens of thousands of consignments a week, the mix of creased, shadowed and handwritten notes moves with the season and the weather, so a rate that sat inside the ceiling in March is a fact about March.

The limits of a 200-item sample#

A harness measures the candidates you gave it on the sample you gave it, and both halves are limits. A 200-item sample estimates a 0.91 accuracy to about ±2 points, so a two-point difference between candidates is not a result; a four-point lead is worth taking seriously and a one-point lead is not. The sample also has to contain the hard cases in their real proportion, since a set drawn from clean scans will show two models as equivalent and mislead you about exactly the tail that matters.

A harness has a shelf life as well. Model versions change under a stable alias, providers adjust default reasoning budgets, and image-token accounting has been revised more than once, so a decision made from numbers taken in March is a decision made from numbers that no longer exist by autumn. Wiring it into continuous integration against a fixed sample, run weekly, costs a few dollars and turns the whole question into a regression test — which also catches the failures the earlier chapters were about, because a template change or a doubly-rendered prompt shows up in that same table as a quality drop with a suspiciously unchanged token count.

What none of it decides is whether the requirement is right. Two seconds at p95 was chosen by a product manager who watched drivers use an app in the rain, and it may be too strict or too lenient. The harness makes that a conversation about a number rather than about model preferences, which is the value of building it: when someone proposes a variant, the response is to add a row.

Tail latency is where a reasoning variant's cost is paid, and it is paid before the answer starts rather than around it. Because thinking length varies with the input, the gap between a variant's median and its 99th percentile is not noise to be smoothed away but a property of the variant on your data, and a requirement written at p95 is a requirement about precisely that property. Six dimensions, three of which disqualify, measured on two hundred of your own items, is enough to settle the choice and enough to say what would change it.

With the four variants distinguished, the serialisation made visible, the marker set controlled at ingestion, the code paths diffable and the candidates measurable, the remaining work is ordinary engineering. Pin the model string and its template together, keep a fixed test set current, run the harness when either changes, and treat any argument about which variant to use as a request for one more row in the table.