Choosing an endpoint by workload

Endpoint choice usually arrives as a single comparison — which one is faster — and that question has no answer, because the two endpoints in front of you differ on two axes. Take a concrete pair, benchmarked properly against each other with the harness from chapter 4: warmed, at a fixed prompt length, output pinned, run from the same region, sampled at several times of day. Endpoint A returns its first token in 180 ms and then generates at 45 tokens per second. Endpoint B takes 600 ms to the first token and then generates at 120. They serve the same model weights at similar prices, and two teams in the same company have to pick one.

The arithmetic that decides between them can be written down, and it matters which units it comes out in. Stated in seconds, the answer is a fact about these two endpoints on the day they were measured, and it expires the next time either one is re-provisioned. Stated as a crossover in output tokens — the length at which A's head start stops covering B's faster generation — it is a fact about the workload instead, and it survives a change of provider, because the next pair of endpoints is fed into the same comparison.

Where the two endpoints cross#

The temptation is to read B as nearly three times faster and stop. B does decode 2.7 times faster, and for most requests it also finishes sooner, which is a real and checkable claim:

def total_s(ttft_s, tok_per_s, n_tokens):
    return ttft_s + (n_tokens - 1) / tok_per_s

A = (0.180, 45)
B = (0.600, 120)

for n in (5, 31, 120, 600, 2000):
    print(n, round(total_s(*A, n), 3), round(total_s(*B, n), 3))
# 5     0.269   0.633
# 31    0.847   0.850
# 120   2.824   1.592
# 600  13.491   5.592
# 2000 44.622   17.258

The crossover is at about 31 output tokens: below that A finishes first because B's extra 420 ms of startup outweighs its faster generation; above it B wins, and the margin grows without limit. Since almost every response longer than a sentence clears 31 tokens, B has lower total response time for nearly all real traffic. Total response time is the quantity chapter 8 established should not be the target, so that result settles less than it appears to — but the crossover itself is worth keeping, because it is the point at which the two speed terms change places.

Where 31 comes from is one line of algebra, and working it is what makes the rule portable to the next pair of endpoints. The two totals are equal at the token count where the head start B gives away at the beginning is exactly cancelled by what it recovers during generation. B gives away 420 ms — the difference between 600 ms and 180 ms — and gives it away once, before anything is produced. It recovers on every token after the first, because one token takes a forty-fifth of a second on A and a hundred-and-twentieth of a second on B, and the difference between those two fractions is what B claws back per token. Divide the 420 ms head start by that per-token recovery and the answer is about thirty tokens, with the response needing one more than that because the first token is delivered by the startup term rather than by the rate. Written that way the rule outlives these particular endpoints: it is always a difference in startup divided by a difference per token, and the units of that division are always tokens.

Startup cost amortised across four hundred thousand documents#

One team's workload is a nightly pass over a document archive: 400,000 documents, each producing a summary of about 600 tokens, written to a warehouse table that a morning report reads. Nobody watches it run. The only requirement with a clock in it is that the table is complete by 06:00.

The decision is made by the generation rate, and the first-token figures barely participate. Endpoint A's 180 ms advantage is paid once per document against 13.3 seconds of decoding, so it is 1.3% of the runtime; even if A's first token were instantaneous the job would still take more than twice as long as B's. Running B needs 104 concurrent workers to make the window against A's 250, which is a difference in coordination overhead and, on a per-token-priced endpoint, in nothing at all — the token bill is identical because the same tokens are generated either way.

Every figure in that table is reachable from the two endpoint numbers, and walking it once is what turns the table into something you can rebuild for your own job. Time per document is the composition formula at 600 output tokens. Multiply it by 400,000 documents and you have the serial time, which is the 1,499 hours for A and 621 for B once seconds are converted into hours. Divide each of those by the six-hour window and you have the worker counts, 250 and 104, because workers are what turn serial time into elapsed time when the items are independent. The TTFT share is the first-token figure divided by the time per document. That last division is the one that decides the comparison: 180 ms against more than thirteen seconds is a small fraction, and a small fraction of the runtime cannot decide anything however good it is. The reason the same 180 ms decides the voice case in the scenario below is that there it is not divided by thirteen seconds of generation. It is nearly the whole budget.

That is the general shape for offline work: startup cost is amortised across a long generation and then across a large number of items, while the rate multiplies against every token. It also explains why batch-oriented APIs exist at a discount. A provider that can defer your work, schedule it when its fleet is otherwise idle, and run it at whatever batch depth it likes will offer a large price reduction — commonly around 50% — in exchange for a completion window measured in hours. That trade is only available because the tail-latency cost of chapter 7, where deeper batches stretched the p95 first token fivefold, is not a cost this workload pays: here the correct operating point is deep into the region where interactive traffic would be suffering.

For a batch team the more consequential numbers are usually not on that table at all. A provider's concurrency limit decides whether 104 workers is even permitted, and an endpoint that is 2.7× faster but caps you at 20 in-flight requests loses to a slower one that allows 200. Failure handling matters more than latency, since a 621-hour job spread over 104 workers will encounter rate limits, timeouts and transient errors, and whether a retry costs one document or the whole shard is a bigger lever on wall-clock completion than the difference between these two endpoints.

The same decision appears in self-hosted form as a configuration rather than a purchase, and it is worth recognising because the mechanism is the batching mechanism of chapter 6 rather than a vendor's pricing. The two teams here are choosing operating points on one throughput/latency curve: an interactive service wants a shallow batch and spare capacity so that no request queues, and a summarisation job wants the deepest batch the KV cache will hold. Running both against a single deployment tuned to one operating point guarantees that one of them is served badly, which is why interactive and bulk traffic are normally given separate replica pools even when they share a model.

Where the crossover cuts across the interactive boundary#

The 31-token crossover is a real decision rule, and it is worth noticing that it does not respect the batch/interactive boundary. A nightly job classifying 400,000 support tickets into one of twelve categories emits about 5 output tokens per item, and at 5 tokens the table above gives A 0.269 seconds and B 0.633 — A finishes that batch job in 40% of the time. Startup cost dominates whenever generation is short, whether or not anybody is watching, so the question to ask is not "is this interactive" but "how many tokens does one call produce, and does anyone experience the interval before the first one".

The rule is also stated in output tokens rather than in seconds, which keeps it usable when the endpoints change. An agent loop that makes eleven model calls per user turn, ten of which emit a tool invocation of about 15 tokens and one of which writes a 60-token confirmation, pays the startup cost eleven times and never generates enough tokens for the rate to assert itself. Summing the two columns for that shape gives 6.40 seconds on A and 8.26 seconds on B, so the endpoint that loses badly on a single long response wins the loop, and the reason is entirely that all eleven calls sit below the crossover.

Both sides of that can be sized without running anything. Eleven calls each paying B's 420 ms of extra startup is well over four seconds of additional waiting, spent before any of the generation happens. Set against it, the whole loop produces only a couple of hundred output tokens, and B's per-token advantage over that many tokens recovers under three seconds of the four. More startup costs than there are tokens to amortise them against is the shape of every agent loop, and it is why a chatty sequence of short calls belongs on the endpoint with the fastest first token even when every published comparison favours the other one. Counting calls and their output lengths is a five-minute exercise that answers the question directly, and it beats any intuition about which endpoint is faster.

Both figures also assume a benchmark that survives chapter 4's objections. A quoted 120 tokens per second is a median on an idle endpoint at some prompt length nobody stated, and the number that governs a 621-hour job is the rate sustained at your concurrency, at your prompt lengths, at 03:00 on the provider's fleet. A p95 TTFT of 4 seconds behind a median of 600 ms would change the voice decision completely and change the batch decision not at all, which is a good illustration of how much of the choice lives in the tail rather than in the headline.

And neither figure says anything about output quality, cost per token, context window, rate limits or whether the provider will still exist in a year. Latency is one input to an endpoint decision and rarely the deciding one; what this topic contributes is the ability to state precisely what a latency number means so that it can be weighed against the rest instead of being quoted as a single word.

Two numbers, and which of them matters is a property of the workload rather than of the endpoint. Startup cost is paid once per call and disappears into a long generation; generation rate multiplies against every token and disappears when there are few of them. The crossover between them is arithmetic anyone can run in five minutes, and running it replaces an argument about which provider is faster with a statement about which one is faster for a call that produces this many tokens with this person waiting.

The practice that follows from the whole topic is short enough to keep in a checklist. Measure the two phases separately, at fixed prompt and output lengths, after warmup. Vary prompt length and output length independently before attributing a regression to either. Generate load open-loop and record queue time next to execution time. Measure at the client when the promise is made to a client. Put first-token latency and sustained rate in the objective and keep total time as a diagnostic. And before choosing an endpoint, a model or a serving config, write down how many tokens a call produces and who is waiting, because those two facts decide which of the numbers in front of you is the one that matters.