Self-hosting break-even volume

Renting a GPU to serve a model you downloaded is priced in a unit that has nothing in common with the one on a provider's pricing page. Hosted inference is sold per token, so the bill is proportional to work done and is zero when you are idle. Self-hosted inference is sold per hour, so the bill is proportional to time and is identical whether you serve two million tokens in that hour or none. The comparison people want to make — is self-hosting cheaper — is not a comparison between two prices, because only one of the two is a price.

Converting the hourly rate into a per-token rate is a single division, and it needs two quantities the pricing page cannot supply: how many tokens per second the model actually achieves on that hardware with your sequence lengths and batch size, and what fraction of the hours you pay for are hours you are using. This chapter measures both, computes the resulting cost per million tokens across the range of plausible utilisations, and turns the answer into a break-even volume stated in requests per day, which is the form the question can actually be argued about in. It closes on the costs that sit outside the division and the decisions the division cannot make.

Turning dollars per hour into dollars per million tokens#

The arithmetic is one division performed twice.

cost per token = (instance $/hour) / (tokens per second × 3600 × utilisation)

Read that line as a unit conversion rather than a formula, because every term in it is there to make the units cancel. Dollars per hour divided by tokens per hour leaves dollars per token, so the only work is turning the throughput you measured into tokens per hour. Tokens per second becomes tokens per hour by multiplying by the number of seconds in an hour, which is the 3600. That product is what the replica would produce if it never stopped.

The utilisation term is the one that has to be argued for rather than looked up. You are billed for every hour the instance exists, including the hours it produces nothing, so the denominator has to be the tokens produced per hour paid for rather than per hour working. Multiplying by the fraction of hours that are working hours converts one into the other. Because it sits in the denominator and nothing else in the expression moves with it, halving utilisation doubles the cost per token exactly — which is why the rows of the table below are inverse multiples of one another rather than a curve anybody had to fit.

Throughput is the term people guess at and should measure, because the number that matters is aggregate output tokens per second under concurrency, not the single-stream speed that shows up in demos. A 70B model at bfloat16 across two H100s generates roughly 25 to 30 output tokens per second for one request, which is a latency figure. Run 32 concurrent requests through the same replica and aggregate output throughput lands somewhere near 900 tokens per second, because decoding is limited by memory bandwidth and batching amortises each weight read across many sequences at once. That is a factor of thirty between the number a single test gives you and the number your cost model needs.

That factor is worth deriving rather than accepting, because it is the whole reason self-hosting can be priced at all. Producing one token means reading every weight in the model out of the card's memory and doing a small amount of arithmetic with each one. The reading is slow and the arithmetic is fast, so the step takes about as long as the read takes, whatever is attached to it. Serve one request and a full sweep of the weights has bought you a single token. Serve thirty-two at once and the same single sweep buys a token for every one of them, because the weights being read are identical for all the sequences and only the small per-sequence state differs. The wall clock per step barely moves while the tokens per step multiply, so the aggregate rate climbs by close to the batch factor. Close to it rather than exactly, which is why thirty-two concurrent requests yield about thirty times the single-stream rate: the arithmetic and the cache traffic that genuinely do scale with batch eventually begin to cost something.

The instruction hidden in that derivation is to distrust any throughput figure quoted without a concurrency beside it. A demo streams to one viewer and shows the latency number; a cost model divides by the aggregate number; and the two describe identical hardware serving an identical model while differing by more than an order of magnitude.

Input and output are not equivalent here either, for the same reason they are priced differently by providers. Prefill processes the whole prompt in parallel and runs at tens of thousands of tokens per second on the same hardware; decode is one token at a time. On self-hosted infrastructure it is common to convert input tokens into an output-equivalent at roughly a tenth of their count and then reason entirely in output tokens, which keeps the model simple without distorting it much for typical prompt lengths.

That conversion factor is a blunt instrument, and it is worth knowing which way it is blunt. Setting the two measured rates against each other — tens of thousands of tokens per second for prefill against hundreds for decode — would justify treating an input token as worth far less than a tenth of an output one. Counting it at a tenth therefore charges input more than the raw throughput ratio says it deserves, which is the safe direction to be wrong in when the output of the exercise is a budget, and it leaves room for the prefill work that grows faster than linearly on long prompts. What the shortcut does not survive is a workload whose prompts dwarf its answers, where the input term stops being a correction and becomes the bill.

Measuring throughput takes an afternoon and removes the largest source of error in the whole estimate, because published numbers are almost always produced at sequence lengths and batch sizes chosen to look good. A load generator pointed at your own replica, with prompts sampled from your own traffic, produces the figure you should divide by:

$ vllm bench serve --model some-org/some-model-70b \
    --dataset-name random --random-input-len 2048 --random-output-len 320 \
    --max-concurrency 32 --num-prompts 640

============ Serving Benchmark Result ============
Successful requests:                     640
Request throughput (req/s):              2.81
Output token throughput (tok/s):         898.4
Total token throughput (tok/s):          6653.1
Mean TTFT (ms):                          842.3
P99 TTFT (ms):                           2104.7
Mean TPOT (ms):                          34.9
==================================================

The two lines that go into the cost model are output token throughput and, as a constraint on it, P99 time to first token, which is the delay before the first token of a reply appears for the slowest one request in a hundred. The mean TPOT underneath it is time per output token, the gap between successive tokens that a user reads as typing speed. Running the same command at concurrency 8, 16, 64 and 128 traces the curve those two quantities move along, and the cheapest point that still satisfies your latency target is the configuration you are allowed to price. Input length matters as much as concurrency here: rerunning with --random-input-len 8192 on a retrieval workload will typically cut output throughput noticeably, because prefill work and cache pressure both grow.

Utilisation is where most estimates quietly break. A replica sized for the daily peak sits mostly idle at three in the morning, and traffic to an internal business tool typically concentrates into eight or nine hours on five days a week, which is around 25% of the wall clock even before allowing for the peak-to-mean ratio inside the working day. Autoscaling helps less than it does for stateless web services because a large model takes minutes to load into GPU memory, so scaling down aggressively means requests arriving at a cold replica.

HOURS_PER_MONTH = 730

def cost_per_million_output_tokens(hourly_rate, tok_per_sec, utilisation):
    tokens_per_hour = tok_per_sec * 3600 * utilisation
    return hourly_rate / tokens_per_hour * 1_000_000

def monthly_self_host_cost(hourly_rate, replicas, extras_per_month=0.0):
    return hourly_rate * replicas * HOURS_PER_MONTH + extras_per_month

# 2x H100 @ $4.50/hr, 900 output tok/s aggregate at batch 32
for u in (0.10, 0.25, 0.50, 0.80, 0.95):
    print(u, round(cost_per_million_output_tokens(9.00, 900, u), 2))

Work one row by hand and the rest are the same row scaled. The replica in question is two H100s at $4.50 an hour, so the instance costs nine dollars an hour, and it produces 900 output tokens per second when it is busy. Nine hundred tokens a second for the thirty-six hundred seconds in an hour is three million two hundred and forty thousand tokens an hour at full occupancy. At the 10% row only a tenth of the hours are busy ones, which is the 324,000 in the second column. Nine dollars buys those 324,000 tokens, so a million tokens costs nine dollars multiplied by however many times 324,000 goes into a million — a little over three — and that is the $27.78.

Every other row is that arithmetic with one number changed, which is why the column reads the way it does. Going from 10% to 50% multiplies the busy hours by five and divides the cost by five, and $27.78 divided by five is the $5.56 four rows down. This is the single most useful thing the table says, and it is easy to read straight past: on self-hosted hardware the price of a token is set by how often the machine is working, not by how fast it works when it is. A team that doubles its throughput with a better kernel and leaves the replica idle overnight has bought half of what a team that filled the night shift bought for nothing.

Set those against a marketplace host serving the same open-weight model at about $0.60 per million output tokens and the conclusion is uncomfortable: at no utilisation on this configuration does a self-hosted replica beat the hosted price of the same weights. That is a genuine result and it is specific to this configuration rather than universal, because every term in the division is a choice.

Which levers actually move the number#

Quantisation moves it the most. Serving the same 70B model in FP8 halves the memory footprint, fits it onto a single H100 rather than two, and increases decode throughput because there is less to read from memory per token. A single card at $4.50 an hour delivering around 700 output tokens per second at batch 32 gives $2.48 per million at 80% utilisation and $9.92 at 20% — roughly a threefold improvement over the bfloat16 pair, bought with a quality change you now owe yourself a measurement of.

Two things about that pair of figures are worth pausing on. The improvement comes from the numerator and the denominator at once: one card instead of two halves the hourly rate, and fewer bytes to read per token raises the throughput, and the division multiplies the two gains together rather than choosing between them. And the two utilisations quoted are the same computation, not two measurements — 20% is a quarter of 80%, so the cost is four times as large, and $9.92 is exactly four times $2.48. Anyone quoting a self-hosted unit cost without saying which occupancy it assumes has left out the term with the widest range in the whole expression.

Model size moves it almost linearly in the other direction. An 8B model on one H100 will serve several thousand output tokens per second at high batch, which puts it in the region of $0.30 to $0.60 per million at good utilisation, comparable with or better than hosted pricing. Most credible self-hosting economics involve small models at high volume rather than large models at moderate volume.

Batch size trades cost against latency, and this is the trade a cost model computed in isolation gets wrong. Raising concurrency from 8 to 64 might take aggregate throughput from 400 to 1,300 tokens per second, which is a 3.25x cost improvement, while time-to-first-token rises from 180 ms to 900 ms and inter-token latency roughly doubles.

A throughput ratio converts directly into a cost ratio here, which is worth saying explicitly because it is the reason this lever is worth pulling at all. The hourly rate does not change when you raise concurrency — it is the same card, rented for the same hour — so the numerator of the division is fixed and the whole effect lands in the denominator. Cost per token is therefore inversely proportional to throughput, and 1,300 divided by 400 is the 3.25 quoted above. Notice also what that ratio is not: concurrency went up eightfold and throughput only a little over threefold, because the amortisation described earlier runs out. Past some batch size the weight reads are no longer the thing you are waiting for, and further sequences buy proportionally less. If your product has a p95 latency target — the ceiling that ninety-five per cent of requests have to come in under — that target sets a maximum batch size, and the maximum batch size sets your cost floor. Any self-hosting estimate not stated alongside a latency constraint is quoting the cheapest point on a curve you cannot operate at.

Context length reduces achievable batch size, because key-value cache memory scales with sequence length and competes with the weights for the same VRAM. A configuration that sustains batch 32 at 4k tokens of context may only sustain batch 6 at 32k, which raises per-token cost by a factor of four or five for the same hardware.

The chain there has three links and each one is checkable. The cache holds one entry per token the sequence has seen, so eight times the context is about eight times the cache for each sequence. The weights occupy a fixed amount of the card's memory whatever the context length is, so the memory left over for cache is a fixed budget, and a fixed budget divided among sequences that each want eight times as much holds far fewer of them. Fewer concurrent sequences means proportionally less aggregate throughput, and less throughput at an unchanged hourly rate means proportionally more cost per token. Following it through gives the factor quoted: batch 32 falling to batch 6 is a little over a fivefold drop in the denominator. Long-context self-hosting is a different cost regime from short-context self-hosting, and averaging across a workload that contains both will mislead in whichever direction the tail is.

Hardware rate is the last lever and the one with the widest spread. On-demand pricing from a hyperscaler for an H100 sits around $4 to $10 an hour depending on region and instance family; a one-year reservation typically cuts that by 40 to 60%; specialist GPU clouds and spot capacity can go lower still, at the cost of eviction risk and less certainty about supply. A reserved instance also removes the option to stop paying, which pushes the utilisation term back into the discussion: reserving capacity you use 25% of the time is buying a fixed cost to reduce a rate you were not benefiting from.

The costs that sit outside the division#

Three costs sit outside that break-even and belong in the table anyway, because they are the ones that turn a marginal win into a loss:

Three beliefs get people to the wrong side of that division, and each is tempting for a reason worth naming. The first is that idle capacity is free, which is true of an API and false of an instance: the hosted bill is proportional to work and the rented bill is proportional to time, so the quiet weekend that costs nothing in one column costs full price in the other. The second is that a reservation makes the hardware cheaper. It makes the hourly rate lower and simultaneously removes the ability to stop paying, so a reservation used a quarter of the time has bought a discount on hours you were not benefiting from — the utilisation term and the rate term move against each other, and only the division tells you which won. The third is that a faster model is a cheaper one. Speed is one term among four, and a configuration that doubles throughput while halving the batch size a latency target permits has moved nothing at all.

The last row, engineering time counted in full-time-equivalent engineers, is the one that decides most real cases and the one a per-token comparison structurally cannot see. A per-token figure compares two ways of buying inference; it does not compare two ways of running a company, which is why a break-even that comes out only slightly in favour of self-hosting is usually not a reason to self-host.

It is also worth being precise about what this arithmetic does not decide. Break-even is a cost argument, and cost is frequently not why an open-weight deployment is on the table. Data residency requirements, an air-gapped customer environment with no network route to the public internet at all, a need to fine-tune weights you keep, latency floors that a network round trip cannot meet, or a regulator who wants the model artifact under your control are all reasons to self-host at a price premium, and none of them is refuted by a number showing the API is cheaper. What the break-even gives you in those conversations is the size of the premium, so the decision is made knowing what it costs rather than believing it costs nothing. Chapter seven takes that further, into what the commitment obliges you to beyond the invoice.

The output of the exercise, then, is a volume at which two curves cross rather than an answer about which option is better. Hosted cost is a line through the origin with slope equal to the per-token price; self-hosted cost is a flat line at the monthly instance charge plus a shallow slope for marginal traffic, up to the capacity of the replica. Below the crossing point the API is cheaper and above it the replica is, and both positions move when you change quantisation, batch size, context length or the hardware contract. A team that has computed the crossing point for its own configuration and its own latency target can argue about a number; a team that has not is arguing about a threshold nobody has located.

That fills the deployment and price columns of a candidate table with defensible cells. The quality column is still empty, and the material available to fill it — model cards, leaderboard positions and benchmark claims — is written by people who want you to pick their model.