Half the request is tokens that never change

Three proposals were on the whiteboard when the meeting started. One engineer wanted to move the whole feature to a cheaper model, because a comparison post had shown an 80% price difference. Another wanted to compress the retrieved context with a small summariser, having read that prompt compression can cut input by 20×. The third wanted to cap output length, on the grounds that the assistant was verbose and everyone found it annoying. The line item they were arguing about was $75,000 a month, the meeting had an hour, and none of the three proposals came with a number attached to this feature.

None of the three had named the thing they were arguing about, and naming it plainly is what makes the argument settleable. Cost optimization for a model-backed feature means lowering what that feature pays its provider for each unit of work it does, and the individual changes available are conventionally called levers. Five of them cover most of the ground. Prompt caching bills a repeated block of input at a fraction of list price after the first send, and its mechanics were the subject of 15.6. Model tiering sends the easy share of traffic to a smaller, cheaper model, which is the routing machinery 15.8 built. Prompt compression shrinks the text you send before you send it, usually with a second, smaller model doing the shrinking. Output-length control reduces how much the model writes. Batch endpoints trade a turnaround measured in hours for a discount on the whole request. A cost optimization portfolio is that set held in view together — each lever with its expected saving, the part of the bill it acts on, what it costs in quality, latency or engineering weeks, and what it does to the value of the other four.

Holding them together is what the work requires, because a model bill behaves differently from the rest of an infrastructure bill. A server charges for the hours it is running, so a wasteful line of code costs nothing extra until the fleet has to grow; a model provider charges for every token sent and every token generated, so a prompt that carries 800 tokens more than it needs appears on the invoice in the same month it ships, multiplied by traffic. That makes model spend the infrastructure line that engineering decisions move immediately and the one that grows fastest while nobody is watching it. You meet the portfolio the way the meeting above met it: a finance review flags the provider line item, somebody forwards a post promising 90%, and a date gets attached to a percentage before anyone has checked which part of the bill that percentage applies to.

The way out of that meeting is to decompose one request. Not the invoice, not the month — one request, in tokens, at the prices you are actually paying. Everything else in this topic follows from that decomposition, because a lever is only worth what the term it discounts is worth, and until the terms are on the board every proposal is equally plausible.

What one request costs, term by term#

The feature is a retrieval-augmented assistant. Each call sends about 8,000 input tokens, of which roughly 6,000 are a fixed block — the system instruction, a style guide, twelve tool definitions with their JSON schemas, and four worked examples — and the remaining 2,000 are retrieved passages and the user's message, which differ every time. The model generates about 900 tokens. Input is $3.00 per million tokens, output is $15.00.

At two million requests a month that is $75,000, which matches the invoice closely enough to keep going. That comparison is worth doing explicitly before anything else: divide last month's invoice by last month's request count and check the result against the per-request total. If the two are within a few percent you have a workable model; if the invoice implies $0.06 a request against a model that says $0.0375, the model is missing terms and no lever ranked against it can be trusted.

The distribution across terms is what matters here. Just under half of every request is bytes that are byte-identical to the previous request and to the next one, and those bytes are being priced at the full input rate two million times a month. That is $36,000 a month spent re-uploading text the provider has already read.

Sixteen percent of the request is input that genuinely varies. Thirty-six percent is output, which is priced at five times the input rate and is generated fresh every time. Those three terms have completely different properties, and a lever that helps one of them does nothing for the other two — which is the reason the three proposals on the whiteboard were not comparable.

The 5:1 price ratio between output and input is not an arbitrary margin decision, and understanding where it comes from makes the rest of the topic easier to reason about. Processing input is a single forward pass over the whole prompt, which the hardware handles as large matrix multiplications and parallelises well across the sequence. Generating output is autoregressive: each token requires its own pass through the model, and each pass reads the entire weight set from memory to produce one token, so decoding is bound by memory bandwidth rather than arithmetic throughput. Every provider prices roughly along that split, with output somewhere between three and six times input across the major families. The practical consequence is that a token you stop generating is worth several tokens you stop sending, and any comparison that treats a token as a token will rank levers wrongly.

Why the repeated prefix gets evaluated first#

Prompt caching, whose operational mechanics were covered in 15.6, prices a cached prefix at roughly a tenth of the input rate on a read and about 1.25× on the write that populates it. Multipliers differ between providers and you should check yours rather than trusting a number from a chapter, but the shape holds: reads are heavily discounted and writes carry a modest premium.

Apply that to the 6,000-token block. A cache read costs 6,000 × $0.30 per million, or $0.0018, against $0.0180 at full price. The saving on a hit is $0.0162, which is 43% of the whole request. Hits are not free, though, and they are not universal, so the honest ceiling has two corrections in it.

The first correction is the hit rate. At a 90% hit rate — a figure you have to earn through prefix stability and TTL management, not assume — the expected saving is 0.90 × $0.0162 = $0.01458 per request, or 38.9% of the bill. The second correction is the write premium. The 10% of requests that miss pay 1.25× on the prefix instead of 1×, which adds 0.10 × 0.25 × $0.0180 = $0.00045 per request. Net saving lands at $0.01413, or 37.7% of spend, about $28,300 a month.

That number is the ceiling for this lever on this workload, and it is worth being precise about what "ceiling" means. It is what you get if caching works perfectly on the entire stable block at a 90% hit rate. Every implementation detail — a timestamp in the system prompt, a tool list that reorders, a user-specific line inserted above the examples — moves you down from it. Nothing moves you above it.

IN_PRICE = 3.00 / 1_000_000       # $ per input token
OUT_PRICE = 15.00 / 1_000_000     # $ per output token
CACHE_READ_MULT = 0.10            # provider-specific
CACHE_WRITE_MULT = 1.25           # provider-specific


def per_request_cost(
    prefix_tokens: int,
    variable_tokens: int,
    output_tokens: int,
    cache_hit_rate: float,
) -> dict[str, float]:
    hit = cache_hit_rate
    prefix = prefix_tokens * IN_PRICE * (
        hit * CACHE_READ_MULT + (1 - hit) * CACHE_WRITE_MULT
    )
    return {
        "prefix": prefix,
        "variable": variable_tokens * IN_PRICE,
        "output": output_tokens * OUT_PRICE,
        "total": prefix + variable_tokens * IN_PRICE + output_tokens * OUT_PRICE,
    }


baseline = per_request_cost(6_000, 2_000, 900, cache_hit_rate=0.0)
cached = per_request_cost(6_000, 2_000, 900, cache_hit_rate=0.90)
print(baseline["total"], cached["total"])   # 0.0375  0.023370

Running that gives $0.03750 uncached against $0.02337 cached, which is the 37.7% reduction stated above arriving from the other direction. Two independent routes to the same number is worth a minute of your time, because the arithmetic is the only part of this work that cannot be argued with, and it is the part that most cost discussions skip.

The terms a whiteboard model has to name#

The function above is not the model. It is one request under one set of assumptions, and a cost model you can defend in a room without dashboard access has to carry more terms than that. The full set is short enough to write from scratch on a whiteboard.

Input tokens and output tokens, priced separately, because the ratio between them is usually 4:1 or 5:1 and treating tokens as fungible produces answers that are wrong by a factor of two. The split of input into cacheable prefix and variable remainder, with a hit rate applied to the first part and a write premium applied to the misses. The model mix, because a cascade or a router of the kind 15.8 described means there is no single price per token — there is a weighted average across tiers, and the weights are traffic-dependent. Retries, both the ones your client library performs on a 429 or a 529 and the generations that died after emitting 400 tokens you still paid for. Batch utilisation, meaning the fraction of traffic eligible for a discounted asynchronous endpoint and the fraction of that which actually settles there rather than falling back. And any fixed charges that do not scale with tokens at all: a hosted fine-tune, meaning a copy of the model trained on your own data and kept loaded for you; provisioned throughput, meaning capacity reserved in advance and paid for at a monthly floor whether or not it is used; a dedicated endpoint.

Two of those terms deserve a note on how they are actually written down. The model mix is not a price, it is a weighted average, and the weights belong in the model explicitly rather than being folded into a single blended figure. Writing it as 0.55 × $0.60 + 0.45 × $3.00 = $1.68 per million keeps both the split and the prices visible, so when the router's behaviour changes — and it will, because escalation triggers are sensitive to the difficulty distribution of real queries — you can see which number moved. Fixed charges are the opposite problem: they are invisible per request and obvious per month, so a hosted fine-tune at $1,200 a month contributes $0.0006 to a request at this volume and nothing at all to the ranking of levers, while being the entire explanation for a $1,200 discrepancy that somebody will spend a day chasing. Both belong on the board, in their own units.

For this feature, only the first three terms are populated so far. That is enough to rank the three whiteboard proposals against each other, which was the immediate question. Caching is worth up to 37.7%. A cheaper model at 80% off unit price would apply to all three terms but changes the quality of every answer, so it needs evidence nobody in the room had. Compression addresses the 16% slice, which caps its value at 16% before the compressor's own cost is subtracted. Capping output addresses 36%, which makes it the second largest term available and, notably, the one lever the caching win cannot also claim.

What this arithmetic is not#

A per-request cost model is not a forecast. It says what a request costs given a token distribution, and token distributions move — a retrieval change that returns six chunks instead of four raises the variable term 50% with no code change in the calling path, and a prompt revision that adds two examples raises the prefix by 800 tokens for every request forever. The model is a snapshot of unit economics, and it needs re-running whenever anyone touches the prompt or the retriever.

A per-request model is also not a per-user or per-session model, and confusing the two produces business cases that do not survive contact with pricing. A support agent using this assistant runs an average of eight turns in a session, and turns are not independent: the conversation history grows, so turn eight carries the six previous exchanges as additional variable context and costs roughly $0.055 against turn one's $0.0375. The session costs about $0.34, which is the number that matters if the product is sold per seat and the number nobody computes when the dashboard reports cost per request. Whichever unit you choose, state it, because a 40% reduction in cost per request alongside a 15% increase in turns per session is a bill that went up.

The model is also not the invoice, and the difference between those two things is not a rounding error. The decomposition above assumes exactly one provider call per user request, which is almost never true in a system that has a router, a validator, a retry policy or an evaluation sample running against production traffic. Every one of those adds calls that the per-request formula has no term for, and they do not add up to a few percent.

The more subtle misreading is treating "48% of the request is repeated tokens" as "48% of the bill is recoverable". The discount on a cache read is a discount, not a refund, and the write premium, the misses, and the requests whose prefix was never stable in the first place all take bites out of it. Vendor material tends to quote the read multiplier as the headline, which is where a 90%-off claim comes from; the 37.7% above is what that 90%-off multiplier is worth once it is applied to the fraction of the request it can actually reach.

The gap between the model and the invoice is the next thing to close, because a cost model that is 60% low is not a model — it is a number that happens to be smaller than the truth. Getting the missing terms onto the board turns out to be less about arithmetic than about knowing which calls in your system nobody has been counting.