Each lever discounts a different base and none compound

The director has read a vendor post over the weekend and arrives with three numbers: prompt caching cuts costs by up to 90%, the batch endpoint is 50% off, and the small model in the same family is 80% cheaper. Multiplied together those leave 1% of the bill, and the feature's spend is down 22%. The question in the room is why the gap is so large, and the implied answer is that the implementation must be poor.

The multiplication is the error. Each of the three percentages is a discount on a different base, the bases are different sizes, and two of the three cannot be applied to the same traffic at the same time. Working out what each one is a percentage of takes about ten minutes with the decomposition from chapter 1 in front of you, and it produces a ceiling that is defensible rather than aspirational.

Four claims, four bases#

Take the same request: 6,000 tokens of stable prefix, 2,000 tokens of variable context, 900 tokens of output, $3.00 and $15.00 per million, $0.0375 a request, $75,000 a month across two million requests.

The first column is a property of the provider's price list. The third and fourth are properties of your workload, and they are where the headline numbers go to die. Caching's 90% is a discount on 48% of the request, so its arithmetic ceiling is 43% before hit rates and write premiums pull it to 37.7%. Batch's 50% applies to everything, which sounds like the strongest claim on the list until you ask how much of the traffic can wait: this product has one nightly reporting workflow and some backfill jobs, together about 20% of requests, so a 50% discount on 20% of the bill is 10%.

The model claim is the largest and the only one whose reach is a quality question rather than an arithmetic one. Eighty percent off the unit price applies to all three token terms, so the constraint is entirely how much traffic can be routed down without the answers getting worse — which is the subject of 15.8 and is measured, not assumed. At 55% of traffic routed to a tier priced at a fifth of list, the bill retains 0.45 + 0.55 × 0.20 = 0.56 of its original size, a 44% reduction.

What actually multiplies, and what refuses to#

The residual fractions multiply where the levers are independent. Caching leaves 62.3% of the bill; tiering leaves 56%; applying both, caching still removes 37.7% of whatever spend remains inside each tier, so 0.623 × 0.56 = 0.349, a 65.1% reduction, or $26,200 a month from $75,000. That is a real and large number and it is not 99%.

The dollar value of each lever shrinks as the others land, which is the part that surprises people who expect savings to add. Caching alone is worth $28,300 a month. Applied after tiering, the same caching configuration on the same traffic is worth $15,850, because the prefix it discounts is now priced at a fifth of list on 55% of requests. Nothing has gone wrong with the cache; the base it discounts got smaller. Whichever lever ships second always looks less impressive than its business case said, and teams routinely read that as a failed implementation.

There is a second-order version of the same effect that catches teams comparing quarters. Suppose caching ships in Q1 and is credited with $28,300 of annualised saving in the plan. Tiering ships in Q2 and is credited with $33,000, computed against the pre-caching baseline because that is the number in the original business case. The two credits sum to $61,300 against a bill that only fell $48,800, and finance will notice. Crediting a lever against the state of the world when it shipped, and stating that convention in the writeup, is the arrangement that keeps the arithmetic honest at the cost of making whichever lever went second look worse than its author expected.

Batching does not multiply into the stack at all, because it is not a discount you apply on top of interactive traffic — it is a different endpoint with a different contract. Submitting a request to a batch API means no streaming, no in-request retry, and a completion window measured in hours, so the traffic it applies to is disjoint from the traffic caching was optimising. The two savings sit side by side on separate slices of the bill. Batching also interacts badly with caching in a way that is easy to miss: a batch of ten thousand items sharing a prefix may or may not get cache reads across the batch depending on how the provider schedules them, and if the batch runs once a night the prefix will have aged out of a five-minute TTL by the time the next one submits, so the cached-prefix saving you were counting on the interactive path simply does not exist there.

Compression pays for itself out of its own saving#

A team proposes putting a small model in front of the main call to summarise the retrieved passages, cutting input tokens by 60%, and presents it as a 60% reduction in input cost. Two things are wrong with the framing before anyone measures quality.

The 60% applies to the retrieved context, which is 2,000 of the 8,000 input tokens, so input tokens fall by 1,200 out of 8,000 — 15%, not 60%. And the compression step is a model call with its own bill. Reading 2,000 tokens and writing 800 on a model at $0.15 and $0.60 per million costs $0.00030 + $0.00048 = $0.00078 per request. The saving on the main call is 1,200 × $3.00 per million = $0.0036. Net is $0.00282, which is 7.5% of the request and $5,640 a month.

CHEAP_IN, CHEAP_OUT = 0.15 / 1e6, 0.60 / 1e6
MAIN_IN = 3.00 / 1e6

def compression_ledger(context_tokens: int, compressed_tokens: int) -> dict[str, float]:
    compressor = context_tokens * CHEAP_IN + compressed_tokens * CHEAP_OUT
    saved = (context_tokens - compressed_tokens) * MAIN_IN
    return {
        "compressor_cost": compressor,
        "gross_saving": saved,
        "net_saving": saved - compressor,
        "efficiency": (saved - compressor) / saved,
    }

compression_ledger(2_000, 800)
# {'compressor_cost': 0.00078, 'gross_saving': 0.0036,
#  'net_saving': 0.00282, 'efficiency': 0.783}

Twenty-two percent of the gross saving is consumed by the compressor, and that ratio is worth computing before the design review rather than after, because it moves sharply with the price gap between the two models. Compress with a model at $1.50 per million instead of $0.15 and the efficiency drops from 78% to negative — the compressor costs $0.0078 to save $0.0036.

The instinct that follows is to point the compressor at the largest block instead, which is the 6,000-token prefix, and this is where the lever becomes actively harmful. That block is already being read from cache at $0.0018 a request. Compressing it to 2,400 tokens and caching the result gets it to $0.00072, saving $0.00108 — a tenth of what the uncached arithmetic suggested. And query-aware compression, which is the variant that preserves quality best because it decides what to keep based on the user's question, produces a different prefix for every request, so the cache never hits. That prefix now costs 2,400 × $3.75 per million on a write every time, or $0.0090, which is five times what the cached original cost. A lever presented as a 60% saving lands as a 400% increase on the term it touches.

Prompt compression as a technique comes out of Microsoft Research's LLMLingua line of work, which uses a small language model to score tokens and drop the ones that contribute least, with a budget controller allocating the compression ratio across parts of the prompt; the papers report up to 20× compression with small performance loss on reasoning benchmarks. The LLMLingua repository is the right place to look if you want to run it, and the technique is real. What the benchmark numbers do not carry over is your failure mode, because compressors drop the tokens a language model finds most predictable, and the most predictable tokens in a retrieved passage include the words that scope a fact — "except", "prior to", "not", "per seat". On a 400-question internal evaluation this pipeline moved exact-match accuracy — the share of questions whose answer matched a stored key exactly, with no grader involved — from 91.2% to 86.5%, and four of the five largest regressions were questions whose correct answer depended on a qualifier the compressor had removed while keeping the sentence that looked like the answer.

The claim to refuse without more information#

Of the three the director brought in, the smaller model is the one to decline to evaluate until someone produces a specific artefact: an evaluation set drawn from production traffic, sliced by task type, with per-slice scores for both tiers. Its 80% is the largest number available and it is also the only one whose downside is invisible in the cost data, since a cheaper model that answers worse produces a smaller invoice and a happier dashboard. Aggregate accuracy is not enough either — the tier difference is usually near zero on lookup-shaped tasks and large on multi-document reasoning, and a blended score that mixes them at production frequencies will hide a 40-point drop in a 4% slice.

It also helps to know why the batch discount exists, because that explains its terms. A provider's fleet has diurnal demand — a load that rises and falls on a daily cycle — and a fixed amount of hardware sized for the peak, which leaves scheduling slack overnight and in troughs. A batch endpoint sells that slack: you surrender the right to a prompt response and the provider fills gaps with your work. Everything awkward about the interface follows from that bargain — the multi-hour completion window, the absence of streaming, the file-based submission that lets the scheduler reorder freely, and the fact that no amount of paying more will make a batch job finish sooner. Providers publish this as a straightforward 50% line item; OpenAI's Cost optimization guide is a compact statement of the same checklist from the seller's side, and it is worth reading precisely to see which of the numbers it quotes are price-list facts and which are workload assumptions presented as facts.

Batching therefore needs one number before it can be ranked: the fraction of requests whose consumer genuinely tolerates a multi-hour turnaround, measured rather than asserted. "Nightly report" usually does; "nightly report that a human reads at 8am and re-runs when it looks wrong" often does not, and every fallback to the synchronous endpoint pays full price and erodes the 50%. The measurement is usually available without building anything: look at how often the existing workload is triggered manually versus on schedule, and at the distribution of time between a result being produced and a human opening it.

Caching needs the least new information, which is most of why it goes first. What none of the four claims addresses is the 36% of the request that is generated output, and output is where the largest untouched term sits once the prefix has been dealt with.