The write premium is paid before any reuse exists

The ticket said enable prompt caching on all three inference routes, and the reasoning in the description was four words long: reads cost a tenth. It shipped on a Thursday. Two of the three routes did exactly what the ticket promised. The third — a contract analysis endpoint that stuffs a 50,000-token document in front of a single question — added about $510 a day to the bill and stayed that way for five months, because the dashboard showed its hit rate climbing from 0% to 13% and 13% looked like progress.

Thirteen percent is a loss. The threshold that separates a paying route from a losing one is roughly 22%, and it comes out of two numbers that have nothing to do with your workload: what a provider charges to write a prefix into the cache, and what it charges to read one back.

The premium is charged on a bet you have not yet won#

Chapter 5 of the prompt caching mechanics topic established the shape: at a 1.25x write and a 0.1x read against base input price, an entry that serves two matching requests already costs less than sending the prefix twice. What that framing leaves implicit is the direction of the cash flow. The 25% surcharge is charged at write time, before anything is known about whether a second matching request will arrive within the entry's lifetime. Every write is a wager that reuse follows, and the provider collects the stake whether or not it does.

That asymmetry is what makes caching an operational decision rather than a free optimisation. A cache read is unambiguously cheaper than an uncached send; a cache write is unambiguously more expensive. You are trading a certain small loss on every first request for an uncertain large gain on subsequent ones, and the trade only pays above a reuse threshold you can compute exactly.

Work in prefix-equivalents, where one unit is the cost of sending the prefix once at base input rate. An entry that serves n matching requests costs n uncached. Cached, it costs 1.25 for the write plus 0.1 for each of the n − 1 reads. Setting those equal:

def break_even_requests(write_rate: float = 1.25, read_rate: float = 0.10) -> float:
    """Matching requests one entry must serve, within its lifetime, to beat
    sending the prefix uncached every time."""
    # n = write_rate + read_rate * (n - 1)  =>  n * (1 - read_rate) = write_rate - read_rate
    return (write_rate - read_rate) / (1 - read_rate)

def break_even_hit_rate(write_rate: float = 1.25, read_rate: float = 0.10) -> float:
    """The same threshold expressed as hits / eligible lookups, the metric
    from chapter 1. One write plus (n - 1) reads means hit rate = (n - 1) / n."""
    n = break_even_requests(write_rate, read_rate)
    return (n - 1) / n

break_even_requests()        # 1.2778
break_even_hit_rate()        # 0.2174  -> about 22%
break_even_hit_rate(2.0, 0.2) # 0.5556 -> about 56%, for a 1-hour TTL

Twenty-two percent, and why the number is so low#

A prefix needs to be sent 1.28 times inside one entry's lifetime to pay for itself. Since fractional requests do not exist, the practical statement is that a single reuse clears the bar with room to spare: a write plus one read costs 1.35 prefix-equivalents against 2.00 uncached, a 33% saving on the second request alone. In the hit-rate terms of chapter 1, where the denominator is eligible lookups and the first request against every entry is a miss by construction, break-even sits at about 22%.

That is a low threshold, and it is low because the discount on reads is enormous relative to the surcharge on writes. Each read saves 0.9 prefix-equivalents; each write costs an extra 0.25. One read pays for 3.6 writes. Most routes with any repetition at all clear 22% without anyone thinking about it, which is exactly why the routes that do not clear it go unexamined — caching acquires a reputation as free money, and nobody audits free money.

The threshold is not a constant, because the write premium is not a constant. Longer entry lifetimes cost more to create.

Two properties of that table are worth naming. The threshold is a pure ratio of the multipliers, so it does not depend on the price of the model, the size of the prefix, or the volume of the route — the same 22% holds for a cheap small model and an expensive large one, and for a 1,024-token prefix and a 200,000-token one. What it does depend on is the multipliers themselves, which vary between providers and between lifetime options, so the first thing to do on a new provider is read the two numbers off the pricing page and recompute rather than carrying 22% across as folklore.

The longer TTL — the time-to-live of an entry, the idle window after which the provider discards it — more than doubles the bar. That matters for the routes where you reach for it, because the reason to want a one-hour entry is usually bursty traffic with long gaps — which is also the traffic pattern least likely to produce a high hit rate. Buying a longer lifetime to rescue a marginal route can move the threshold away from you faster than it moves the route toward it.

Three routes against one threshold#

Concrete numbers make the sorting obvious. Three routes, one provider, the 5-minute lifetime, base input at $3 per million tokens.

Before working through them, one thing the threshold does not price. A cache write costs full prefill — the model reading the input tokens and building the internal state it needs before it can emit anything — on the marked span, so a missing request is no faster than an uncached one, and on some providers it is marginally slower. That means a route below break-even is losing on both axes at once — paying a surcharge and receiving no latency benefit — while a route above it collects two separate wins that a cost-only analysis reports as one. On an interactive route where time-to-first-token, the delay between sending a request and the first token of the answer appearing, is a product requirement, the latency argument can justify caching at a hit rate that the money argument alone would reject, and that is a legitimate decision as long as somebody writes down that it was made deliberately.

Route A is the case caching was designed for. One entry, continuously refreshed by traffic arriving far faster than the lifetime expires, serving hundreds of requests before it is ever rewritten. Its hit rate sits so far above 22% that the exact value hardly matters; the saving on the 20,000-token prefix approaches 90% and the only interesting question left is whether anything downstream is quietly invalidating it.

Route C fails, and the failure is worth pricing rather than describing. A 50,000-token prefix at $3 per million costs $0.150 to send uncached. Writing it costs $0.1875, a surcharge of $0.0375 per document. Say 15% of documents get a second question inside the entry's lifetime; each of those reuses saves 0.9 × $0.150 = $0.135, which averages to $0.0203 per document across the whole route. The net is a loss of about $0.017 per document, and at 30,000 documents a day that is $510 a day, roughly $15,300 a month. There is no line on any invoice that says prompt caching cost us $15,300 this month. There is a slightly larger input token bill on a route whose volume has been growing anyway.

The route in the middle has to be measured, not reasoned about#

Route B is where judgment is required, and the honest answer is that the paper calculation does not settle it. Four calls per session implies 75% and looks like a comfortable clear. Two things can turn that into a loss, and neither is visible in the average.

The first is the distribution behind the mean. "Average four calls" is compatible with 55% of sessions making exactly one call and the rest averaging around eight. The single-call sessions each pay a write with no read: 6,000 tokens × 0.25 × $3/M = $0.0045 lost per session. The multi-call sessions save heavily. On that split the route still nets a large saving, so the distribution alone rarely kills it — but a distribution where 90% of sessions are single-call and a handful are enormous can, and the mean will not tell you which world you are in.

The second is the interaction between arrival timing and entry lifetime, which the mechanics topic covered as turning an arrival pattern into an expected hit count. Four calls spread evenly across twenty minutes, against a five-minute entry, means the entry expires between calls and you pay three or four writes to collect three reads. Requests per entry falls to roughly 1.1, below the 1.28 break-even, and a route that reads as 75% at the session level is losing money at the entry level.

A third mechanism catches teams who have reasoned carefully about both of the above. An entry only becomes readable once the request that wrote it has begun returning, so a burst of concurrent requests sharing a prefix all miss together — none of them can read what the others are still writing. A route that fans out six parallel calls per session against one shared prefix therefore pays six writes and zero reads on the first turn, no matter how stable the prefix is. The model of the workload says 75%; the counters say 20%, and the gap is entirely concurrency. Where that bites, the fix is to let one request land, wait for its first bytes, and release the rest, which costs a few hundred milliseconds on the first turn and converts five writes into five reads.

So the thing to measure is not sessions and not averages. It is the two counters from chapter 1, per route: cache.write_tokens and cache.read_tokens over a representative window, ideally a full weekly cycle so that Monday-morning and Friday-afternoon session shapes are both in the sample. Realized reads per write is the number that decides it, and it is cheap to obtain — enable caching on the route for a week, read the counters, and compute the realized net directly rather than predicting it.

def realized_net_saving(write_tokens: int, read_tokens: int,
                        price_per_mtok: float = 3.0) -> float:
    """Dollars saved (negative = lost) versus never having cached this route.
    Reads give back 0.9x; writes cost an extra 0.25x."""
    return (read_tokens * 0.9 - write_tokens * 0.25) * price_per_mtok / 1_000_000

realized_net_saving(write_tokens=1_500_000_000, read_tokens=270_000_000)   # -$396.00/day
realized_net_saving(write_tokens=1_500_000_000, read_tokens=4_400_000_000) # +$10,755.00/day

What to do with the route that fails#

Turning caching off on route C recovers the $510 a day and is a legitimate answer. It is not the only one, and it is not the best one, because the failure there is a property of how the prompt is laid out rather than of the workload.

Two changes make caching pay on that route. The first is to move the reuse: if the product asks users one question per document today, batching the questions a workflow already knows it needs — the six standard extraction prompts a contract review runs — into one session while the entry is live turns n from 1.15 into something above 6, and the same 50,000 tokens go from a $0.017 loss to roughly a $0.60 saving per document. The second is to change what is cached. If there is a 4,000-token instruction and tool block that is identical across every document, ordering it ahead of the document and placing the breakpoint after it caches something with genuine cross-request reuse, while the document itself is sent uncached at base rate. That block clears break-even on its first reuse and keeps clearing it forever, and the 50,000-token document never pays a write premium it cannot recover.

Both of those are changes to the request, and both are measurable as a before-and-after on the counters. The 22% threshold tells you which routes to leave alone, which to fix, and which to turn off. What it does not tell you is what fixing them is worth to the invoice, because the prefix is only part of what a request costs — and on some workloads it is a small part.