Cache hit rate and cost forecasting

A prefix cache stores the key and value tensors computed for the front of a prompt so that a later request carrying the identical front can skip recomputing them. Providers price that reuse directly: tokens read from a resident entry are billed at roughly 0.1× the base input rate, and the request that fills the entry is billed at a premium — commonly 1.25× for a five-minute entry, 2× for a one-hour one. The hit rate is the fraction of requests that find an entry waiting for them, and it is the single number a cost model built on those discounts depends on.

The trouble is that a forecast written before the workload runs has no measured hit rate to use, so it uses 1.0, usually in an unlabelled cell.

The arithmetic of a hit rate below one#

Take a shape typical of a tool-using agent: a 20,000-token static prefix, about forty turns per task, cached reads at 0.1× the base input rate and cache writes at 1.25× for a five-minute entry. Count only the static prefix, since that is the part caching addresses; the per-turn tool results and assistant text are billed at full rate either way.

With caching and a perfect hit rate, turn one writes at 1.25× and the remaining thirty-nine read at 0.1×:

25,000 + 39 × 2,000 = 103,000 units      →  12.9% of uncached, an 87% saving

That is the number a business case tends to carry. Now introduce a hit rate h over the thirty-nine post-first turns, and note the detail that makes the whole model bend: a miss is not merely the absence of a discount. A miss re-prefills the prefix and rewrites the entry, at 1.25×.

Two baselines make the table below readable, and each is one multiplication. The uncached cost of a task is the prefix billed at full rate on every turn, which is 40 × 20,000 = 800,000 units, and every row is measured against that figure. The perfect-rate cost is 25,000 for the first turn's write plus thirty-nine reads at a tenth of the prefix each, which is the 103,000 above. Everything between those two rows is a mixture of the same two per-turn prices in different proportions.

Why the curve joining them is steep rather than gentle is entirely a fact about what a miss costs. If a miss simply forfeited the discount it would be billed at the plain rate, and losing a tenth of your hits would cost a tenth of the prefix on each lost turn. Instead a miss re-prefills and rewrites, so it is billed above the plain rate, and the swing between a hit and a miss spans the whole distance from the read rate to the write rate rather than from the read rate to 1.0. Every percentage point of hit rate lost therefore costs more than a percentage point of the projected saving, which is why 0.85 — a perfectly respectable rate — has already given back more than half of it.

def prefix_cost(prefix=20_000, turns=40, h=1.0, read=0.10, write=1.25):
    first = prefix * write                       # turn 1 is always a write
    rest  = (turns - 1) * (h * prefix * read + (1 - h) * prefix * write)
    return first + rest

uncached = 20_000 * 40                           # 800,000
for h in (1.0, 0.90, 0.85, 0.70, 0.58, 0.50, 0.0):
    c = prefix_cost(h=h)
    print(f"h={h:<5} units={c:>9,.0f}  vs uncached={c/uncached:6.1%}  saving={1-c/uncached:6.1%}")

The curve is steep near the top and the top is where forecasts live. Dropping from a perfect rate to 0.85 — which is a good rate — costs more than half of the projected saving, because each miss carries the write premium as well as the lost discount. The last row is the one to keep in mind when someone proposes enabling caching on a workload nobody has measured: at a hit rate of zero you are paying 25% more than if you had left it switched off.

That row is one multiplication as well. At a hit rate of zero each of the forty turns writes the prefix at 1.25×, so the task bills 40 × 20,000 × 1.25, which is the 1,000,000 in the table, against an uncached 800,000. Dividing the first by the second gives 125.0%, and the extra quarter is the write premium applied forty times over. A deployment in that state is not neutral and not merely useless. It is a surcharge on every request, levied to create entries nothing ever reads.

Six mechanisms that keep the rate under 100%#

Each of the six has appeared earlier in this topic in a different costume, and each subtracts from the rate independently of the others.

The first turn of every task is a miss by definition. There is nothing resident to hit, and the entry has to be written. At forty turns per task that is a floor of 2.5% misses; at five turns per task it is 20%, which caps the achievable saving well below what the same prefix would deliver on longer tasks. The most useful question to ask about an agent's cost model is not how big the prefix is but how many turns amortise it.

Entries expire on a TTL. A five-minute default is generous for a tight tool loop and short for anything with a human or a slow dependency in it. An analyst who reads the agent's turn-12 output for seven minutes before replying returns to a cold entry; a tool call that waits ninety seconds on a third-party API is fine, one that waits six minutes is not; an overnight scheduled task starts every run cold. The one-hour TTL exists for exactly these gaps and costs 2× on the write instead of 1.25×, which — by the break-even arithmetic of chapter 3, where a cache write only pays for itself once enough reads follow it — needs three reads rather than two.

Routing sends requests to nodes that do not hold the blocks. An entry lives in the memory of one replica. An autoscaler adding four replicas to a pool of eight moves roughly a third of subsequent traffic onto cold nodes until they fill; a load-balancer change from prefix-hash affinity to least-connections, which picks whichever replica is least busy and knows nothing about prefixes, takes the rate down to roughly 1/N and looks, from every dashboard except this one, like a routine configuration improvement.

Template edits reset everything, silently. One character in the system block changes the first differing token and everything downstream of it, which is the exact-prefix keying of chapter 4. Deploy-cadence changes are fine and cost one cold fill per node. A config-driven prompt that some team can edit through an admin UI at 14:00 on a Wednesday is not deploy-cadence, and the resulting hit-rate cliff arrives with no error, no alert, and no correlation to any deployment record.

Concurrent identical requests all miss. An entry becomes readable only once the first response has begun streaming, so an agent that fans out eight parallel sub-tasks on turn one pays eight writes rather than one write and seven reads. The fix is structural — issue one request, wait for its first token, then fan out the rest — and it is invisible until you look at cache_creation_input_tokens and find it eight times larger than the prefix.

Two quiet floors produce a flat zero rather than a reduced rate. A prefix shorter than the model's minimum cacheable length — 512 tokens on some models, 1,024 or more on others — silently never caches, with no error and cache_creation_input_tokens stuck at zero. And a breakpoint searches backward only a bounded number of content blocks for a prior entry, commonly twenty, so an agentic turn that appends thirty tool-result blocks can step past the previous turn's entry entirely. Both present as "caching seems to do nothing", and neither is visible in the prompt.

The number a cost model should carry#

The quantity that belongs in a cost model is not the hit rate as a count of requests. It is the effective input multiplier: what one token of prompt actually costs, averaged over real traffic, expressed against the uncached baseline of 1.0.

def effective_multiplier(usages, read=0.10, write=1.25):
    """usages: iterable of response.usage over a real traffic window."""
    billed = sum(read * u.cache_read_input_tokens
                 + write * u.cache_creation_input_tokens
                 + 1.0  * u.input_tokens for u in usages)
    raw = sum(u.cache_read_input_tokens
              + u.cache_creation_input_tokens
              + u.input_tokens for u in usages)
    return billed / raw

# 0.19 → excellent.  0.62 → the workload below.  1.08 → paying to make it worse.

What that function computes is a weighted average price per token of prompt, and the weighting is what makes it the right quantity to forecast on. A hit rate counts requests, so it treats a request that read the whole 20,000-token prefix and one that read a couple of hundred tokens as the same event, which a cost model cannot afford to do. The numerator here prices every token at whatever it was actually billed at — a tenth for reads, a premium for writes, full rate for everything outside the cached region — and the denominator counts those same tokens once each at face value. Dividing one by the other gives what a token of prompt cost on average, against a baseline of 1.0 for a deployment with no caching at all. A multiplier above 1.0 is the arithmetic saying you are paying more than you would with the feature switched off, which is a sentence a hit rate can never produce.

Graph that one series, per endpoint, alongside the distinct-fingerprint count from chapter 4 and the KV usage and preemption series from chapter 7, where a pool sized by KV bytes rather than by context window starts evicting under load. Those four together diagnose almost every regression here. Fingerprint cardinality jumping means a prompt assembler stopped being byte-stable, KV usage pinned near 1.0 with rising preemptions means the store is being evicted out from under you, and the multiplier climbing with neither of those means TTLs or routing.

Sample it at ten-minute resolution over a fortnight rather than averaging the whole window, because the distribution is what you are after and an average of a bimodal series describes neither mode. Model deploy days and autoscaling events separately if either is frequent; both are heavy-tailed, and a median computed across them understates the good periods and the bad ones at the same time.

From token-units to the invoice#

Take a model at $5 per million input tokens and $25 per million output, a task of forty turns, a 20,000-token static prefix, roughly 1,200 tokens of per-turn conversation growth billed at full rate, and 600 output tokens per turn.

Turning those units into money is one multiplication per line, and doing it in the open is the point, because the interesting figure is not in the row people quote. The uncached prefix line is the 800,000 units from earlier at $5 per million, which is $4.00. The two hit-rate columns replace 800,000 with the token-unit figures from the previous table — 452,800 at h = 0.58, 192,700 at h = 0.90 — multiplied by that same rate, giving $2.26 and $0.96. The per-turn context line is 1,200 tokens on each of forty turns, so 48,000 tokens at $5 per million is $0.24, and it is identical across all three columns because none of it is cacheable. The output line is 600 tokens on each of forty turns, so 24,000 tokens at $25 per million is $0.60, identical everywhere for the stronger reason that caching has no opinion about output at all.

The prefix line moves by 76% between the two hit rates and the task total moves by 42%, because output and per-turn context are fixed costs that caching has no opinion about. Both percentages are now visible in the columns: the prefix line falls from $2.26 to $0.96, and that drop over its starting figure is the 76%, while the total falls from $3.10 to $1.80 and the same division on those two gives 42%. What separates the two percentages is the context and output lines sitting underneath, unchanged in every column. At 40,000 tasks a month that is the difference between $124,000 and $72,000 — a real number, and less than half the number you get by applying the prefix-line percentage to the whole bill.

Two habits keep this honest. Quote the saving as a change in total monthly spend rather than as a percentage of anything, since a percentage invites the reader to apply it to whatever total they have in mind. And build the table from the observed effective multiplier rather than from an assumed hit rate: for a fortnight of real data, a p50 multiplier of 0.55 gives the expected case and a p90 of 0.78 gives the figure to commit to, a spread of roughly $12,000 a month at this volume. A forecast that quotes one number has hidden that spread rather than eliminated it.

When a forecast should not be given#

There are conditions under which the honest answer to "how much will caching save us" is that you cannot responsibly say yet, and naming them is more useful than producing a number. Refuse if there is no hit-rate instrumentation in production, because every figure is then an assumption about a quantity that has never been observed. Refuse if the prompt assembler has not been shown byte-stable across the fleet — the twelve-worker divergence of chapter 4, where unordered JSON keys gave each worker a different prefix, does not appear in staging and turns an 87% saving into 7%. Refuse if anything volatile sits ahead of the stable block, since the position-zero arithmetic of chapter 5 makes the saving zero and the cost negative.

Refuse, too, if the prompt can be edited outside the deploy pipeline, because the rate then depends on a human's editing cadence. Refuse if tasks are short, since the first-turn floor dominates. And refuse if routing has no prefix affinity, because the achievable ceiling is then a function of replica count rather than of anything about your prompt.

Above all, refuse to let the input-side percentage stand in for the bill. Caching does nothing to output tokens, so an agent that reads 20,000 and writes 3,000 per turn at a 5× output rate sees an 87% cut applied to a minority of its spend. Present the saving as a change in total monthly cost, computed from both sides, or the number will be technically correct and commercially misleading.

A prefix cache is one of the few optimisations that can be worse than not doing it, and the hit rate is what decides which side of that line a workload lands on. The mechanisms that hold the rate down are ordinary operational facts — first turns, expiry, routing, editing cadence, fan-out, and two silent floors — and none of them shows up as an error, which is why a forecast built on an assumed rate survives review and dies on the invoice. The effective input multiplier is what replaces the assumption, and it is quoted as a range.

Everything in this topic reduces to a short sequence, and it is worth running in that order rather than the order the tooling suggests. Measure the uncached baseline first — time to first token (TTFT), the effective multiplier, and total spend — so there is something to compare against. Freeze the prefix: sort your JSON, sort your tool list, eliminate set iteration, and move every per-request value behind the last breakpoint. Verify the boundary with a token-ID diff and the fingerprint cardinality graph before touching cache_control at all. Place the breakpoint at the end of the stable region, then read the usage fields on the second call of a pair to confirm cache_read_input_tokens is what you expect rather than what you hoped. Compose a cache key that carries model, weights, tokenizer, template digest and tenant, and log create, hit, expire and invalidate against it. Size the KV budget from the memory formula rather than from the context window, leaving headroom for the entries you intend to retain. Then graph the four series, alert on the multiplier and on fingerprint cardinality, and only then write the saving into a forecast — as a range, from measured data, on the pessimistic end of what you saw.