Ninety percent hit rate, four percent off the bill

Two teams inside the same company, on the same provider, at the same prices, both finished their caching work in the same quarter and both reported a 90% hit rate measured the way chapter 1 describes — hits over eligible lookups, per route, no ineligible traffic in the denominator. Team A's monthly model bill fell 60%. Team B's fell 4%. In the review that followed, most of the hour went on the assumption that one of the two numbers had to be wrong, and the finance analyst who eventually settled it did so with a decomposition that took about four minutes and no new data.

Both numbers are right. The difference is not in the cache, the measurement, or the diligence of either team. It is in what fraction of each request the repeated prefix represents.

The discount lands on the prefix and nowhere else#

A request's cost has three parts under caching, and only one of them the cache can touch. The repeated prefix is billed at 1.25x on a write and 0.1x on a read. The variable input — the user's question, the retrieved document, the conversation turn — is billed at exactly the base input rate it always was. Every output token is billed at the output rate, which is typically five times the input rate and which caching does not affect at all, in any provider's pricing, ever.

This is obvious stated plainly and is nonetheless the most common error in the topic, because the natural way to hear "90% hit rate at a 90% discount" is as a claim about the bill. The discount is 90% of the cached prefix, applied to 90% of the eligible requests. Whether that is 60% of the invoice or 4% of it depends entirely on how much of the invoice the cached prefix was in the first place.

Take base input at $3 per million tokens, output at $15 per million, writes at $3.75 and reads at $0.30 — the 1.25x and 0.1x from chapter 2 expressed in dollars. Then the two teams:

Team A's endpoint fell 74%; the endpoint was about four fifths of the team's model spend, so the monthly bill fell 60%. Team B's endpoint fell 4.1%, and since that endpoint is essentially all of Team B's spend, the monthly bill fell 4%.

Where each team's money actually was#

The decomposition that ends the argument is a spend breakdown rather than a hit rate. Written out for one uncached request:

Team A pays $0.0900 for the 30,000-token instruction block, $0.0006 for the question, and $0.0045 for the answer. The instruction block is 94.6% of the request's cost. Cache it at 90% and 94.6% of the cost drops by roughly 79% (nine reads at a tenth, one write at a quarter more), which is where the 74% comes from.

Team B pays $0.0036 for the 1,200-token prefix, $0.0360 for the 12,000-token user document, and $0.0300 for the 2,000-token response. The prefix is 5.2% of the request's cost. Even a cache that never missed could only remove 90% of that 5.2%, and Team B is already collecting most of it.

Side by side, the two spend profiles are the whole explanation:

Two workloads whose cacheable fraction differs by a factor of eighteen were never going to produce comparable savings, and no amount of engineering on either side changes that. The review spent fifty minutes looking for an error in a place where the answer was three numbers that both teams already had.

PRICES = {"input": 3.0, "output": 15.0, "write": 3.75, "read": 0.30}  # $ per Mtok

def cost_per_request(prefix: int, variable_in: int, out: int, hit_rate: float) -> float:
    """Expected cost of one request, given a steady-state hit rate."""
    prefix_cost = (
        hit_rate * prefix * PRICES["read"]
        + (1 - hit_rate) * prefix * PRICES["write"]
    )
    return (prefix_cost + variable_in * PRICES["input"] + out * PRICES["output"]) / 1e6

def savings_ceiling(prefix: int, variable_in: int, out: int) -> float:
    """The most caching could ever remove from this workload: the prefix's
    share of spend, times the 0.9 discount, at a hypothetical 100% hit rate."""
    uncached = cost_per_request(prefix, variable_in, out, hit_rate=0.0) \
        - prefix * (PRICES["write"] - PRICES["input"]) / 1e6
    return 0.9 * (prefix * PRICES["input"] / 1e6) / uncached

savings_ceiling(30_000, 200, 300)     # 0.851  -> 85% is Team A's ceiling
savings_ceiling(1_200, 12_000, 2_000) # 0.047  -> 4.7% is Team B's ceiling

The ceiling function is the useful artefact here, because it takes three integers you already know before writing any code and returns the largest number the project can possibly produce. Team B's ceiling is 4.7%. Their realized 4.1% is 87% of everything that was ever available to them. By the only measure that matters, Team B's caching work was close to a complete success, and the 4% that made it look like a failure in the review was the correct and expected result.

A team with a genuinely small prefix — a few hundred tokens of framing in front of long documents — sits at the limit of this, where the cacheable fraction of spend rounds to nothing and, on most providers, the prefix is below the minimum cacheable length so the entry is never created at all. The reported hit rate in that case is not 4%, it is zero, and the correct engineering decision is to leave caching off that route and spend the effort somewhere the money is.

Why ninety to ninety-eight can be worth nothing#

The same arithmetic settles the follow-up question, which arrives predictably whenever a team has just shipped a caching win: we're at 90%, let's push to 98%.

For Team B, run the numbers at both. At 90% the endpoint costs $0.066774 per request; at 98% it costs $0.0664428. The difference is $0.00033 per request. At Team B's 60,000 requests a day that is $19.87 a day, roughly $600 a month, against a monthly model bill of about $125,000. The work to get there involves restructuring the session affinity layer — the routing rule that sends a user's later requests back to whichever machine handled their earlier ones — so that follow-up questions land where the entry is resident, which is three to four engineer-weeks and a new class of production risk. Nobody should approve that trade.

For Team A the same eight points is worth considerably more — the prefix is 94.6% of spend, so eight points of hit rate is roughly 6.8% of the endpoint bill, which at their volume is well over $3,000 a day. The identical proposal, phrased identically, is obviously right for one team and obviously wrong for the other, and the hit rate alone does not distinguish them. Marginal value per point of hit rate is prefix share times 0.9 times volume, and it is worth putting that quantity on the report before anyone asks for headroom.

There is a version of "push the hit rate" that is worth negative money, and it is worth naming because it is the obvious move. Switching to a longer entry lifetime raises the hit rate on a bursty route by keeping entries alive across gaps, and on most pricing it also doubles the write premium. Chapter 2's table put the break-even at 56% for the longer lifetime against 22% for the shorter one. A route that goes from 70% to 88% by extending the TTL can come out behind, and the dashboard will show an eighteen-point improvement the whole time. Any change that moves the hit rate by changing what a write costs has to be evaluated on the money rather than on the rate.

Team B's actual lever is elsewhere. Output is 43% of their spend and variable input is another 52%. Shortening responses by a third takes $18,000 a month off the bill on its own, which is three times what the entire caching project delivered. Moving the retrieval step to a smaller model, trimming the retrieved context from 12,000 tokens to 7,000, or routing the overnight backfill through a batch endpoint — where requests are queued and answered within hours instead of immediately, for roughly half the price — each move more money than perfecting a cache on 5% of the bill ever could. Caching is a lever on repeated input, and when repeated input is not where the money is, it is the wrong lever regardless of how well it performs.

The institutional fix is smaller than the analysis. The review went wrong because a hit rate and a saving were being read as the same fact, so the two teams' identical 90% created an expectation of identical outcomes. Once the report carries hit rate, prefix share of spend, and realized saving as three separate columns — which is what chapter 6 builds — the question "why did A save more than B" answers itself in the second column, and neither team has to defend a number that was never theirs to control.

What the number is for, once it is not for the bill#

None of this makes hit rate useless. It remains the right operational signal — it is fast, it is per-route, and it moves the instant something breaks, which is exactly what chapter 1 built it for. What it is not is a proxy for savings, and the two need to travel as separate columns on any report that a director will read, because the moment they are collapsed into one figure somebody will multiply a percentage by an invoice.

Latency is a second reason the two must stay separate. A cache read skips prefill work on the cached span, so a route whose 30,000-token prefix is served from cache can shave a meaningful fraction off time-to-first-token even on a workload where the money barely moves. Team B's 4% is a poor cost story and may still be a good latency story, and a report that only carries dollars will get their work cancelled.

The ceiling calculation is also the cheapest planning tool in the topic, and it belongs before the work rather than after it. Three integers — repeated prefix tokens, variable input tokens, output tokens for a typical request on the route — produce an upper bound on what any amount of caching effort can return. It takes a token count and a calculator. On the routes where the answer comes back at 4%, the correct decision is usually to instrument the route so the number is visible, leave caching enabled if it is already on and above break-even, and put the quarter's engineering time somewhere the ceiling is 60%. Teams rarely regret the caching work they did; they regret the caching work they did on the wrong route while the output bill grew unattended.

Which leaves the harder question. Suppose you have done the arithmetic, found a route where the prefix genuinely is most of the spend, made a change, and watched the hit rate climb from 44% to 81% with cost per request down 30%. The ceiling calculation says that is plausible. The dashboard says it happened. Neither of those establishes that your change is what caused it, and in a live platform the number of other things that moved in the same week is never zero.