Hits over eligible lookups, split by route

The platform dashboard carries a tile labelled prompt cache hit rate, and it reads 71%. It has read between 69% and 73% for two quarters. The query behind it is one line of SQL: the sum of cache_read_input_tokens divided by the sum of all input tokens, over every request the service made in the window. On a Tuesday in March the document Q&A route stopped caching entirely, because a batch job began stamping a per-document header onto the front of the prompt. The tile went from 71% to 69% and came back to 70% the following week when unrelated traffic shifted. Nobody looked at it, because 70% is what it has always said, and six weeks later the finance channel asked why cost per request on that endpoint had risen 34%.

The tile is measuring something worth naming plainly before anyone argues about how to divide it. A prompt cache is a store the model provider keeps of the work it has already done on the front of a prompt. A model has to read the whole input before it produces anything, and in production a large share of that input is identical from one call to the next: a system prompt, a block of tool definitions, a policy document, the opening turns of a conversation. Without a cache, that repeated span — the prefix, the leading run of the prompt that is byte-for-byte what it was last time — is processed and charged again on every request. With a cache, the provider keeps the processed state and a later request beginning with the same bytes reads it back, at a small fraction of the input price and with the corresponding work skipped.

Every request therefore ends in one of three states, and counting those states correctly is the subject of this chapter. A request that finds its prefix already stored is a hit, billed at a fraction of the normal input rate. A request that has to put its prefix into the store first is a write, which costs more than sending the prompt with no cache at all. A request whose prompt has no reusable prefix was never a candidate for either. The prompt caching mechanics topic covered the machinery that produces those states — where the cacheable span is marked, how long an entry survives, what invalidates one. Prompt cache operations, which is what this topic is about, is the job that begins once the machinery is switched on: measuring whether it still works, pricing what it returns, and noticing the day it quietly stops. You meet it on any route that sends the same instructions thousands of times an hour, which on most services is where the model bill lives.

The tile is arithmetically correct. It is also the wrong quantity to put on a wall, for two reasons that compound: the denominator is a fact about traffic rather than about the cache, and a single platform-wide figure averages away exactly the failures that matter. Both are fixable, and the fix is a definition rather than a tool.

The tile divides two things that answer different questions#

The numerator, cached tokens read, measures effectiveness — of the prefixes that could have been reused, how many were. The denominator, all input tokens, measures the shape of the traffic. Requests that never had a cacheable prefix in the first place sit in that denominator permanently: a search endpoint whose whole prompt is 380 tokens, below the provider's minimum cacheable length, contributes tokens that no cache could ever have served. So does a one-shot migration script, and so does any route where nobody placed a breakpoint.

The consequence is that the figure moves whenever the traffic mix moves, with the cache behaving identically throughout. Route a marketing campaign's worth of short ad-hoc queries through the service and the tile falls, because you added denominator without adding numerator. Retire that endpoint and the tile rises. In both cases the cache did nothing differently. What the number cannot do — and this is the operational failure, not the aesthetic one — is separate a cache that stopped working from a change in the questions being asked. Those are the two hypotheses you actually need to distinguish at 2 a.m., and this metric collapses them into one movement.

Token weighting adds a second distortion on top. Because the numerator counts tokens rather than requests, a route with a 48,000-token prefix carries roughly twenty times the weight of a route with a 2,400-token prefix at equal volume. The tile is therefore dominated by whichever route happens to have the biggest prompts, which is rarely the route you most want to watch.

An eligible lookup is a request that could have hit#

The denominator you want is the count of requests where a cache lookup was possible and meaningful: a breakpoint was placed, the marked prefix cleared the provider's minimum length, and the request went to a model on which such an entry could exist. Call each of those an eligible lookup. The numerator is the count of those that read an existing entry. Everything else — short prompts, unmarked routes, models you have not enabled caching on — goes into a separate ineligible counter, where it is still visible as a coverage question but no longer contaminates an effectiveness measurement.

That definition has one property worth stating plainly, because it surprises people who expect a good cache to approach 100%. The first request against any entry is a write, and a write is a miss: there was nothing to read. A route whose prefix is used exactly four times within the entry's lifetime and then expires has three hits in four eligible lookups, which is 75%, and 75% is the ceiling for that route rather than a shortfall. Hit rate is bounded above by (n − 1) / n, where n is the number of requests an entry serves before it dies. A hit rate is therefore only interpretable next to the reuse pattern that produced it, which is the subject of the next chapter.

The provider tells you which of the three outcomes occurred through the usage fields on the response. cache_read_input_tokens above zero is a hit. cache_creation_input_tokens above zero with no read is a write, which is a miss that also paid the premium. Both at zero on a request you marked as cacheable is the interesting third case: the provider accepted the breakpoint and cached nothing, almost always because the marked prefix fell below the minimum length.

from dataclasses import dataclass

@dataclass(frozen=True)
class CacheDims:
    route: str          # "agent-chat", "doc-qa", ...
    model: str          # entries are model-scoped; a model split is a cache split
    tenant: str         # or a tenant bucket, if you have thousands
    prefix_class: str   # "shared", "tenant", "session", "document"

def record_cache_outcome(dims: CacheDims, marked_cacheable: bool, usage) -> None:
    """Emit exactly one counter per request. Ineligible requests never
    reach the hit-rate denominator."""
    if not marked_cacheable:
        metrics.increment("cache.ineligible", tags=dims)
        return

    if usage.cache_read_input_tokens > 0:
        metrics.increment("cache.eligible.hit", tags=dims)
        metrics.increment("cache.read_tokens", usage.cache_read_input_tokens, tags=dims)
    elif usage.cache_creation_input_tokens > 0:
        metrics.increment("cache.eligible.miss_write", tags=dims)
        metrics.increment("cache.write_tokens", usage.cache_creation_input_tokens, tags=dims)
    else:
        # Marked cacheable, provider stored nothing: prefix under the minimum,
        # or the breakpoint landed somewhere that produced no cacheable span.
        metrics.increment("cache.eligible.miss_nocache", tags=dims)

# hit rate = hit / (hit + miss_write + miss_nocache), per dimension combination

Splitting miss_nocache out from miss_write costs one branch and buys a distinct alert. A route that is all writes and no reads has a reuse problem. A route that is all miss_nocache has a configuration problem, and no amount of traffic will fix it.

The one thing in that function that cannot be derived from the response is marked_cacheable, and that is the point. Eligibility is a property of the request you built — did you place a breakpoint, on which span, and was that span long enough — so it has to be carried out of the request builder and handed to the metrics call. Teams that try to infer it from the response end up with the ambiguity described above baked permanently into their denominator: a route where somebody quietly removed the breakpoint six months ago looks identical to a route where the prefix is a hundred tokens short of the minimum, and both look identical to a route nobody ever enabled. Passing one boolean out of the assembler removes all three confusions at once, and the boolean is free because the assembler already knows the answer.

Segments cost something, so pick the cheap ones#

Cardinality — the number of distinct label combinations a metric carries, and so the number of separate time series your monitoring system has to store and bill you for — is the practical objection to segmenting, and it is a real one. Four dimensions multiply: eight routes times three models times four prefix classes is ninety-six series before tenants enter at all, and a platform with twelve thousand tenants cannot carry a tenant label on a counter without the metrics bill exceeding the savings the metric was built to measure.

The workable compromise is to make tenant a bucketed label rather than an identity. Name the top twenty or fifty accounts by volume, which are the ones whose behaviour can move a platform figure on their own, and collapse everything else into a single other bucket. That keeps the series count in the low thousands, and it preserves the property that actually matters: a large tenant whose traffic pattern defeats caching shows up as its own line rather than disappearing into an average. Small tenants moving individually is not a thing you would act on anyway.

Model deserves a mention because it is the dimension people forget. Cache entries are scoped to the model that created them, so a route running an A/B experiment, splitting live traffic between two models to compare them, has two independent caches, each seeing roughly half the traffic and each reaching a lower hit rate than the single-model version would. The blended number for that route falls during the experiment and recovers when it ends, and nothing is wrong. Without the model split, that looks like a regression with no cause.

One route at zero, one at ninety-five, and a blend that never moves#

Segmentation is the other half. The four dimensions worth carrying are model, route, tenant and prefix class, and each earns its place for a different reason. Entries are scoped to a model, so a partial migration silently halves reuse until both halves are counted separately. Routes have entirely different prompt shapes and should never be pooled. A tenant split catches the case where one large customer's traffic pattern defeats caching for everybody in the aggregate. Prefix class — is this a globally shared instruction block, a per-tenant block, a per-session block, or a per-document one? — is what tells you whether a low number is a defect or a fact about the workload.

Here is the service behind that 71% tile, with the same window broken out.

Token-weighted across everything, cached reads come to 5.00 billion of 7.06 billion input tokens a day, which is 70.8% and rounds to the number on the wall. Every route in that table is a different situation. The two at 95% are working as designed. doc-qa at 10% is the March regression, and it is invisible in the blend because it contributes about a fifth of the tokens and the two healthy routes contribute most of the rest. bulk-reprocess at exactly zero has never worked; it was enabled in a config change eighteen months ago and each worker builds its prefix in a different order, so no two requests ever match. adhoc-search is 140,000 requests a day that the platform tile counts as failures and the segmented view correctly declines to judge, because a 380-token prompt was never going to cache.

The stable blend is not evidence of stability. It is the arithmetic of averaging a large number that cannot move much against small numbers that can move a great deal. When doc-qa fell from 88% to 10%, the blend moved 2.3 percentage points, which is inside the week-to-week variation that traffic mix produces on its own. Per route, that same event is a fall of 78 points, and no threshold you would ever pick misses it.

The adhoc-search row deserves its own treatment rather than a zero. Those 140,000 daily requests are not a cache failure; they are a coverage question, and the right response to them is a periodic review rather than an alert. Sometimes the answer is that the route should be left alone forever. Sometimes it is that the prompt has grown since the route was written and now sits just above the minimum, at which point enabling caching is a config change worth a few hundred dollars a month. Keeping the ineligible count visible in a separate column is what makes that review possible; folding it into the hit-rate denominator makes it permanently invisible while dragging every other route's number down with it.

A hit rate is not a savings figure#

Two things this metric is not. It is not a cost measurement, and treating it as one is the single most expensive mistake in this topic. A hit rate of 95% on a route describes how often a prefix was reused; what that reuse was worth depends on how much of the request the prefix represents and on the price of everything the cache does not touch. Chapter 3 takes a workload where 90% is worth 60% of the bill and one where 90% is worth 4%, with neither team measuring anything wrongly.

It is also not the cache hit rate you know from a CDN — a content delivery network, the layer of edge servers that holds copies of your static assets near the user — or from a read-through database cache, where the denominator is unambiguous because every request is a lookup by construction. Here, whether a request is a lookup at all is a property of how you built the prompt, which means the denominator is a decision you make rather than a fact you observe. That is precisely why the definition has to be written down and enforced in one place in the code, and why marked_cacheable in the snippet above comes from the request builder rather than being inferred from the response.

What you now have is a number per route that responds to the cache and not to the traffic. What you do not have is a way to say whether any given value of it is good. Ninety-five percent on the classifier is excellent; 10% on document Q&A might be a regression, or it might be the best that workload can do. The threshold that separates those two answers is not a matter of taste — it falls out of the price of a cache write against the price of a cache read, and it is a smaller number than most people guess.