Silent cache invalidation

Prompt caching matches on the exact serialized bytes of a request's leading tokens, which means the mechanism has one systemic failure: the bytes move. A prefix that differs on every request still qualifies and still returns a perfectly good completion — it simply creates an entry nothing will ever read, at the write premium, on every call. Nothing in the API reports this. The request is valid, the response is normal, and the only evidence is two counters in the usage block that most services never look at.

The silence is worth deriving rather than accepting, because the instinct is that a failure costing real money every second must trip something somewhere. Caching is an optimisation on prefill and not a property of the answer. A miss means the provider computes the prefix state instead of fetching a copy of it, which is exactly what it did before the feature existed, and the model then conditions on the same tokens it would have conditioned on either way. So the completion is the completion you would have got. There is nothing for the API to report because, by its own contract, nothing failed: a cacheable marker is a request for reuse, not a promise of it.

Follow one such request outward and every layer that could have caught it reports success. The transport sees an HTTP success status. The client library sees a well-formed response and raises nothing. The application sees a completion of the usual shape and length. A test suite asserting on content passes, because content is the one thing that did not change. A latency alarm stays quiet as well, and for a reason that is easy to miss: a hit removes prefill time, so a miss merely restores the timing the endpoint had before caching was switched on, and there is no regression to detect against a baseline measured then. What is left is the invoice, which arrives monthly, aggregated by model, with input tokens as one line.

Accounting for a 3% hit rate and a 15% bill#

Take an assistant whose system prompt is 18,000 tokens, with each request also carrying about 7,900 tokens of conversation and retrieved context, running at around 5,200 requests a day, and whose prompt template renders the current clock time on its first line so the model knows the date. The hit rate such an endpoint reports and the increase in what it bills are both derivable, and deriving them is what tells you whether you have found the whole problem or one instance of it.

The hit rate comes out near 3% rather than 0% because a rendered clock changes only once a second. At 5,200 requests a day the arrival rate is 0.06 requests per second, so occasionally two requests land inside the same clock second and the second one finds the first one's entry. For Poisson arrivals at rate λ per second — requests landing independently of one another at a steady average rate, the standard model for traffic nobody is coordinating — the fraction of requests that are not the first in their second is 1 − (1 − e^(−λ))/λ, which at λ = 0.06 comes to 0.030. A hit rate that tracks traffic volume rather than prompt structure is the signature of a time-varying prefix, and it is why this class of defect survives staging: at a tenth of production traffic the same template reports 0.3%, which looks like caching simply being switched off.

Each part of that has a step under it. The arrival rate first: a day holds a little over eighty-six thousand seconds, and 5,200 requests spread across them is 0.06 a second. Then the partition: the only field varying inside the prefix is a clock rendered to the second, so two requests produce identical bytes when and only when they landed in the same clock second. That turns the day's traffic into groups, one group per second, and inside a group of any size the first request writes the entry and the rest read it. The hit rate is therefore the fraction of requests that are not first in their group.

Counting that fraction needs two averages rather than a simulation. Over a long run of seconds, the number of entries written equals the number of seconds that held at least one request, which for Poisson arrivals happens with probability 1 − e^(−λ) in any given second. The number of requests over the same stretch is λ per second. Both totals are sums of independent per-second contributions, so their long-run ratio is the ratio of their averages, and the first-in-group fraction is (1 − e^(−λ))/λ. Everything else is a hit, which is the expression above.

The value of that expression at these rates can be got without a calculator. For λ well under one, 1 − e^(−λ) is λ minus about half of λ squared, so dividing through by λ leaves 1 minus about half of λ. The hit fraction is what remains, which is half the arrival rate: half of 0.06 is 0.03. The picture behind the algebra is that at this traffic almost the only seconds producing a hit are the ones holding exactly two requests, and such a second yields one hit across two requests.

Two consequences follow from "half the arrival rate". The first is the staging figure quoted above: the hit rate is proportional to λ, so dividing traffic by ten divides the hit rate by ten, and no change to the template is needed for the number to move. The second is a diagnostic. Had the varying field been a request id rather than a clock, no two requests could ever agree and the hit rate would sit at 0% at any volume at all. A small hit rate that grows with traffic says the volatile field is derived from time; a hit rate pinned at zero says it is unique per request.

The cost follows from what the other 97% of requests do. Each writes a new 18,000-token entry at 1.25× the base input rate, so the prefix alone bills 0.97 × 1.25 + 0.03 × 0.1 = 1.216 times what it billed uncached — a 21.6% premium on the prefix. The 7,900-token suffix is unaffected, being outside the cacheable region either way, so input per request goes from 25,900 tokens to 18,000 × 1.216 + 7,900 = 29,788, which is 15% higher. Output tokens are untouched, so the all-in figure is a little lower than 15% and the input-side figure a little higher. The shape to carry away is that a broken prefix cache is worse than no prefix cache by roughly the write premium, scaled by the fraction of the request the prefix represents. There is no configuration in which a 3% hit rate is a partial win.

The weighted average in the middle of that is allowed because each request's prefix bills at exactly one of two multipliers and never at anything between them. A day's prefix bill is a sum over requests, so the average multiplier is the share-weighted mean of the two: ninety-seven hundredths of the requests at 1.25 contribute a little over one and a fifth, three hundredths at 0.1 contribute three thousandths, and the two together give the 1.216. The 25,900 is then just 18,000 + 7,900, and inflating only the 18,000 by that multiplier before adding the untouched suffix back is what produces the 29,788. Output stays out of it because caching acts on prefill, and generation runs off the same state whether that state was fetched or recomputed.

The claim that no 3% hit rate is a partial win is the one people resist, so it is worth saying why. The tempting reading of a 3% hit rate is that the system collects 3% of the benefit and is therefore a little ahead of where it started. It collects 3% of the benefit while paying the premium on the other 97%. A read is worth more per token than a write costs — a tenth of the base rate against a quarter above it — and a ninety-seven to three split settles the question regardless. The prefix multiplier only falls back to 1 once the read fraction reaches the break-even established in chapter 6, and 0.03 is nowhere near it.

Three classes, and they fail differently#

A timestamp is one instance of a class, and the class has two siblings that produce identical counters from entirely different causes.

That a single response cannot separate them is a consequence of what the key is made of. The lookup hashes the serialized prefix together with the model identity and the organization, and compares one hash against another. A difference anywhere in that tuple produces the same outcome — no entry found — and by the time the usage block is filled in, the only fact available is that a hash did not match a hash. The response reports what happened, meaning tokens stored and none served, and it has no field in which to report why. What separates the three causes is when the failure started and what shipped just before it, which is what the middle columns of the table below record.

The second row produces the most confused incident channels, because the deploy that caused it did not touch a prompt. An agent whose hit rate falls from 92% to 4% overnight, with no prompt edit in the diff, has usually had its tool definitions reordered underneath it — a schema library upgraded from one major version to the next, iterating a registry in a different order or emitting required before properties instead of after. The tool block sits at the very front of the prefix, so the same set of tools serialized in a different order is a different byte sequence from token zero, and every entry becomes unreachable at once. It stays at 4% rather than recovering because the new serialization is stable; the system is caching perfectly well against a prefix that has simply been renumbered.

The third row is the hardest to find by reading the template, because the template is correct. Adding a tenth tool to a nine-tool agent changes the prefix for every request in the system, which is expected and fine as a one-off. A feature flag that adds that tool for half of traffic splits the cache into two populations and halves the reuse of both, which is not fine and does not look like a caching change to anyone reviewing it. Model aliases behave the same way: pointing at a floating alias means the provider can resolve it to a new snapshot on its own schedule, and the key includes the model, so a hit rate resets on a day nobody deployed.

A check that survives the next engineer#

Deleting one timestamp fixes one week. It does not fix the following month, when someone adds Request ID: {{request_id}} to the top of the same template for tracing, or the month after that, when a locale string moves up so the model answers in the right language. The date requirement is usually real as well — an assistant that thinks it is 2024 gives wrong answers about anything time-relative — so the repair is to move the value into the suffix, where the ordering rules from chapter 2 already put per-request content, and then to make the invariant enforceable. The invariant is small enough to state in one line: the cacheable prefix must be byte-identical for two different requests at two different times.

Both halves of that invariant are load-bearing and each rules out a different class. "Two different requests" rules out anything derived from the request — a user, a request id, an experiment bucket — and "two different times" rules out anything derived from the clock. A test that renders the prefix once and compares it against a stored string checks neither, because it never produces a second prefix to compare against. What the check has to do is build the prefix twice, from contexts that differ in every field a request is allowed to differ in, and assert that the bytes came out the same.

import hashlib

def prefix_bytes(ctx) -> bytes:
    """Everything up to and including the last cache breakpoint, as it is sent."""
    return serialize(build_prompt(ctx)[:BREAKPOINT_INDEX + 1])

def test_prefix_is_stable_across_requests_and_time():
    a = prefix_bytes(Ctx(user="alice", request_id="r-1", now="2026-03-11T09:14:22Z"))
    b = prefix_bytes(Ctx(user="bogdan", request_id="r-2", now="2026-03-11T17:02:05Z"))
    assert a == b, "cacheable prefix varies per request"

def test_prefix_fingerprint_is_pinned():
    digest = hashlib.sha256(prefix_bytes(Ctx.fixture())).hexdigest()
    assert digest == PINNED_PREFIX_SHA, (
        f"prefix bytes changed: {digest}\n"
        "If this change is intentional, update PINNED_PREFIX_SHA in this PR "
        "and expect one cold fill per cache window after deploy."
    )

The first test catches the whole rendered-metadata class, including the instances nobody has invented yet, because it varies every field a request can vary and asserts the bytes do not move. It runs in milliseconds, needs no network, and fails in the pull request that introduces the problem rather than in a finance channel two days later.

The second test catches serialization drift, which the first cannot, since a dependency that reorders tool definitions reorders them identically for both contexts. Pinning the digest — the fixed-length hash standing in for those exact bytes, so any difference at all produces a different value — turns an invisible change into a failing build with a message that tells the next person what to do about it. The failure is expected on any change that legitimately edits the prompt or adds a tool, and that is the design: the pin is not there to freeze the prefix, it is there to make every change to it a deliberate line in a diff with a known cost of one cold fill per cache window.

Neither test can see a model alias resolving to a new snapshot or a provider-side change, so the third class needs a production signal rather than a test. Emit the two counters as metrics, compute the read fraction per endpoint, and alert when it drops below the break-even from chapter 6.

read = usage.cache_read_input_tokens
write = usage.cache_creation_input_tokens
metrics.gauge("llm.cache.read_fraction", read / max(read + write, 1),
              tags=[f"endpoint:{endpoint}", f"model:{resolved_model}"])

Tagging by the resolved model rather than the alias is the detail that makes an alias change legible: the hit rate does not mysteriously fall, it falls on a new tag while the old one goes quiet. Alert on a rolling one-hour read fraction below 0.22, which is where caching starts costing more than not caching, and the alert has an unambiguous meaning — this endpoint is currently paying a premium.

The belief worth dismantling here is that a dashboard would have caught this already, since providers report both counters and a hit rate is one division away. What is missing is not the number. It is that a hit rate is nobody's availability signal, so no rotation pages on it and no release gate blocks on it. A fleet-wide average makes it worse rather than better, because the average is dominated by whichever endpoint sends the most traffic, and one endpoint falling to 3% moves a fleet number by less than its ordinary week-to-week variation. On top of that, 3% reads as a plausible fact about a workload rather than as a defect — plenty of endpoints genuinely have little to reuse. A gauge per endpoint with a threshold attached is a different object from a chart, because it turns a number somebody might look at into a claim somebody has to act on.

Reproducing all three on purpose#

None of this has to be learned from an outage, and the three failures take about ten minutes to reproduce against a scratch key. Start from a working baseline: a qualifying prefix, sent twice, with the second response reporting a read of the full segment. Then break it three ways in turn and watch the counters move.

Interpolating datetime.now().isoformat() into the first line reproduces the metadata class, and the second call comes back with a write equal to the whole prefix and a read of zero. Shuffling the tool list before serialization — random.shuffle(tools) on a copy — reproduces serialization drift with the same counters and a completely different cause, which is the point of doing both: the evidence in the usage block is identical, so the diagnosis has to come from what changed rather than from what the numbers say. Switching the model string to a different member of the same family, or adding a tenth tool to a nine-tool set, reproduces the third class while the prompt text stays byte-identical, which is the case that convinces people the key is wider than the template.

Run those four calls once and the conversation in an incident channel afterwards is much shorter, because everyone has seen that a zero read counter is a symptom with three quite different causes.

A stable fingerprint is not a hit rate#

The prefix can be perfectly identical on every request and still cache nothing, because it is below the minimum length, because the boundary is in the wrong place, or because matching requests arrive further apart than the entry survives. The two tests answer only the question "are the bytes the same?", and a green build with a 0% hit rate points at chapters 4, 6 and 7 rather than at this one.

Invalidation is also not always a defect. Prompts are supposed to change — instructions get sharpened, tools get added, playbooks get edited — and each of those costs one refill per cache window and then recovers within minutes. The distinction that matters is cadence: a prefix that changes when someone deploys is healthy, a prefix that changes when someone sends a request is broken, and the counters look similar for about an hour after a deploy.

Cadence is the right test because of how the two cases scale. A change that lands once costs a refill, after which the entry it created is read by everything behind it, and the count of premium writes it causes does not grow with traffic at all. A change that lands per request makes the count of premium writes equal to the count of requests, so it scales with volume and never settles. The two are indistinguishable in the minutes after a deploy and differ by orders of magnitude over a week, which is why the read fraction is alerted on over a rolling window rather than read at a point in time.

Silent invalidation is therefore the failure mode this topic ends on because it is the one that cannot be caught by reading the prompt. Every other condition in these eight chapters is visible in a single request; this one lives in the difference between two, which no single request displays. Two tests and one gauge make that difference visible at the point it is introduced, and they cost less to write than a single afternoon spent explaining an invoice.

Composed together, the eight chapters give a short order of operations for any endpoint about to be cached. Establish the regime, since it decides whether markers are needed at all. Sort the blocks by how many requests share them, and confirm the divergence point from logs rather than from the template. Check the segment against the minimum for the model actually configured. Estimate the reuse count from the arrival gaps and the entry lifetime, and compare it against the 22% break-even before spending the effort. Place boundaries at the seams between readerships, not at the far end of the prompt. Then pin the prefix, alert on the read fraction, and read the counters for a week before believing any of the estimates you started with.