Prompt caching and prefix stability

Prompt caching stores the processed form of a leading span of a prompt so that repeated requests do not have to compute it again. The hit rate is the share of requests whose leading tokens were found already processed; the thing being reused is the work of prefill, the forward pass over the prompt that produces the attention state the model generates from. A hit shows up in two places at once. It shows up on the bill, because cached input is charged at roughly a tenth of the base input rate, while writing a cache entry costs about 1.25× base for the default five-minute entry and about 2× for a one-hour entry. And it shows up as latency, because prefill over several thousand tokens is a real fraction of the time before the first token appears.

That second effect follows from the order the work happens in rather than from a separate mechanism. Nothing can be generated until prefill has produced a state covering the whole prompt, because the first output token is conditioned on every input token. The interval before the first token therefore contains the entire prefill, and a hit removes the prefill work for the span it covers. The same reasoning says what a hit cannot do: the tokens after the first are produced one at a time from a state that already exists, so a cache has nothing to contribute to them, and a request that spends most of its time generating a long answer will barely notice one.

The reason this belongs in a topic about instruction placement is that the two subjects are usually thought to be in tension and are mostly not. A rule moved late is a compliance decision; a block moved early is a cost decision; and the mechanism that decides the second is precise enough that you can tell, from the diff alone, whether a placement change touches the cache at all. Getting it backwards costs teams twice — once by blocking a change that was free, and once by permitting a layout that is not.

Matching starts at token zero and stops at the first difference#

The cache is a prefix cache. The provider computes a key from the exact bytes of the rendered prompt up to each cache breakpoint — a marker the caller sets on a content block, saying "store the state for everything up to here" — and a request can reuse the stored attention state for a span only if every byte before that span matches a previous request's. Rendering order is fixed: tool definitions first, then the system block, then the messages, in order. There is no matching on content, no fuzzy alignment, and no ability to reuse a middle section whose prefix has changed.

The reason it can only be a prefix cache is worth deriving, because a reader who takes it as an arbitrary product decision will keep hoping for a fuzzier version of it. What is being stored is the state prefill produced, and prefill computes a representation for every token by attending to every token before it. The representation at position ten therefore depends on positions zero through nine; the representation at position nine thousand depends on all nine thousand before it. Change one byte at position k and every representation from k onward is computed from different material and is a different number. Representations before k attended to nothing that moved and are bit-for-bit what they were. There is no stored state for a middle section that could be reused independently of its prefix, because no such state exists — the middle's state is a function of the prefix.

The consequence is asymmetric, and the asymmetry is the whole answer to the objection. Editing at position n invalidates everything from n onward and nothing before it. A static line moved from the middle of the prompt to the end changes bytes at two places — where it was removed and where it was added — and the earlier of those two is the removal point, so everything before the old location is untouched and everything between the old and new locations has shifted. If the breakpoint sits at the end of the system block and the moved line lived in the messages, nothing cached moves at all.

The objection this chapter exists to answer is worth stating in the form that makes it tempting, because it is not a foolish belief and dismissing it teaches nobody anything. "Moving things around breaks prompt caching" is true of exactly one class of move: an edit whose earliest changed byte falls before the breakpoint destroys everything behind it, and a team that has spent a quarter raising a hit rate has met that class more than once. The false step is the generalisation from edits early break the cache to edits break the cache, and it is an easy step to take because a prompt is normally discussed as one object with one hit rate. Cached and uncached are not properties of a prompt. They are properties of a position inside one, and the position is the thing to compare a diff against.

The break-even follows from the write premium. With a five-minute entry, one write plus one read costs 1.35× base against 2× for two uncached requests, so a prefix that is reused even once has paid for itself; with a one-hour entry the write costs 2× and you need three requests to come out ahead. That is the entire economic argument for caching, and none of it depends on where a static line sits, as long as it sits inside the span that is identical across requests.

Both of those comparisons are additions, and doing them settles a question teams otherwise settle by preference. Take one unit to be what the prefix costs at base rate. Two uncached requests pay a unit each, so two units. Two requests across a five-minute entry pay the write premium on the first, which is one and a quarter units, plus a read on the second at about a tenth of a unit — the 1.35 above, against two. Two requests across a one-hour entry pay two units for the write plus a tenth for the read, which is a shade over two and therefore slightly worse than not caching at all. The third request is what turns it: it adds another tenth of a unit where an uncached request would have added a whole one. Which lifetime to buy is therefore a statement about how often a given prefix recurs, and not about how large it is.

resp = client.messages.create(
    model="claude-opus-5",
    max_tokens=1024,
    system=[{
        "type": "text",
        "text": STATIC_POLICY,                       # identical for every request
        "cache_control": {"type": "ephemeral"},      # 5-minute entry; "ttl": "1h" available
    }],
    messages=[{"role": "user", "content": build_body(docs, tenant_rules, question)}],
)

u = resp.usage
assert u.cache_read_input_tokens > 0                 # the only proof that any of this worked
print(u.input_tokens, u.cache_creation_input_tokens, u.cache_read_input_tokens)

The assertion is not decoration. input_tokens reports only the uncached remainder, so a request whose prefix is being served from cache shows a small input_tokens and a large cache_read_input_tokens, and a team reading input_tokens alone will conclude their prompt is much shorter than it is. If cache_read_input_tokens stays at zero across repeated requests that should share a prefix, something in the prefix is varying, and the next step is to diff the rendered bytes of two consecutive requests rather than to reason about it.

The layout that does destroy the cache#

A conversational assistant sends about 9,000 tokens on every step of every session and makes roughly 3 million model calls a day. The first 400 tokens are per-user context that product added for personalisation: display name, plan tier, locale and spelling preference, and a rolling summary of the user's last three actions. The remaining 8,600 tokens are static policy and tool descriptions, byte-identical across every user and every request. The cache hit rate is under 5%.

The mechanism is now mechanical rather than mysterious. Matching starts at token zero, and token zero belongs to a block that differs for every user; the rolling action summary makes it differ between two consecutive requests from the same user. So the prefix diverges immediately, the 8,600 static tokens sit behind a wall of variability, and no breakpoint placed anywhere can rescue them, because a breakpoint marks where a cacheable span ends, not where matching is allowed to begin. The residual few percent comes from the small number of sessions where two steps fire within the same second and nothing in the personalisation block changed between them.

The bill makes the size of the mistake legible. At 3 million calls of 9,000 tokens, the system processes 27 billion input tokens a day; at $3 per million that is about $81,000 a day with no caching. With the static block first and the personalisation moved after it, 8,600 tokens per call are served as cache reads at roughly $0.30 per million and 400 are charged at full rate, which is about $0.00378 per call, or $11,340 a day plus a modest write cost — the entries are rewritten only when the static block itself changes, not per user. Around $69,000 a day, from moving a paragraph.

Every figure there is one multiplication, and separating them shows that the two layouts differ in exactly one of them. Three million calls carrying nine thousand tokens each is twenty-seven billion input tokens a day, and twenty-seven billion at three dollars a million is the $81,000. With the static block first, the same call is billed in two pieces instead of one: eight thousand six hundred tokens read from cache at thirty cents a million, which comes to a little over a quarter of a cent, plus four hundred tokens at full rate, which comes to a little over a tenth of a cent. Those two add to the $0.00378 per call, and three million of those is the $11,340. The token count did not change, the model did not change, and nothing about the answers changed. What changed is which rate applies to which part of the same prompt.

There is a second reason the 400-token block cannot be rescued in place, and it is easy to miss. The minimum cacheable prefix is model-dependent — 512 tokens on Claude Opus 5, 1,024 on Claude Opus 4.8 and Claude Sonnet 5, 4,096 on Claude Opus 4.6 and Haiku 4.5 — and a span shorter than the minimum silently fails to cache with no error and no warning, reporting cache_creation_input_tokens: 0. A 400-token per-user block is below every one of those thresholds, so even a per-user cache that could in principle serve a returning user has nothing to store.

Forty thousand tenants and a 4% hit rate#

The same shape appears in multi-tenant systems in a form that looks more defensible. The prompt is 6,000 tokens of static policy, and each tenant's own instructions — "use UK spelling", "never use emoji", "address the customer by surname" — are interpolated at the very top, where a prompt author reasonably puts the thing that identifies whose request this is. Compliance with the tenant rules is high. The hit rate is 4%.

Two facts explain the 4%, and both are worth computing rather than estimating. First, a per-tenant prefix can only be shared with that tenant's own subsequent requests. At 900,000 calls a day across 40,000 tenants, the average tenant sends 22 calls a day, which is one every 64 minutes against a five-minute entry, so the average tenant's second call arrives long after their first entry expired. Second, the distribution is heavily skewed: the handful of tenants sending thousands of calls an hour do get hits, and they are essentially the whole 4%.

Both of those divisions repay being done rather than read past. Nine hundred thousand calls shared between forty thousand tenants is twenty-two calls per tenant per day, and dividing the minutes in a day by twenty-two is where the 64 comes from. A five-minute entry has been gone for the better part of an hour by the time that tenant's next request arrives, so the average tenant writes an entry on every request and reads one on almost none — which is worse than not caching at all, because a write costs more than an ordinary request. The skew is what keeps the number from being lower still. An average of twenty-two calls a day is perfectly compatible with a handful of tenants sending thousands an hour, and those tenants, and only those, have a second request arriving while their first entry is still alive.

The fix is to put the static policy first so that one prefix is shared by all 40,000 tenants, with the breakpoint at its end, and move the tenant instructions to the end just before the request — which is also, by everything in the first four chapters, the strongest position for a short set of rules to survive to generation time. That coincidence is genuinely convenient and is also a trap for the measurement, because the change moves two things at once: the cache economics and the compliance of the tenant rules. Both have to be measured, per rule, with the replication discipline from chapter 2, or a compliance regression on one tenant rule will be attributed to the cache work and go unfixed for a quarter.