Prompt caching

A call to a hosted language model is stateless. Whatever the provider computed for the previous request is gone, so each prompt is read from the beginning and converted into the internal working state the model needs before it can emit a first token — a phase called prefill, billed as input tokens and paid for on every call. Prompt caching is the mechanism the model-serving API offers to stop paying for that work repeatedly: the provider retains the working state for a leading run of tokens after computing it once, then hands that state to the next request whose prompt begins with exactly the same tokens. Those tokens are not prefilled again and not billed at the ordinary input rate; on hosted APIs a reused token typically bills at around a tenth of it. The waiting goes too, since prefill is time the caller spends before the first word of the answer appears. This topic is about the distance between switching the feature on and actually collecting that discount, and every chapter in it turns on one question about the mechanism just described: what the retained state is keyed on.

The answer is content rather than identity. The rest of this chapter separates the provider's cache from the key/value cache that runs inside a single generation, because carrying the mental model of the second one upward produces a specific and expensive wrong conclusion about which traffic benefits. You meet the feature in three places in a working system. It appears in the request as a marker on one content block declaring that the reusable region ends there, or as nothing at all on providers that look for repeated prefixes by themselves. It appears in the response as two extra counters in the usage object, reporting how many tokens were written into the cache and how many were read back out of it. And it appears on the invoice as input line items priced above and below the ordinary rate, because writing an entry costs more than an ordinary send while reading one costs far less.

Two caches that share a word and not much else#

Topic 1.10 established what happens inside a single generation: the keys and values computed for each token are retained so that step n+1 does not recompute the attention state for tokens 1 through n. That structure is bound to one forward pass over one sequence. It is addressed by position, it lives in the memory allocated for that request, and when the final token is emitted the allocation is released. Nothing about it is designed to be found again, because nothing is expected to look for it — the only consumer is the next decoding step of the same generation.

A prior question sits underneath both caches, and the whole feature rests on it: how state one request computed can be handed to a different request without corrupting the answer. That turns out to be a property of prefill rather than of caching. When the model reads a prompt, the state it computes for position n — the keys and values topic 1.10 describes — is built from the tokens at positions 1 through n and from nothing else. A token is never influenced by a token that arrives after it, because the whole arrangement exists to predict what comes next, and a prediction allowed to see ahead would be measuring nothing.

Run that property forward and the handover stops looking risky. If request B's first 15,000 tokens are byte-identical to request A's first 15,000 tokens, then the state B's own prefill would compute for those positions is built from exactly the inputs A's prefill was built from. Nothing in B's suffix — the question its caller actually asked — could have reached back and altered any of it, because reaching back is the one thing the arrangement forbids. So A's state is not an approximation of what B needs. It is the thing B needs, and skipping the recomputation is not a shortcut with a quality cost attached to it.

The suspicion also runs the other way, and that version deserves a direct answer: if a request is served state computed for somebody else's request, does anything of theirs arrive with it? What transfers covers only the positions where the two prompts agree, and by the property above nothing from A's suffix ever entered those positions. Scoping entries to one organization and one model version is a second line of defence on top of that, not the reason it is sound.

A provider's prompt cache sits a layer above the in-generation cache and answers a different question: has anyone, recently, sent a request that begins with these exact tokens? The stored object is still key/value state for a span of tokens, but the addressing is what changes everything. The entry is looked up by a hash chain computed over the serialized prefix itself — serialized meaning the exact byte sequence your request objects are turned into on the wire, punctuation and field order included — walked forward from position zero, scoped to a model version and to your organization. Nothing in that key mentions a session, a conversation ID, a user, or a previous request. Two requests match if their bytes match.

In that description the word doing most of the work is chain rather than hash. A single hash over the whole prompt would answer one question — have I seen this entire request before — and that is a response cache, a different product. A prefix cache needs an answer at many points along the prompt, because the case it serves is two requests that agree for a long stretch and then stop. So the key is not computed once. The prompt is cut into fixed-size runs of tokens, and the value stored for each run is computed from two inputs: the bytes of that run, and the value already computed for the run before it.

Folding the previous value into the next one is what makes every position's key depend on every byte in front of it. The value at the third run was computed from the value at the second, which was computed from the value at the first, which was computed from the opening bytes of the prompt. Unrolled, the value at any run is a function of the entire prefix leading up to it, condensed into one number that no other byte sequence is expected to produce.

Follow what happens when one byte near the front changes. The run containing that byte computes a different value, because one of its two inputs changed. The run after it computes a different value as well, because one of its inputs was the value that just changed, and every run after that inherits the same fate to the end of the prompt. The lookup for the first altered run finds nothing, and so does the lookup for every run behind it. This is the mechanism underneath the flat statement that matching runs from position zero. It is not that the provider decides to stop at the first difference. It is that past the first difference there is nothing left to find, because no entry was ever written under those values.

The chain also settles the question a reader is right to ask next, which is why the provider does not simply resume the match after the changed field. Suppose a whole 15,000-token preamble sits behind one interpolated name. Two separate things block its reuse, and only the first is about addressing. The chain values across those 15,000 tokens have all moved, so nothing addresses the stored entry any more. And even given a way to address it, the state inside it would be the wrong state: the keys and values at those positions were computed while attending to the bytes in front of them, and those bytes are no longer the same bytes. Diffing the two prompts would find the text identical and could not make the state identical.

So the hashing does one job and the causal structure of prefill does another. The hash turns a lookup into a single probe rather than a comparison against every prefix the provider has ever stored. The causal structure guarantees that an entry found that way is genuinely valid for the request that found it. Both run forward from the first byte and both stop where the bytes stop agreeing, which is why every rule in the rest of this topic turns out to be a rule about the front of a prompt.

The last two rows are the reason this topic exists at all. The in-generation cache is not a thing you design against; it is arithmetic the runtime does on your behalf and gets right without your involvement. The provider cache is a thing you design against, and the design is the shape of your prompt.

Reuse is a property of a population of requests#

Once the key is content and the scope is an organization rather than a session, the question "does this help a one-shot endpoint?" becomes a question about the population of requests arriving in a window, not about the structure of any one of them. Fifty requests a minute that share 15,000 leading tokens produce one cache entry and forty-nine reads, because the fiftieth caller's prefix is byte-identical to the first caller's prefix and the cache has no way of knowing or caring that they are different people. That count covers the first minute only, and it is worth being exact about which minute it is: one fill and forty-nine reads while the entry is being established, then fifty reads a minute for as long as it survives.

That needs saying because "whose cache is it?" is the natural first question and it has no answer here. Almost every other cache an engineer meets is keyed on identity — a session store keyed by session, a browser cache scoped to one profile, a CDN told to vary its entries on a cookie so that two users never see each other's page. Each of those exists because serving one user's stored bytes to another would be a bug. Here it is the entire point, for the reason established above: the bytes being shared are bytes both callers sent.

Diagram

Take the common hosted pricing shape, where a write costs 1.25 times the base input rate and a read costs 0.1 times, and run it against that traffic. Those two multipliers set the unit the rest of the arithmetic is in. An effective token is a token priced at the base input rate, so a token read back from cache counts as a tenth of one and a token written into cache counts as one and a quarter. Counting in effective tokens rather than in currency keeps the comparison true whatever a particular model happens to charge per million.

In the uncached world every request bills 15,200 tokens at full rate — the 15,000-token shared preamble plus the customer's 200-token message, none of it discounted, on every one of the fifty requests a minute. The cached world cannot be counted until the entry's lifetime is fixed, because the lifetime is what decides how many requests get to share a single fill. Take the five-minute figure this topic uses throughout. Fifty requests a minute for five minutes is two hundred and fifty requests arriving against one entry: the first finds nothing and fills it, and the rest read it. So in the cached world, one request per five-minute window pays 15,000 × 1.25 + 200 for the fill and the other 249 requests in that window pay 15,000 × 0.1 + 200.

Work the second of those out, since it is what almost every request pays. A tenth of 15,000 is fifteen hundred effective tokens for the preamble. The 200-token message was never in the cache and never could be, since it is different on every call, so it is billed in full and added on unchanged. The request comes to 1,700 effective tokens against the 15,200 it would have cost uncached, which is a little over a tenth.

Per minute that is roughly 50 × 1,700 effective tokens instead of 50 × 15,200. The word carrying the weight in that sentence is roughly, and it is worth saying what is being averaged and why averaging it is legitimate. One request in every two hundred and fifty is a fill rather than a read, and a fill is dear: a quarter above full rate on 15,000 tokens comes to something over eighteen thousand effective tokens, against a read's 1,700. Spreading that excess across all the requests that share the window adds well under a hundred effective tokens to each of them. Quoting the whole window at 1,700 a request therefore understates the bill by a few per cent rather than by a factor, which is the accuracy a comparison against 15,200 needs. The savings do not come from any request being repeated; they come from a prefix being repeated across requests that have nothing else in common.

Multi-turn conversation is one way to generate that repetition and it is not a particularly good one. A single person in a chat produces one new matching request every few seconds at best, gated by how fast a human types, and the growing conversation means each turn extends the prefix and pays a fresh write for the extension. A stateless classifier produces fifty matching requests a minute with no extension at all, from a prefix identical down to the byte on every call. Between a chat with a hundred concurrent sessions and a stateless endpoint at fifty requests a minute, the stateless endpoint is the better caching target.

The population view gets complicated in exactly one place, which is the first second after an entry disappears — after a deploy, after a model change, or after a quiet period long enough for it to expire. Fifty requests a minute means requests arrive roughly every 1.2 seconds, and a prefill of 15,000 tokens takes on the order of half a second, so the number of requests that miss before the write lands is small. Push the rate to five hundred a minute and several requests are in flight simultaneously against an entry that does not exist yet, each of them paying the write premium for a fill the others are also performing. The steady state is unaffected and the transient is worth knowing about, because it makes a cold cache more expensive than the same traffic uncached for as long as it lasts.

Where the shared prefix quietly stops being shared#

The failure that actually shows up on an endpoint like that one is not that the traffic shape is wrong. It is that something inside the 15,000 tokens is not the same on every request, and it only takes one field. A rendered clock at the top of the policy, a request ID injected by a middleware for tracing, a customer's display name interpolated into the greeting line, an experiment bucket appended to the instructions — any of these makes each request's prefix unique, and because matching runs from position zero, uniqueness anywhere in the prefix discards everything behind it. The endpoint then reports a hit rate of zero while paying the 1.25 write premium on every one of its 72,000 daily requests, which is a bill about 25% higher than the uncached baseline it replaced. That specific failure has three common causes and one cheap standing defence, which is chapter 8.

Those four examples differ in where the varying field sits, and the chain decides what that costs. A clock at the very top moves the first run's value, and every run behind it inherits the move, so the whole 15,000 tokens go. An experiment bucket appended to the end of the instructions moves the last run's value and whatever follows it — which, if the bucket really is the final thing in the preamble, is nothing at all. A varying field is therefore not uniformly fatal. It costs exactly what sits behind it, which is why moving a stray field down past the stable blocks is usually cheaper than removing it, and chapter 2 turns that into a rule.

The bill in the zero-hit state is worth deriving rather than accepting, because a hit rate of zero reads as a missed opportunity and is considerably worse than one. On a provider that charges a premium to write, every request that finds no entry creates one: its runs carry values nothing has ever been stored under, so there is nothing to read and the segment is filled afresh at 1.25 rather than at the base rate. Paying a quarter more on 15,000 of a request's 15,200 tokens is paying a quarter more on very nearly the whole request, which is where the figure of about 25% higher comes from. The write premium is the condition that makes it a surcharge, and the condition belongs attached to the claim: chapter 3 covers the other provider design, under which a miss bills at the ordinary rate and a hit rate of zero genuinely is neutral.

There is also a scoping condition that people discover the hard way when they start measuring. The key includes the model identity and the organization, so a routing layer that spreads traffic across two model versions for an A/B test splits one entry into two populations, each with half the reuse, and a change of model on a Tuesday deploy resets the hit rate to zero until traffic refills it. On self-hosted stacks the equivalent condition is physical: the entry lives on the replica that computed it, so a load balancer that spreads identical prefixes evenly across eight nodes gets eight cold fills instead of one. Neither of these is a bug in caching; both are cases where a decision made elsewhere in the system quietly partitioned the population that was supposed to be sharing an entry.

Two further conditions reliably block the win and are worth naming now so they are not mysteries later. The prefix has to be long enough to qualify — providers publish a minimum, typically in the region of 1,024 tokens, below which a cacheable marker is accepted and quietly ignored. And the entry has to still be alive when the next matching request arrives, which turns the arrival pattern of your traffic into the thing that decides your hit rate. Fifty requests a minute against a five-minute lifetime is comfortable; four requests an hour against the same lifetime means every request is a cold write.

Prompt caching next to a response cache#

A prompt cache is not a response cache. If an entire request repeats — same prefix and same user message — then a hash-keyed store in front of the model returns the completed answer in a millisecond for no inference cost at all, which beats any prompt cache by orders of magnitude. Prompt caching earns its place precisely where a response cache cannot help, which is when the suffix differs on every call and the model genuinely has to run. The two are complementary, and a system with a high rate of exactly-repeated questions should have both.

The word cache suggests one thing here that is not true, and it is worth blocking before it does damage. A hit does not return a cheaper copy of an earlier answer. The model still runs on every request: the state for the shared preamble is loaded instead of recomputed, the differing suffix is prefilled as normal, generation proceeds from there one token at a time, and the output side is billed exactly as it would have been. Two requests reading the same entry usually get entirely different answers, because they asked different questions. Even two byte-identical requests that both hit can differ, since the step that picks each next fragment is allowed to be random — that is topic 1.0's subject — and nothing about a cache hit reaches that step.

Nor is the matching semantic. Nothing in the lookup asks whether two prompts mean the same thing; it compares serialized bytes forward from the first one. "You are a support triage assistant." and "You are a support triage assistant" differ by one character and share nothing. Share nothing is literal rather than rhetorical: the run holding that sentence hashes to an unrelated value in the two cases, and an unrelated value is exactly as unfindable as a wildly different one. Hash construction has no notion of nearness, deliberately — a scheme in which similar inputs produced similar values would be one in which a near-match could be mistaken for a match, and a near-match is precisely the case where the stored state is wrong. That is why prefix caching is reliable and boring in a way semantic caches are not — there is no similarity threshold to tune and no risk of serving an answer conditioned on the wrong context — and also why it is so easy to lose by accident.

The saving is confined to the input side. Everything above concerns input tokens and the prefill phase that consumes them, so an agent that reads 15,000 tokens and writes 3,000 tokens of tool calls and prose sees its input bill collapse and its output bill unchanged, and the headline percentage on the input side always overstates the effect on the invoice total. The same is true of latency: a hit removes prefill work and therefore time to first token, while the rate at which subsequent tokens arrive is exactly what it was before.

A prompt cache, then, is a content-addressed store of prefill work, shared across every request your organization sends to one model version, and the unit of sharing is a run of identical leading bytes. That single fact reorganises what counts as a good caching target: not the long conversation, which generates a handful of matching requests at typing speed, but the busy stateless endpoint whose callers have nothing in common except the enormous preamble they all carry.

None of it is automatic in the sense of being free of design. Matching runs forward from the first token, which makes the arrangement of blocks in a prompt something other than a stylistic choice. It decides how much of the prompt is eligible for reuse and how many different requests are able to reuse it. Getting that arrangement right is the next thing, and it is the one change that converts a correct intuition about populations into an actual line on the invoice.