A cached prefix only pays off if nothing mutable sits in front of it

Prompt caching is the provider feature that lets a stable run of tokens at the front of a request be paid for once rather than on every request that carries it. Having read that run once, the provider keeps what it computed while reading it; when a later request begins with exactly the same tokens, it charges a cache read at a deep discount rather than reading them again, and it charges a cache write at a premium on the request that stored them in the first place. Chapter 1 priced a feature that answers questions about a single 250,000-token master services agreement, twelve questions to a session, and every figure in it rested on the assumption that no such discount exists: the model reads the agreement from scratch on each of the twelve requests and is charged full input price each time.

The condition attached to that discount is narrower than the API makes it look, and it is what this chapter is about. An entry can only be matched by a request whose tokens agree with it from the very first position onwards, so the discount reaches a document only while nothing that varies sits in front of it. Whether that holds is settled by the order in which the blocks of a prompt are assembled, a decision somebody makes while writing a template and usually does not know they are making. The consequence is a factor of five on the session bill, and the instrument that tells you which end of that factor you are on is a counter rather than a code review.

What a cache entry is keyed on#

A model does not answer a prompt by looking things up in it. It reads the prompt through, position by position, in order, and at each position it computes a representation of everything it has seen so far, which is what the next position is then computed against. Reading a 250,000-token prompt is therefore a chain rather than a set of independent lookups. The state the model holds at the last token of the agreement is a function of the agreement, of everything printed in front of the agreement, and of the order the two were placed in.

That one property forces everything else in the chapter. If the stored state at some position was computed from every token before it, then handing that stored state back instead of recomputing it is only sound when every token before it is identical this time round. A single differing token anywhere earlier changes the state at its own position, and therefore at every position after it, so nothing behind the difference can be reused. There is no version of this in which a provider stores "the agreement" as a named object and matches it wherever it turns up, because the agreement's stored state was never a property of the agreement alone.

So the key is the leading run of tokens itself, counted from the first one. That run is the prefix, and a breakpoint — the marker that tells the provider how far into the request to store — chooses where the run stops. It does not choose where the run starts. A breakpoint placed after a quarter of a million tokens of contract asks for a very large entry, and it asks for that entry to be keyed on everything in front of the contract as well as on the contract.

The sentence a reader would confidently write after seeing a cache marker in a request is that the block it is attached to is cached. That sentence is true, and it is not the sentence that decides the bill. The marker does exactly what it advertises: it stores from the start of the request up to and including that block. What it cannot do, and what nothing else in the request can do either, is make the tokens in front of the block repeat. A prompt can be marked correctly, honoured correctly by the provider, and still produce entries that will never be matched by anything — every part of that is compatible with counters showing writes and almost no reads.

How many hits repay a write#

Cache writes cost 1.25× the base input rate and cache reads cost 0.1×, which are the common multipliers on hosted APIs; substitute your provider's if they differ, since the shape of the result does not depend on the exact figures.

Those two multipliers settle how many matching requests it takes before caching is worth switching on, and the answer is short enough to derive rather than assume. Take a prefix of any size, and let one unit be whatever a single uncached read of that prefix costs. Send m requests in a row, all of them beginning with that prefix. Uncached, the prefix is billed at full rate every time, which is m units. Cached, the first request writes it and the remaining m − 1 read it, which is 1.25 + 0.1 × (m − 1) units. Caching is the cheaper of the two exactly when

1.25 + 0.1 × (m − 1)  <  m

Both sides describe the same m requests, so the comparison can be simplified by doing the same thing to each side. Subtract 0.1 × m from both, which is allowed because subtracting one quantity from both sides of an inequality leaves the inequality true. On the left, 0.1 × (m − 1) minus 0.1 × m leaves −0.1, so the left side becomes 1.25 − 0.1. On the right, m minus 0.1 × m is m × (1 − 0.1). Then divide both sides by (1 − 0.1). That factor is positive, and dividing an inequality by a positive number preserves its direction — the step is worth pausing on, because dividing by a negative one would reverse it.

What survives is m on one side and, on the other, the amount by which a write exceeds a cache read divided by the amount by which a plain uncached read exceeds a cache read. With these multipliers that quotient is a little over one. Two requests sharing a prefix are already enough to repay the write, and every request after the second is pure discount. That is why prompt caching reads as free money on a whiteboard.

Production traffic does not arrive in tidy runs of m, so the useful form of the same comparison is written in terms of the hit rate: the fraction of requests that find their prefix already stored. Call that fraction h. A hit costs 0.1 units. A miss costs 1.25 units rather than 1.0, because a miss does not merely fail to save anything — it writes a fresh entry. The average request therefore costs 1.25 × (1 − h) + 0.1 × h units, against 1 unit for not caching at all. Collecting the terms in h turns the left side into 1.25 minus h times the gap between a write and a read, so caching wins when h exceeds the amount by which 1.25 overshoots 1, divided by that same gap. That threshold works out at a little over one request in five.

Two statements are now licensed, and they are worth keeping apart because they sound like the same statement. A cache that is hit at all repays its write almost immediately: that is the m derivation, and it describes the mechanism. A cache hit less often than roughly one request in five is a net cost against having no cache: that is the h derivation, and it describes a deployment. The gap between the two is the gap between asking whether the mechanism works and asking how often it fires, and a design review answers the first question while an invoice answers the second.

What eleven cache hits do to the session bill#

Take the same session as chapter 1 — one 250,000-token agreement, twelve questions, one conversation — and price it three ways.

Read that table across rather than down, because the teaching is in the differences between columns and not in the levels inside them. Between the first two columns exactly one thing changes: eleven full-price reads of the agreement become eleven discounted ones, so $8.25 becomes $0.83, while the write premium moves turn one the other way, from $0.75 to $0.94. Between the second and third columns the discount plays no part at all; what changes is how many tokens are sent, a quarter of a million against the 57,600 that twelve retrieved prompts carry between them. Those are two independent levers, and neither of them can do the other's job.

Before leaning on the totals, it is worth checking that they close, and one of them does not close on the numbers printed above it. Recompute each stuffed column from its token counts rather than by adding its printed cents. With 46,200 in the question-instructions-history row, each of the two stuffed columns lands one cent below the total printed beneath it. With the 49,800 that the retrieval column carries in the same row, both land exactly on the printed totals. A single cent looks like rounding, and rounding does not miss in the same direction in two columns computed from two different multipliers.

The reason is visible in the row's own label. It promises the question, the instructions and the history, and 46,200 is the conversation history alone; 49,800 is that same history plus twelve turns of the 300-token instruction-and-question block. The stuffed design sends those blocks too — it is the same conversation, asked of a different prompt — so the row is one component short in two of its three columns while both totals were computed as though it were not. The correction is worth about a cent a session and changes no argument here. What it buys is the right to use $9.22 and $1.98 as given, which several later steps do.

The stuffed design got 4.7× cheaper and the gap to retrieval fell from 23× to 4.9×. That is the correction the shallow version of this comparison misses, and it is worth stating precisely what it does and does not do to the decision. It does not make retrieval pointless — a factor of five is a factor of five, and at 400 sessions a day it is $229 a day, about $84,000 a year, which is more than the pipeline costs to build and run. It does move the break-even. In chapter 1 the retrieval pipeline paid for itself at fourteen sessions a day; at a saving of $1.58 per session rather than $8.82 it pays for itself at about seventy-nine. A break-even count is a fixed engineering cost divided by the saving per session, so dividing that saving by about five and a half multiplies the threshold by about five and a half, which is the whole of the move. A feature doing thirty sessions a day should stuff and cache and spend the engineering weeks on something else; a feature doing four hundred should still build the pipeline.

Notice which quantity the whole table pivots on. It is not the size of the document, and it is not the number of questions on its own; a 240,000-token contract and a 250,000-token one behave identically, and twelve questions behave much like ten. It is whether eleven of those twelve requests actually match a stored entry. At a hit rate of 92% the stuffed session costs $1.98; at a hit rate of 0% it costs $9.22 plus the write premium, which is $9.41, worse than not caching at all.

That $9.41 is worth taking apart, because it is a third instrument landing on the same reading of the row above. It is the uncached session plus the premium on one write that nothing ever matched: a quarter of a million tokens at the quarter by which a write exceeds a plain read, which comes to about nineteen cents. Added to $9.22 it gives $9.41, and it does so only if $9.22 is the total that includes the instruction blocks. The figure also carries an assumption worth saying out loud, because the chapter uses it as an endpoint: it prices one unmatched write for the whole session. Where the mutable block changes on every request rather than once a session, the provider writes a fresh entry every time, and the session costs more than $9.41 rather than exactly that.

Every intermediate hit rate lands proportionally in between, and the two endpoints are separated by a factor of five. Before you compare the cached column against the retrieval column, you owe yourself a defensible claim about which end of that range production will sit at, and a prompt that looks obviously cacheable is entirely capable of landing at the wrong end.

That threshold is the actual output of this analysis, and it is worth more than either architecture recommendation, because it survives changes in token prices in a way that "use retrieval" does not. Recompute it when the numbers move rather than re-litigating the architecture.

The prefix includes everything you put before the document#

A cache entry is keyed on the exact leading bytes of the request, so the practical question is not "did I cache the agreement" but "what is the longest run of tokens at the front of this request that is identical to the last one". Anything that varies per request truncates that run at its own position, and everything after it — including a quarter of a million tokens of contract — is re-read at full price no matter how carefully you marked it.

Here is a request that measured a 4% hit rate in production. The offending block is nine lines long.

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": (
                    f"Current time: {datetime.now().isoformat()}\n"
                    f"User: {user.display_name} ({user.role})\n"
                    f"Workspace: {workspace.name}\n"
                ),
            },
            {
                "type": "text",
                "text": agreement_text,          # 250,000 tokens
                "cache_control": {"type": "ephemeral"},
            },
            {"type": "text", "text": question},
        ],
    }
]

The cache_control marker is doing exactly what it says: it asks the provider to store everything from the start of the request up to and including that block. The problem is upstream of it. datetime.now().isoformat() changes on every call, so the stored entry can never be matched again, and the provider writes a fresh 250,000-token entry each time.

Follow that through to the bill rather than stopping at "the cache is not working", because the two are different situations. Each request writes a quarter-million-token entry at 1.25×, and almost none of those entries is ever read back at 0.1×, so the average request costs more than the 1.0× it would have cost with caching switched off. The derivation above puts the crossover at a little over one request in five, and 4% sits well below it. Correctly configured caching, honoured by the provider, made this feature more expensive per request than not caching would have been.

The fix is ordering, and it is not subtle once you see it.

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": agreement_text,          # 250,000 tokens, first
                "cache_control": {"type": "ephemeral"},
            },
            {
                "type": "text",
                "text": (
                    f"Current time: {datetime.now().isoformat()}\n"
                    f"User: {user.display_name} ({user.role})\n"
                ),
            },
            {"type": "text", "text": question},
        ],
    }
]

Everything mutable now sits after the breakpoint, where it is re-read at full price — 300 tokens at $3 per million, which is a tenth of a cent — and the 250,000 tokens in front of it are matched.

The asymmetry between those two arrangements is the entire technique. Moving a small block from in front of a large one to behind it changes what the small block costs by an amount too small to measure, and changes what the large block costs by the difference between 1.0× and 0.1×. The reordering is free. What it decides is not.

Ordering is part of the design, not part of the implementation#

The reason a timestamp ends up in front of a document is that prompt assembly usually grows outward from a template, and templates are written top-down in the order a human would read them: who you are, when it is, what you are looking at, what I am asking. That order is the reverse of what caching wants. Caching wants the largest, most stable thing first and the smallest, most volatile thing last, which produces a prompt that reads badly to a person and bills well.

Sorting the blocks is a five-minute exercise if you do it deliberately, and it needs to be a decision someone owns rather than an artefact of whoever last edited the template.

One column in that table does not hold the unit its header names, and it is better said than worked around. The agreement's entry is a token count — the same 250,000 the rest of this chapter prices — while the small blocks are given in bytes, and a token runs to several bytes. Nothing in the ordering depends on that column, which records what each block would be worth protecting rather than where it belongs.

Sorting by rate of change puts the breakpoint at the natural boundary, and the ordering generalises: any block that changes more often than the block below it should move down. The instructions go above the agreement rather than below because they change on a release cadence, which is slower than the session cadence, and putting them first means a second cache entry covering just the instructions is available to sessions with different documents. That is a small win here and a large one for a shared corpus, where instructions plus corpus are stable across every user in the product.

Measure the hit rate, do not infer it#

Reasoning about a cache from the code rather than from the counters is what keeps a failure like this alive, and the code looks right in every version of it. Every provider returns per-request usage that separates the three categories, and the only defensible statement about a cache is one computed from them.

def cache_stats(usage) -> dict[str, float]:
    written = usage.cache_creation_input_tokens
    read = usage.cache_read_input_tokens
    uncached = usage.input_tokens
    total = written + read + uncached
    return {
        "hit_rate": read / total if total else 0.0,
        # cost in units of "one uncached input token"
        "effective_tokens": 1.25 * written + 0.10 * read + 1.0 * uncached,
    }

The second field is the derivation above turned into a line of code. It weights each category by its multiplier, so it reports what the request cost measured in uncached input tokens, and dividing it by the plain token count gives the ratio the whole chapter turns on directly, with no pricing table in the loop. A value above the plain token count means the cache is costing money on that request.

The first field is a hit rate computed from token counts rather than request counts, and the two answer different questions. Token-weighted is the one that predicts the bill, since a request that misses on a quarter of a million tokens costs far more than one that misses on a few hundred. Request-weighted is the one that describes what users are experiencing. The two agree while every request carries the same prefix, and they diverge as soon as some sessions send a large document and others do not — which is precisely the situation in which somebody quotes whichever of the two is reassuring.

Emit hit_rate and effective_tokens as metrics on every request and alert when the seven-day hit rate falls below the value your cost model assumed. The alert matters because prefix stability is not a property you can establish once. It degrades whenever anyone adds a block to the prompt template, and the person adding a feature flag or an A/B variant to the system prompt has no reason to know that inserting it above line 40 costs $84,000 a year.

The quantity that stays reassuringly constant through all of this is the configuration. The breakpoint sits where it was put, it reviews correctly, and it does not move between releases — while prefix stability, the property that actually decides the bill, moves every time the template above it does. A reviewer looking at the marker is looking at the one part of the arrangement that is guaranteed not to have changed.

When a warm cache still does not replace retrieval#

It is tempting to conclude from the table that caching makes retrieval obsolete for any corpus that fits in a window, and the conclusion holds only under conditions this chapter's example happens to satisfy. Twelve questions amortise one write across eleven reads; a feature where users ask one question and leave pays 1.25× for a full read and gets nothing back, which is worse than not caching. That is the m derivation read from the wrong side: at one request per prefix there is no second request to collect the discount, and the write premium is the entire effect. The agreement is the same bytes for all twelve requests; a corpus assembled per query cannot be a shared prefix at all. And the entry has to still exist when the second question arrives, which chapter 1's arithmetic and this chapter's table both quietly assume by writing "turns 2–12: cache read" as though the eleven reads were guaranteed.

One condition sits underneath all of those. The multipliers are a provider's pricing decision rather than a law, so a smaller write premium moves the one-in-five threshold down and a larger one moves it up, which is why the derivation is worth carrying in symbols rather than memorising the number it produced here. A tenant override block, a locale, an experiment arm and a feature flag are all small, all individually harmless, and all fatal in the wrong position, because the cost of a block has almost nothing to do with its size and almost everything to do with how much sits behind it.

An analogy helps here, and it is only an analogy. The cache behaves like a bookmark that is checked by re-reading every page up to it rather than by its page number: change one word on page one and the bookmark finds nothing, however deep it was placed. Where the comparison breaks is the price of failure. A bookmark that fails costs nothing beyond the re-reading, while a cache write that is never matched costs a premium over having made no attempt at all, and that premium is what turns a broken cache from a missed saving into an active loss.

The eleven reads are not guaranteed. A cache entry has a lifetime measured in minutes, it is destroyed by any edit to the document it holds, and a feature whose traffic arrives unevenly re-warms far more often than its edit count suggests. Chapter 3 builds the cost model with those two facts in it, for a corpus large enough and mutable enough that they dominate.

What survives into that chapter is the shape rather than the figures. A cached design has two terms — a write term paid at a premium whenever an entry has to be created, and a read term paid at a discount whenever one is found — and every question about whether caching pays is a question about how often each term fires. The prompt template decides that, the provider reports it, and nothing in between is a reliable place to guess it from.