The minimum cacheable prefix

Cache savings are the product of two numbers: how many tokens sit in the shared prefix, and how many times that prefix is read before it expires. Both can be small, and when either is small enough the mechanism this topic has been about does not run at all. A 12,000-token prefix read 90,000 times a day and a 220-token prefix read 1.4 times are four orders of magnitude apart, and the habits that are correct on the first are not scaled-down versions of anything — they simply do not apply on the second.

The floor is the part that surprises people, because it is enforced silently. A request carrying a cache_control marker on a block below the provider's minimum is accepted, validated and served normally, with no error and no warning, and the only symptom is a pair of counters that stay at zero. Topic 2.15 established that a prefix below the minimum caches nothing; what a small endpoint adds is that you can ship the optimisation, watch it deploy cleanly, and never find out.

The thresholds that decide whether the mechanism exists#

These numbers change with model releases and with pricing pages, so treat the table as the set of questions to ask about a provider rather than as a set of facts to memorise. The shape is stable across all of them: there is a floor on prefix length, a lifetime that starts short, and a read discount large enough that the whole argument in this topic only exists above the floor.

A 220-token prefix is under every floor in that table, on every provider, at every tier. There is no configuration that makes it cacheable and no model in the list where a marker would have done something.

Making the prompt longer to clear 1,024 tokens is the move that occurs to people next, and it is worth doing the arithmetic rather than dismissing it, because the intuition behind it is not silly: the discount applies to the whole prefix, so a longer prefix is a larger discount. The cost side is roughly 700 tokens of padding carried on every request. Work through what each outcome is then worth. Today the 220 tokens are simply sent, at the base rate, once per request. Padded to 1,024 and read from a warm entry, they are charged at a tenth of 1,024, which is about half of what sending 220 uncached costs — a real saving. Padded to 1,024 and written cold, they are charged at a quarter above 1,024, which is roughly six times what sending 220 costs. So the padded version wins narrowly when it hits and loses heavily when it misses, and on a surface where nearly a third of requests are the first in their window the misses dominate. Padding never comes out ahead, and the reason is that it inflated the quantity that both the discount and the premium are computed against.

What the optimisation is worth if it works#

Suppose the floor did not exist. A ticket-classification endpoint taking about 40 requests an hour is 40 times 24, or 960 requests a day, and at about 300 tokens a prompt that is 288,000 input tokens a day. At $3 per million that is $0.86 a day, and if the endpoint runs on a cheaper model at $0.80 per million the real number is $0.23 a day — about $7 a month for the whole surface, prefix and ticket text together. That figure is the ceiling on everything that follows. No layout change to this endpoint can save more than $7 a month, because $7 a month is all of it.

A week of request logs run through the reuse measurement from chapter 2 says the arrivals are burstier than the average suggests: the 40-per-hour figure is a mean over a nine-hour working window with nothing overnight, and 31% of requests were the first in their five-minute window, so the achievable hit rate is around 69%.

The two multipliers in what follows are differences from the base rate rather than the rates themselves, which is the step to get right. A hit is charged at 0.1× where it would otherwise be 1.0×, so it saves 0.9 of a token for every token in the prefix: 0.9 × 220 = 198 effective tokens saved. A miss is charged at 1.25× where it would otherwise be 1.0×, so it adds 0.25 of a token per prefix token: 0.25 × 220 = 55 effective tokens. Weight those by how often each happens across 960 requests, 69% saving 198 and 31% adding 55, and the net is roughly 116,000 effective tokens saved across the day. On this endpoint's cheaper model that is $0.09 a day, or $2.80 a month — against a prompt that now reads in an order chosen for a mechanism rather than for a human, and against the two hours it takes to write and review the change.

Checking that the mechanism engages, in about ten lines#

The check belongs in the test suite for every prompt that carries a breakpoint, because the failure is silent by construction and it recurs. A prompt that cleared 1,024 tokens last quarter can fall under it after someone trims the policy, and the counters go quietly to zero again.

def assert_cache_engages(build_prompt, case, client, model):
    """Two identical calls: the second must read from cache, or the marker is decoration."""
    req = build_prompt(case)
    first = client.messages.create(model=model, **req)
    assert first.usage.cache_creation_input_tokens > 0, (
        f"no entry written — prefix is {first.usage.input_tokens} tok, "
        "likely under the provider minimum"
    )
    second = client.messages.create(model=model, **req)
    assert second.usage.cache_read_input_tokens > 0, "entry written but never read"
    return {
        "prefix_tokens": second.usage.cache_read_input_tokens,
        "fresh_tokens": second.usage.input_tokens,
    }

The two assertions are checking two different failures and neither one implies the other. The first call must report a non-zero cache_creation count, which is the provider saying an entry was written — if it is zero, the block in front of the marker was under the floor and the marker was ignored. The second call must report a non-zero cache_read count, which is the provider saying that entry was found again. Writing without reading is the layout in chapter 1 that cost more than no caching at all, and it would pass a check that only looked at the first counter.

Two calls and about $0.02 of tokens answer the question that a pull request description asserted. Pair it with the reuse measurement against production logs, since the assertion proves the prefix is long enough while the log measurement proves it is read often enough, and a prefix that clears the floor but is reused 1.1 times inside the TTL is worth as little as one that never cached — the savings are the product of prefix length and reuse count, and a product with a small term in it is small however large the other term is. Both numbers belong in the layout record from chapter 6, so the next person can see that the mechanism was checked rather than assumed.

One layout across three surfaces#

The proposal that follows a successful caching project is to standardise every prompt in the company on stable content first, variable content last, for consistency and so that new services get caching by default. Three surfaces at one company answer it three different ways.

The agent is the surface such a proposal is written for, and the answer is the layout from chapter 5: immutable policy and schemas first, bounded tail, a short restatement of the rules that carry floors, with the paired eval scoring both arms before anything ships. The one caveat is that "agree" here means agreeing to the layout, not to skipping the measurement, and at 40 requests a second a compliance regression reaches a lot of customers before anyone posts in a channel about it.

The summariser agrees for a different reason. Stable-first is right because the 6,000-token prefix is read 4,000 times in a 25-minute window, so the single write is amortised over thousands of reads and the write premium disappears into the rounding. There is no latency requirement either, which also means extra tokens are the cheapest currency available here — if the eval prefers instructions late, duplicating them at the tail costs money and nothing else.

The failure specific to this surface is concurrency at the start of the run, and it comes from a gap the reuse arithmetic does not model. An entry exists only after a request that wrote it has completed. Firing 200 requests in parallel at t=0 means all of them look the entry up before any of them has finished writing it, so all of them miss and all of them pay the write premium. The fix is one warm-up request completed before the fan-out, which turns 200 cold writes into one — and the general form is that concurrency, not just arrival rate, decides how much of the premium is wasted.

The classifier declines, and the reason is not that the layout is harmful but that it is inert. Below the minimum there is no reuse to gain, and in a 300-token prompt there is effectively no middle for a rule to get lost in — every position is within a few hundred tokens of an edge, which is the regime where the position curve is flat. What remains is a prompt whose order was chosen to satisfy a mechanism that does not run, which the next engineer will read as a constraint and preserve. That is the whole cost, and it is small, and it is also the entire benefit, which is what makes it a decline rather than a debate.

Standardise everything except the order#

What is worth standardising across surfaces is real, and it is not the layout. One assembly interface that takes named segments and a declared breakpoint, so a layout is data that a test can read rather than a shape buried in string concatenation. Usage telemetry on every surface — cache_read, cache_creation, input per request — so the question of whether the mechanism engages is answered by a dashboard rather than by an experiment. A registry of compliance floors with an owner against each. And the layout record itself, required per prompt, naming the traffic mix and the degradation accepted.

Those four make every surface's decision legible and let each surface decide differently, which is the outcome a standardisation proposal usually wants underneath the wording it arrived in. Imposing one order does the opposite: it takes the surface where the mechanism does not run and freezes a shape into it that nobody will be able to justify in a year, while doing nothing about the surface where a compliance regression at 40 requests a second is the actual risk.

Taking that to the next prompt you assemble is three questions, in this order, and the order is the point: each one is cheaper to answer than the one after it, and a no on any of them makes the rest unnecessary. Does the mechanism engage — is the invariant block above the provider's floor, and is it read more than a couple of times inside the TTL against real logs? Which rules in this prompt have floors with owners, and where do those rules currently sit relative to the end? And what does the record say you gave up — extra tokens, cache misses, accuracy or latency — with the traffic mix that made that the right trade?

A prompt that has answers to those three is a prompt somebody chose. Anthropic's "Effective context engineering for AI agents" is the natural next thing to read, because it takes the same discipline out past ordering into what belongs in the context at all, which is the question that decides how much material you are ordering in the first place.