Minimum cacheable prefix length

The minimum cacheable prefix length is the shortest segment a provider will create a cache entry for, measured in tokens from the start of the prompt to the boundary of the segment. Hosted APIs put it in the region of one to two thousand tokens; self-hosted inference engines put it at their internal block size, which is commonly sixteen. The number is published on the provider's caching page, it differs between model families from the same provider, and it is the one caching parameter that fails without saying anything.

That silence is what makes it worth a chapter. A prompt below the threshold does not error, does not warn and does not degrade — it simply bills at ordinary input rates while the request that carries the marker looks, in a diff and in a review, exactly like a request that is caching correctly. This chapter covers where the thresholds sit, what the response's own counters say about which failure you have, and how to widen a segment that falls short without paying for filler on every call.

The threshold is a cliff, not a slope#

Providers enforce the minimum as a hard qualifying condition. A segment of 1,024 tokens caches in full; a segment of 1,023 tokens caches nothing. There is no partial credit, no rounding up, and — this is the part that produces six-week-old bugs — no error. The marker is a request for caching, the provider evaluates it, the answer is no, and the response comes back with a normal completion and a set of zeros where the cache counters would be.

An error would be the wrong response, and the reason is worth stating because it is the same reason the failure survives review. A provider returns an error when it cannot do what was asked, and here it can: the marker changes neither which tokens are sent nor which model reads them, and the completion that comes back is the same completion either way. Caching sits underneath a request that is already complete without it, so refusing the whole call over an unmet optimisation condition would fail a request the provider is able to serve perfectly. What the provider does instead is answer in the channel built for it — the usage counters, present on every response — and the silence is your code's, which reads the completion and discards the rest. An advisory marker is also what lets the threshold move at all: a provider that lowers its minimum starts caching prompts whose code nobody has touched, and one that raises it costs a discount rather than breaking a deployment.

The belief that trips people is that marking a block cacheable is what causes it to be cached. It is tempting because every other field in a request behaves that way — set a temperature and you get that temperature, set a maximum output length and it is enforced — so a cache marker reads as one more setting with an effect. It is also how caching works inside your own process, where the decorator on a function is the cache. A provider's marker is a different kind of object: it declares where a reusable region ends, and it hands that declaration to a service that evaluates it against conditions your process cannot see, among them the threshold for the model actually serving the request and the shape of the request itself. What you control is eligibility. Whether an entry exists is the provider's answer, and the usage block is where it gives it.

Those figures move, so treat the table as the shape of the landscape rather than as a source of truth, and read the current number out of your provider's documentation before designing around it. The shape is what matters: hosted APIs sit in the region of one to two thousand tokens, self-hosted engines sit at their block size, and the difference between the two is a factor of sixty. A prompt architecture that caches beautifully on a local vLLM deployment can qualify for nothing at all on a hosted endpoint, which is a genuinely surprising thing to discover during a migration.

That factor of sixty is not two opinions about the same trade-off. A self-hosted engine already allocates the state for every running request in fixed blocks, so a block exists whether or not anybody wants to share it, and making it shareable adds a hash of its contents and a reference count to memory that was being spent anyway. There is no separate store to stand up and no invoice line to attach to any of it. A hosted provider is doing something else entirely: keeping an object alive after the request that filled it has gone, on hardware serving every other customer, with a price on each entry it holds. The two numbers are answers to two different questions.

Granularity and the minimum interact in a way that costs a little more than people expect near the bottom of the range. On a provider that matches in 128-token increments, a 1,100-token prefix qualifies and then rounds down to 1,024 cached tokens, so 76 tokens are billed at full rate on every request; at 1,024 tokens exactly, the prefix qualifies and rounds to 1,024 with nothing lost. The effect is a few percent and is worth knowing only because it explains a reported cached-token count that is stubbornly lower than the prefix you believe you sent. Higher up the range it vanishes into the noise, which is why nobody notices it on a 40,000-token document and everybody notices it on a 1,200-token classifier.

The direction of that rounding is a correctness requirement rather than a billing choice. A cache read hands an incoming request state that was computed from some earlier request's bytes, and it is only allowed to do that for bytes it has checked are identical. The checking happens at the boundaries where the provider stored a hash, so on a 128-token grid the last position it can vouch for inside a 1,100-token prefix is the eighth of those boundaries, at 1,024 tokens. The 76 tokens past it have been compared against nothing. Serving them from the entry would mean reusing state computed from one sequence of bytes for a request that may carry a different one, which changes the answer rather than the bill — so rounding up is not a cheaper approximation, it is wrong, and down is the only direction available.

The reason for the threshold is that an entry has a fixed cost to the provider — storage, an index entry, and the lookup work on every incoming request — and below some size the deep discount on reads does not cover it. Setting the bar at a thousand tokens keeps the cache populated with entries whose reuse is worth the bookkeeping. It also means the mechanism is designed for system prompts, tool schemas and documents, and simply not designed for the 200-token instruction blocks that make up most of the prompts people write first.

Those three costs behave differently as an entry shrinks, and the difference is the whole of the threshold. What the provider stores is not your text; it is the state that prefill produced from it — prefill being the phase in which a prompt is read and converted into the key and value vectors the model's attention needs, one pair per token per layer — which is why an entry is very much larger than the prompt that made it, and why its storage cost tracks the token count. The index entry tracks nothing: a hash and a pointer are the same size for sixteen tokens as for two hundred thousand. Neither does the lookup, which every incoming request performs against the index whether or not it will match, so an entry nobody ever reads still costs a comparison on every request that goes past it.

Put the benefit next to that. A hit saves the prefill compute for the tokens it matched, so what an entry is worth scales with its length, while the two overheads just described do not move at all. Shrink the entry and the value falls away while the bookkeeping stays where it is, until there is a size below which an entry costs more to carry than the work it saves. That is the threshold, and it has to be settled at the moment the entry would be written, when the only thing known about it is how long it is. How often it will be read is the other half of the answer and it does not exist yet, so the token count stands in for it.

The decision the token count stands in for is a yes or a no, and that is what makes the rule a cliff rather than a slope. There is no coherent way to cache a segment partially: whatever fraction were stored would still need its own index record and would still be compared on every lookup, so the flat costs are incurred in full by an entry of any size at all. The provider's two available moves are to create the entry or not to, and a rule whose output is include or exclude cannot produce a proportional effect. A segment of 1,023 tokens is not nearly cached.

The model swap that silently switches caching off#

The most expensive version of this failure is not a block that never qualified. It is a block that qualified and then stopped. A team running a 1,500-token classifier prompt on a mid-tier model decides to move it to the cheapest model in the family for cost reasons, changes one string in the config, and ships. The per-token price drops by a factor of four, which is the change they were looking for, and the minimum cacheable segment rises from 1,024 to 2,048 tokens, which is a change nobody looked for. The prompt is now below the threshold, the cache goes cold permanently, and the effective input cost per request rises from roughly 0.1× the prefix to 1.0× of it.

Work through it with numbers. At 2 million requests a day, a 1,500-token prefix on the mid-tier model at $3 per million input tokens costs 1,500 × 0.1 × 2e6 = 300 million effective tokens, or about $900 a day. On the cheap model at $0.80 per million with caching disabled by the threshold, it is 1,500 × 1.0 × 2e6 = 3 billion tokens, or about $2,400 a day. A migration to a model priced at a quarter of the rate made that prompt block 2.7 times more expensive, and every dashboard tracking price-per-million-tokens rather than spend-per-request reported an improvement.

Notice what is held fixed across those two calculations, because that is what makes them comparable at all. The prompt is the same 1,500 tokens on both sides, the traffic is the same 2 million daily requests, the multiplication is the same multiplication, and exactly two quantities moved: the price per token and the minimum. Everything a cost dashboard watches got better. The price fell, the token counts did not move, and the request count did not move. What changed is the multiplier sitting between the prefix's length and what it bills, and that multiplier is reported in the usage block rather than on the price list.

The migration was not a bad idea, it was an idea evaluated with one term missing. On the mid-tier model with the segment qualifying, the prefix billed at a tenth of three dollars per million. On the cheap model with the segment qualifying it would have billed at a tenth of eighty cents, and the change would have been the win it was sold as. What it bought instead was the whole of eighty cents, because the second term — whether the segment still clears the minimum — moved at the same time and in the opposite direction. Ranking two models on price alone compares one term of a two-term product.

Confirming from the response instead of from the config#

The counters in the usage block are the only honest evidence, and the diagnostic value is in the combination rather than in any single field. Send the same request twice and read the second response.

def probe(build_request, label):
    """Send the same request twice; the second call is the one that tells you."""
    for _ in range(2):
        u = client.messages.create(**build_request()).usage
    print(f"{label:22} write={u.cache_creation_input_tokens:>6} "
          f"read={u.cache_read_input_tokens:>6} fresh={u.input_tokens:>6}")

probe(router_900,  "900-token block")     # write=     0  read=     0  fresh=  1140
probe(router_1400, "1,400-token block")   # write=     0  read=  1400  fresh=   240

Read those two printed lines against the prompts that produced them. The 900-token block reports 1,140 tokens of ordinary input, which is that block plus the 240 tokens of ticket appended after it — the whole prompt, billed as though no marker had been sent, because in effect none was. The 1,400-token block reports 1,400 tokens read from cache and 240 of ordinary input, so the fresh counter has collapsed to the part of the prompt that genuinely differs per request. That subtraction is the cheapest check available. The fresh count on a working request should be about the size of the suffix you know varies, and a fresh count the size of the entire prompt is a segment that never qualified.

Each row of that table is a deduction rather than a convention, and the deductions are short. Zeros in both cache counters mean the provider had no entry to write and none to read, which happens only when no entry was created at all. A large write count on every request means an entry was created every time, which means every lookup missed, which means the bytes differed from one request to the next. A large read with a small fresh count is the mechanism working. Large on both at once requires two boundaries rather than one, with the earlier segment matching and the later one not, which locates the instability between them.

The first row and the second row are frequently confused in a hurry, and they call for opposite fixes. Zeros everywhere is a qualifying problem: the segment is too short, the marker is missing, or the request shape disqualifies it. A large write counter on every request is a stability problem: the segment qualifies fine and its content is different every time, which is chapter 8's territory. Diagnosing the first as the second sends a team hunting for a volatile field in a block that has not changed in months.

Alongside the token count, providers attach conditions on the request's shape that are easier to trip than the length. Content types other than plain text may or may not count toward the segment; enabling certain features can move where the cacheable region is allowed to end; and some parameters, when changed, disqualify a match that the token count alone would have permitted. These are short lists and they are specific to each provider, so the useful habit is reading the caching page in full once and writing the conditions into the assembler as assertions rather than as comments. The assertion that pays for itself immediately is the length check: count the tokens in the segment you are about to mark, compare against the threshold for the model actually configured, and fail the request in development rather than shipping a marker that will be ignored in production. That check catches the model swap above as well, because the threshold it compares against comes from the model name rather than from a constant somebody typed once.

Widening a segment without padding it#

The obvious fix for a 900-token block is to make it 1,100 tokens, and the obvious way to do that is wrong. Padding with filler — repeated instructions, a block of whitespace, a comment banner — buys the discount by paying for 200 tokens of nothing on every request, and it advertises to the next reader that the prompt contains material nobody believes in. The saving is real and so is the cost: 1,100 tokens read at 0.1× is 110 effective tokens against 900 uncached, so it does come out ahead, and it also permanently attaches 200 junk tokens to every call and to every evaluation run against it.

The running cost of the filler is not really the objection, because once the entry exists those 200 tokens bill at a tenth like everything else in the segment. The objection is that it has to be maintained without ever being understood. The day the model string changes to one with a higher minimum, the padding has to grow, and whoever grows it has no way to tell which parts of the block are load-bearing and which are ballast, because the ballast was written to look like prompt. The same block is equally exposed from the other direction: a later reader tidying up what looks like a duplicated instruction, or a harness varying the prompt to see what the wording is worth, moves the segment back under the line and gets no error for doing it. Padding is not being ruled out here because it fails to work — it works — but because the qualifying length it buys is available from tokens that have a job.

The better move is to widen the segment with content that was going to be sent anyway. Tool schemas are the usual candidate, since they are stable, they are already in the request, and on several providers they serialize ahead of the system blocks, so a boundary after the instructions covers them for free. Few-shot examples — worked pairs of an input and the answer you want for it, included in the prompt so the model can see the job rather than only read about it — are the other common source. A router with nine categories and no examples is usually a router that would classify better with two examples per category, and 900 tokens of examples both improve the output and carry the prefix over the threshold.

A minimum on the segment, not on the prompt#

The threshold applies to the qualifying length measured from the start of the prompt to the boundary you declared, which is not the same as the size of the request. A request carrying 40,000 tokens can still cache nothing if the marker sits 600 tokens in, because the 39,400 tokens after it are outside the segment by construction. Long prompt, short segment, zero cached tokens is a consistent and frequently confusing outcome, and it is the shape most often misread as a provider fault.

The reason the measurement runs from position zero, instead of covering the block you actually marked, is the matching rule this topic opened with: an entry is found by hashing a request's leading bytes and walking forward from the start, so every entry is a prefix by construction. A run of tokens beginning 600 tokens in is not a short entry, it is not an entry at all, because no key could find it — any such key would have to describe the bytes in front of it, and those bytes are precisely what distinguishes one request from another. The marker names where a reusable prefix ends. There is nothing in the mechanism for naming where one begins.

Clearing the bar is also not a guarantee of a hit. A qualifying segment creates an entry that can be read; whether anything reads it depends on another request arriving with the same bytes before the entry expires or is evicted, which is a property of your traffic rather than of your prompt. An endpoint that sends a qualifying 30,000-token prefix twice a day will report writes and no reads indefinitely, and it is behaving exactly as designed.

The minimum is therefore best treated as a gate with a number attached to the model rather than to the prompt, checked in code at the point the marker is placed, and rechecked whenever the model string changes. Below it, everything else in this topic is inert: the ordering is correct, the marker is present, the review passed, and the counters read zero. Above it, the mechanism works and the remaining questions are about how often the entry is read.

Which is the question the arithmetic has been circling since chapter 1. A qualifying, well-ordered, stable prefix costs more to cache than to send when it is sent once, and far less when it is sent often. The break-even between those two is a specific number, it is computable in one line, and it decides which endpoints in a fleet deserve the effort at all.