Layered cache breakpoints

Under the explicit-breakpoint regime, a marker on a content block declares that everything from the start of the prompt up to and including that block is a cacheable segment. Providers allow several markers, commonly four, and the segments they create are nested: each is a prefix of the next, and a request reads the longest one whose bytes it reproduces exactly. Where those markers go is a separate decision from the block ordering of chapter 2 and it is made after it, on a prompt whose blocks are already sorted from most widely shared to least.

The instinct people bring to that decision is to maximise the number of tokens inside the segment, which is the wrong objective and produces a specific, expensive failure. What a boundary actually selects is a readership: the set of requests capable of producing the exact bytes in front of it. This chapter works through what that costs when the readership is one person, what the nested arrangement looks like in code, the traffic assumptions it rests on, and what to do when a prompt has more seams than the provider allows markers.

An entry's readership is the set of requests that agree up to the boundary#

That readership is a consequence of what a provider actually stores, and the derivation takes three steps. What an entry holds is prefill state: the internal working state the model builds while reading a prompt, before it emits anything. Every position's state was computed with all the tokens in front of it present, and there is no version of that computation in which it was not. A stored state is therefore valid for a later request only if that request presents the same tokens, in the same order, up to the same point.

The consequence of that does all the work in this chapter. Take two requests that agree for their first 3,000 tokens, differ on one token immediately after, and then agree again for a further 9,000. None of that second agreeing stretch is reusable, because the state at each of its positions was computed against a token the other request does not have. Matching is a forward walk from position zero that ends at the first byte that differs, and everything behind that byte is unreachable however identical it looks. That is also why the ordering decision of chapter 2 — sorting blocks from most widely shared to least — is made first: ordering decides how far the walk is able to reach, and placement decides where it is allowed to stop.

Whoever can produce the bytes from position zero to the marker can read the entry, and nobody else can. Consider an assistant whose prompt is a 12,000-token shared policy, then a 400-token block naming the current user and their account tier, then the conversation. Putting the boundary after the user block means the key includes that user's name, so the readership of the resulting entry is that one user — and the 12,000 tokens of shared policy sitting in front, identical across the entire customer base, are written into a separate entry for every account.

The last column of that table is not a measurement. Each figure in it is the number of distinct byte sequences that reach the boundary in question, which is what decides how many entries have to be filled. A boundary after the shared policy has one such sequence, so the only event that forces a refill is the entry expiring. A boundary after the user block has one per user, and a boundary after the conversation has one per conversation that moves forward every turn.

At 5,000 daily active users and twenty turns each, the extra 400 tokens captured by the second row cost 4,999 additional write premiums to obtain. At 1.25× on 12,400 tokens those writes are 77.5 million effective tokens a day against 4.4 million for the shared arrangement, and the 400 tokens they bought back are worth 400 × 0.1 × 100,000 requests = 4 million. The trade is roughly seventy million effective tokens a day, about $210 at $3 per million, paid for a segment 3% larger.

Two of those terms are worth taking apart, because they are the ones the conclusion rests on. The 77.5 million is 5,000 entries a day, each covering 12,400 tokens, each billed at the 1.25 write multiplier: 5,000 × 12,400 is 62 million tokens of content, and 62 million at 1.25 is 77.5 million effective ones. The 100,000 requests in the other term is those same 5,000 users at twenty turns each, and the 0.1 is the read multiplier that tokens inside a matched segment bill at. The shared arrangement, by contrast, writes 288 entries of 12,000 tokens at the same 1.25, which is a little over four million effective tokens. The tokens are not cheaper there. There are fewer than a twentieth as many entries to fill.

The second row of the table is not reporting a broken cache, and that is what makes the mistake durable enough to survive a review. Each user writes one entry on their first turn and reads it on the nineteen that follow, so the endpoint reports a hit on nineteen requests in twenty and any dashboard showing hit rate looks healthy. A hit rate answers the question "what fraction of requests found an entry". It says nothing about how many entries had to be created for them to find, and the write premium is charged once per entry created rather than once per request served. What separates that row from the one above it is the last column, and no hit rate displays it.

The version that hurts most is the one-shot case. If those 5,000 users send one request each rather than twenty, the per-user entries are written and never read at all, so caching costs 5,000 × 12,400 × 1.25 = 77.5 million effective tokens against 62 million with no caching whatsoever. The endpoint is then paying a 25% surcharge for a feature that, correctly placed, would have cut its input bill by 88%.

Both halves of that sentence are checkable. The 25% is the write multiplier and nothing else: with no reads at all, every request pays 1.25 where it used to pay 1, and 77.5 million against 62 million is exactly that ratio. The other half is what the same traffic does with the marker one block earlier. Each request reads 12,000 tokens at 0.1 and sends its own 400 at full rate, so a tenth of 12,000 is twelve hundred effective tokens and adding the 400 brings the request to sixteen hundred — roughly an eighth of the 12,400 it billed uncached. The feature was capable of removing about seven eighths of the input bill and was configured to add a quarter to it, and the whole of the difference is which block the marker sits on.

Three boundaries, three readerships#

The same mistake reappears one level up in a form that sounds more sophisticated. A multi-tenant agent builds each request from 2,000 tokens of tool definitions, a 9,000-token shared system policy, a 3,000-token tenant playbook, a running conversation of anywhere from 2,000 to 40,000 tokens, and the new user message. The instinct is to put a single breakpoint immediately before the new message, because that is the largest cacheable span available.

It is, and it makes every entry specific to one conversation. The 11,000 tokens of tools and shared policy — identical for every tenant and every conversation in the system — are then written once per conversation instead of once per TTL window. With 20,000 conversations a day that is 20,000 × 11,000 × 1.25 ≈ 275 million effective tokens of write premium, against about 4 million if those tokens live in their own shared segment. The difference is on the order of $800 a day, and the single-boundary design also has the property that any change anywhere in the span invalidates all of it, so a one-token edit to the tenant playbook discards the tools and the policy along with it.

Every step of that comparison is short arithmetic on the figures already given. Twenty thousand conversations carrying 11,000 tokens each is two hundred and twenty million tokens written in a day, and applying the 1.25 write multiplier to them gives the 275 million. The shared alternative writes those same 11,000 tokens 288 times rather than 20,000, which at 1.25 comes to a little under four million. The difference is a bit over two hundred and seventy million effective tokens a day, and at $3 per million that is the $800.

Why a second and a third marker recover most of that is worth deriving rather than asserting. Matching is a walk from position zero, but the walk cannot stop wherever it likes and claim a saving. A provider keeps entries, not a checkpoint at every token position, and an entry exists at exactly the positions where a marker declared one — the marker is the instruction to store the state as it stands here, under a key that is the hash of every byte up to here. The places the walk can stop and still find something to read are therefore the marked positions, and nothing else.

With one marker that set has one member, which is what makes the single-boundary design all-or-nothing. A request from a second tenant agrees with the stored entry for 11,000 tokens and then diverges inside the playbook, before the only stopping place there is, so it collects none of the 11,000 tokens it genuinely agreed on and writes the whole span again. Adding a marker after the shared policy puts a stopping place in front of that divergence. The same request now walks as far as BP1, matches, reads 11,000 tokens and writes only what lies beyond it — an all-or-nothing miss has become a partial hit, and the partial is most of the prompt.

Three boundaries produce three nested entries with three different readerships, and each request reads the longest one it matches. A new conversation for an existing tenant hits BP2 and pays a write only for its own turns. A brand-new tenant hits BP1 and pays a write for its 3,000-token playbook. A request arriving after everything expired pays one write for the whole span, which is what the single-boundary design pays every time.

def build(tenant, conversation, new_message):
    return dict(
        tools=TOOL_SCHEMAS,                                     # 2,000, no marker needed
        system=[
            {"type": "text", "text": SHARED_POLICY,             # 9,000
             "cache_control": {"type": "ephemeral"}},           # BP1 → 11,000 shared
            {"type": "text", "text": tenant.playbook,           # 3,000
             "cache_control": {"type": "ephemeral"}},           # BP2 → 14,000 per tenant
        ],
        messages=[
            *conversation[:-1],
            {**conversation[-1],
             "cache_control": {"type": "ephemeral"}},           # BP3 → rolling, per conversation
            {"role": "user", "content": new_message},
        ],
    )

The rolling marker on the last completed turn is what keeps a long conversation cheap. On turn N the entry written at turn N − 1 is a strict prefix of the current request, so the provider serves everything up to it as a read and writes only the delta — one turn's worth of tokens rather than the whole history. A twenty-turn conversation that reaches 40,000 tokens costs roughly 40,000 × 1.25 spread across twenty small writes plus 0.1× on the accumulated history, instead of 40,000 tokens prefilled at full rate on every turn near the end.

The property that makes the rolling marker work is that a conversation is append-only, and it is worth naming because teams break it on purpose. Turn N is turn N − 1 with more text on the end, which is exactly the shape the forward walk rewards. Two common changes destroy it. Summarising older turns to keep the context small rewrites the front of the message list, and dropping the oldest turns to fit a window shifts every surviving byte forward. Either one makes turn N differ from turn N − 1 near the beginning rather than at the end, which sends the walk back to BP2 and converts every turn into a full write of the conversation.

The three entries overlap rather than sitting side by side, and that follows from the key rather than from a design decision anyone made. Every marker's key is a hash of everything from position zero, so BP1's segment is contained inside BP2's and BP2's inside BP3's. No arrangement of markers can produce two segments neither of which contains the other, because every segment starts in the same place and only the end moves.

Adding boundaries is close to free, which surprises people who expect each one to be billed. The segments are nested rather than independent, so a request that fills the whole prefix pays 1.25× once on the span whatever number of markers sit inside it; what the extra markers buy is more candidate prefixes for later requests to match against. The scarce resource is the limit itself — commonly four — and the discipline is to spend those four on boundaries that separate genuinely different readerships rather than on arbitrary points in a block that always changes together.

Reading a layered prompt off the counters#

The two counters in the usage block report different things, and holding them apart is what makes a layered prompt readable at all. The read counter counts tokens the provider served out of an entry that already existed. The write counter counts tokens it stored into an entry that did not. A prompt with a single marker in a fixed position produces one or the other and never both: either the segment matched in full, in which case there was nothing new to store at that boundary, or it did not, in which case there was nothing to read.

Verifying that a layering does what its diagram claims takes one look at the usage block, because a layered prompt has a signature a flat one does not: reads and writes are both non-zero on the same response, and the read total sits at a boundary you can name. A request from an existing tenant starting a new conversation should report a read of about 14,000 and a write of whatever its first turn adds. A request from a new tenant should report a read of about 11,000 and a write of about 3,000. If the read figure comes back at 11,000 for requests from established tenants, BP2 is not being matched and the playbook is varying between requests. If it comes back at zero across the board while writes sit at the full span, the markers are landing on blocks that change every time. Sampling one request per tenant per hour and asserting the expected read boundary is a cheap standing check, and it catches a broken layer weeks before a monthly invoice does.

The traffic assumptions behind a layering#

The three-boundary layout above is right for a specific traffic shape, and it is worth being able to say which. It assumes there are far fewer tenants than conversations, so a per-tenant entry is read many times before it expires. It assumes tenant playbooks change on a human editing cadence rather than per request. It assumes conversation turns arrive within the TTL of each other, which is usually true of a person typing and not true of an agent that waits on a slow tool call. And it assumes the shared policy and tool definitions are byte-identical across tenants, which is a claim about the assembler rather than about the content.

Each of those four is a claim you can check against something rather than a premise you have to accept. The first is a ratio between two counts already in your database. The second is an editing cadence visible in the update timestamps on the playbook table. The third is an arrival-gap distribution visible in request logs, and it is the assumption most often wrong in a system with agents in it, because an agent's turns are separated by tool calls rather than by typing. The fourth is the only one that is a claim about your own code: two tenants whose assembled policies differ by a trailing newline are two populations, and nothing about the content of those policies says so.

Change any of those and the plan changes with it. Ten thousand tenants each sending four requests a day means a per-tenant entry is written, read three times, and expires — still positive by the chapter 5 arithmetic, but marginal enough that the marker could go to a better use. A tenant playbook regenerated from the database on every request is not a cacheable layer at all, and marking it as one converts BP2 into a per-request write. If conversations routinely pause for twenty minutes between turns, the rolling marker writes far more than it reads and the honest design keeps BP1 and BP2 and drops BP3.

The ten-thousand-tenant reading carries a condition, and without it the case is worse than marginal rather than better. It assumes those four requests arrive close enough together to fall inside one entry's lifetime. Spread evenly across a working day they do not, and the entry is instead written and left to expire unread, four separate times over — a marker that has stopped being marginal and started being negative. What decides a per-tenant layer is the gaps between a tenant's requests, not how many requests they send.

There is also a placement error that costs nothing dramatic and wastes a marker: a boundary a few hundred tokens after the previous one. Two markers separated by 200 tokens create a second candidate prefix that differs from the first by 200 tokens, which is worth 20 effective tokens per hit and consumes a quarter of the budget. Boundaries earn their place at the seams between readerships, and if two adjacent blocks are always read by the same set of requests there is no seam there to mark.

Spending four markers on five seams#

Real prompts often have more seams than the budget allows. An agent might have tool definitions shared across the fleet, a shared policy, a regional legal addendum, a tenant playbook, a per-user profile, and a conversation — six blocks and five seams for four markers. The one to drop is the seam with the smallest product of tokens behind it and requests that stop there. A regional addendum of 300 tokens sitting between two layers that are both heavily reused is a seam worth merging into its neighbour, because collapsing it costs 300 tokens of extra write on whichever side it joins and frees a marker for the conversation, where the rolling boundary is worth thousands of tokens per turn.

The other resolution is to remove the seam rather than the marker. If the regional addendum has only three variants, appending it to the shared policy produces three shared entries instead of one, each still read by a third of the fleet — three writes per window rather than one, in exchange for a marker. That trade is usually good when the variant count is small and terrible when it is large, and the variant count is a fact about your data that you can look up.

The product that decides which seam to drop has two factors, and the second is the one people leave out. The first is how many tokens sit behind the seam and would otherwise be written again — what a marker there rescues on a request that stops at it. The second is how many requests a day actually stop there, meaning requests that agree with everything in front of the seam and disagree with something after it. A seam nothing ever stops at rescues nothing however large it is, and that is the ordinary reason a marker turns out to be wasted: it sits between two blocks that always change together, so no request has ever diverged between them.

Placement cannot rescue a prefix whose blocks are ordered wrongly, which is why ordering is the first decision and layering the second. A boundary after a tenant playbook that sits ahead of the shared policy still produces a per-tenant entry containing everything, because the shared policy is behind the tenant-specific bytes and unreachable from another tenant's request wherever the marker goes.

Nor does a layer become independent of the ones in front of it. The entries are nested prefixes, so an edit to the shared policy invalidates the tenant segment and the conversation segment too, all at once, across every tenant. That is the correct behaviour, and it means the blast radius of a change — how much of the cached prefix a single edit destroys — is everything downstream of it, which in a four-layer prompt is most of the prompt. A deploy that edits the shared policy costs one full refill per TTL window and then recovers, which is fine and expected.

What is not fine is the same invalidation arriving without a deploy, on every request, because something inside a supposedly stable block is quietly different each time. A layered prompt is more exposed to that than a flat one, since one volatile byte near the front takes three entries with it instead of one.

The question to put to each marker, then, is not how many tokens it captures. It is which requests can produce the bytes in front of it, because that set is the entry's whole readership, and the write premium is paid once by every member of it that has to fill a copy of its own. Ordering gives the forward walk somewhere to reach; markers at the seams between readerships give it somewhere to stop that still counts. A prompt with both arranged correctly is one where a new tenant, a new conversation and a returning user each pay to write only the part of the prefix that is theirs alone, and nobody pays twice for the part that belongs to everybody.