An agent loop usually invalidates its own cacheable prefix

Every request an agent loop sends contains a long run of tokens the previous request also contained: the system prompt, the tool definitions, and the whole transcript up to the newest turn. Prompt caching exists to stop you paying full price to have the provider process those tokens again, and topic 2.15 covered the mechanism.

What that mechanism costs to keep working, in a loop specifically, is the subject here, and the answer is unintuitive enough to have caught most teams that have tried it. The cache is not a lookup on "have I seen these tokens before". It is a match on a prefix, forward from the first token, and an agent loop performs several operations on its message list per turn that each move the point where two consecutive requests stop agreeing. Nearly all of those operations were added for good reasons by somebody who was not thinking about caching, which is why the failure is common and why it is invisible in code review.

The prefix is a byte comparison from token zero#

The only property that matters here is the one an agent loop routinely violates: the cache is keyed on an exact prefix, matched forward from the very beginning of the prompt. A single differing token at position 4 invalidates everything from position 4 onward, no matter that positions 5 through 5,300 are identical to yesterday's. There is no partial credit and no reordering.

That is stricter than a cache of text usually is, and the reason is worth holding because it makes the rest predictable. What the provider stores is not the tokens but the processed state the model built while reading them, and the state after any token depends on every token before it. Change one token near the front and every state after it is a state the provider has never computed, so there is nothing to reuse. The later tokens have not changed; what they produce has. Reordering fails for the same reason: two sequences with the same tokens in a different order build different states from the first difference onward.

An agent loop appends. That is the one access pattern this design rewards, because a message list that only grows at the end shares an ever-longer prefix with its own previous version. Almost everything else a loop does to be helpful works against it. The list below is what has actually broken prefixes in production agents, and the common shape is that each of them was introduced for a good reason having nothing to do with cost.

Read that table by the middle column and the third column stops needing to be memorised, because it follows from the position. Reuse survives up to the mutation point and dies after it, so a mutation at position 0 leaves nothing and a mutation at the end leaves everything. The rows are sorted, in effect, by how early the damage is done.

The last two rows are the interesting pair, because they are the same operation applied at different positions. Compaction, which drops or rewrites older turns so that a long-running loop's transcript stops growing, is a standard response to that growth and it is not wrong, but performing it every turn means the mutation point moves every turn, and the cache is effectively off from the mutation point to the end. Compacting on a threshold — once, when the transcript crosses 40,000 tokens, and then not again for several turns — costs the same tokens and keeps the prefix stable in between.

The dynamic tool list deserves its own mention because it is usually sold as an optimisation. Trimming the tool block from eleven definitions to the four a step plausibly needs saves 2,100 input tokens on that request at $3 per million, which is $0.0063 per run. It also means that steps with different tool sets share no prefix, which on this agent cost more than four times that. The two effects are not the same shape: the saving is a fixed quantity of tokens on one request, while the loss is proportional to how much reuse was available across the run. Whether the trade is worth it depends on how many distinct tool sets your steps produce; with two sets you keep most of the reuse, with seven you have none.

Where to put the breakpoints in a transcript that grows#

Anthropic's API takes up to four cache_control breakpoints per request, and each marks the end of a cacheable segment; the system checks for a hit at each breakpoint working backwards, so a request can hit on an earlier one when the later segment has changed. In a tool-calling loop that gives you a useful two-tier arrangement. One breakpoint sits at the end of the static material — tools and system prompt — and never moves. A second one rolls forward along the transcript, placed on the last message of the previous turn, so each request reads back everything the previous request wrote.

def build_request(state: RunState) -> dict:
    tools = STATIC_TOOLS                      # same order, every request
    tools[-1]["cache_control"] = {"type": "ephemeral"}   # breakpoint 1

    messages = [m.copy() for m in state.messages]        # append-only history
    if len(messages) >= 3:
        # breakpoint 2: last message of the previous turn
        messages[-3]["content"][-1]["cache_control"] = {"type": "ephemeral"}

    return {
        "model": state.model,
        "system": [{"type": "text", "text": SYSTEM_PROMPT}],
        "tools": tools,
        "messages": messages,
        "max_tokens": 1024,
    }

The comment on the first line is the load-bearing part of that function. A breakpoint placed at the end of a tool block only helps if the block serialises identically each time, so the tools have to come from something with a defined order rather than from a dictionary whose iteration order depends on registration.

A run summary that was previously prepended still has a home under this arrangement; it moves from the front of the message list to the end, appended as a user-role block alongside the newest tool result. The model reads it just as well there, and everything before it stays byte-identical. That single relocation is typical: the repair for a broken prefix is almost never to remove the helpful thing, it is to move it behind the material you want to reuse. The platform documentation's Tool use with prompt caching page works through the placement rules for exactly this case if you need the edge conditions around images and thinking blocks.

The difference the arrangement makes on a seven-call run, measured on 500 replayed tickets, is large enough to be worth laying out. The run's input tokens grow from 6,200 on the first call to 22,600 on the last, totalling 98,600 across the run.

Those four costs are one calculation with three prices in it, and doing it once makes every row checkable. Each of a run's 98,600 input tokens is billed in exactly one of three ways: fresh at the base rate, written to the cache at 1.25 times the base rate, or read from the cache at a tenth of it. Total cost is therefore fresh tokens at full price, plus written tokens at a small premium, plus read tokens at a large discount, and the three counts always add to 98,600.

Take the third row. The static prefix is 5,300 tokens, written once on the first call and read on each of the 6 calls after it, which is where the 31,800 reads come from. Everything else in the run — the growing transcript — is still fresh, so most of the tokens are still at base rate, and the discount applies to a fixed 5,300 tokens per call rather than to a growing quantity. That is a 28% saving and it is bounded above by design.

Now the fourth row. With a breakpoint rolling along the transcript, each request reads back everything the previous request had already sent, so reads grow with the run instead of staying fixed, and the 76,000 read tokens are most of the run's input. What is left over is 22,600 tokens, which happens to be the size of the final call — and it is written rather than fresh, so the run has essentially no tokens billed at the base rate at all. The jump from 28% to 64% is the part most teams leave on the table.

The reason the second arrangement is worth so much more than the first is structural rather than incidental. Caching the system prompt and tools reuses a fixed quantity per call; caching the transcript reuses everything the run has accumulated, which grows with the run, so the saving is largest exactly on the long expensive runs that dominate the bill. It also cuts time to first token, which is where the 4.7 seconds from the previous chapter's breakdown came from.

Finding out whether it is working today#

The question is not whether caching is enabled, since enabling it is a configuration flag and hitting it is a property of what your loop did this morning. Every response carries the answer in usage, and one response is enough to read a verdict off:

{
  "usage": {
    "input_tokens": 8900,
    "cache_creation_input_tokens": 5300,
    "cache_read_input_tokens": 0,
    "output_tokens": 210
  }
}

A cache creation of 5,300 tokens with zero reads is the signature of a prefix that is being written and never matched, and on a request after the first it is conclusive: the segment was stored during an earlier call of the same run, so a miss means the bytes in front of it moved. The only work beyond reading one response is aggregating the same three fields by step index rather than by request, because the aggregate over all requests hides the shape.

def record_cache_usage(step_index: int, usage: dict) -> None:
    fresh = usage["input_tokens"]                      # billed at base rate
    written = usage["cache_creation_input_tokens"]     # billed at 1.25x
    read = usage["cache_read_input_tokens"]            # billed at 0.1x
    total = fresh + written + read
    metrics.histogram(
        "agent.cache.read_ratio", read / total, tags={"step": step_index}
    )
    metrics.counter("agent.cache.written_tokens", written, tags={"step": step_index})

Those three fields are the same three categories the cost calculation used, so the ratio the function records is a direct measurement of the split the bill is computed from. Tagging by step index rather than summing is what makes it diagnostic: reuse is supposed to change over a run, from none at the start to most of it later, and an aggregate flattens exactly the curve you need to see.

A healthy agent shows a read ratio near zero at step one and rising toward 0.8 or higher by step five, with written_tokens decaying to the size of one turn's delta. The broken shapes are recognisable at a glance. A flat zero ratio across all steps means something mutates the front on every request. A ratio that is healthy for the first three steps and collapses at step four points at a compaction threshold or a tool-filtering rule that fires mid-run. A ratio that is healthy in load tests and near zero in production usually means traffic is too sparse for the five-minute TTL — a request arriving eight minutes after the last one for that prefix finds nothing, and pays the write premium to repopulate a cache the next request will also miss.

The two breakpoints also have different sharing behaviour, which is easy to miss when reading the ratio. The static segment is shared across every run of the agent, because it is identical in every run, so ticket 41,882 warms it for ticket 41,883 and the first call of a run can and should hit; if step one shows a zero read ratio at steady traffic, the static segment is not stable across runs, and the usual culprit is a system prompt templated with the ticket id or the customer's name. The rolling segment is private to one run by construction, since no other run has that transcript, so it can only ever hit from step two onward. Seeing those two expectations separately is the difference between "caching is at 62%" and knowing which half of the arrangement is broken.

That last case is worth pricing before you chase it. With the shipping-exception agent's 1,270 runs per thousand tickets spread over a working day, prefix entries stay warm; a low-volume internal agent that runs eleven times a day would have been better off with no caching at all, since it would pay the 1.25× write premium on every single request and read nothing. Anthropic's one-hour TTL exists for that traffic shape, at a higher write multiplier, and whether it pays is arithmetic you can do from your own inter-arrival times.

What caching does not touch#

Output tokens are unaffected. On this agent output is 210 tokens per call against 14,000 input, so it barely registers, but on a workload that generates long documents the cacheable fraction of the bill is small and the whole exercise is close to a rounding error. The check is one division: cached-eligible input tokens over total billed tokens, before you spend a sprint on breakpoint placement.

It is also a different thing from the document cache in the previous chapter, and confusing the two leads to the wrong fix. Caching policy_retrieval in your own process removes the tool call entirely, saving 16.4 seconds of wall clock and the tokens the result would have added to the context. Prompt caching keeps the call, keeps the tokens in the context, and reduces what the provider charges to re-read them. The first is a dependency you control and can invalidate on a schedule; the second is a provider-side artefact with a TTL you do not control and no invalidation API. Systems generally want both, and they fail in different ways: a stale process cache serves last week's refund policy, while a cold prompt cache only costs money.

The asymmetry in those two failure modes is worth carrying, because it decides how much care each deserves. A cache that can serve a wrong answer needs an invalidation story, a maximum staleness and somebody who owns it. A cache that can only fail to save money needs a metric and nothing else, which is why the read ratio above is the whole of the operational burden here.

Nothing in this chapter changed what the model does with those tokens. A cached prefix is the same prompt read by the same model at a lower price, and that is what makes it the one lever in this topic that needs no evaluation before it ships. The run still spends 24.6 seconds and most of its output budget on seven calls to a frontier model, several of which are doing work that does not need one.

Choosing a different model per step is the larger lever, and it does not have the same safety property at all. Where caching can only fail to save money, tiering can change what the agent decides — which makes the interesting question not how much it saves but which steps can take it.