Tool definitions in the cached prefix

The forty tools a support agent exposes are not code a provider runs and not anything registered with it in advance. Each one is a small JSON document that travels in the request alongside the prompt: a name, a description written to be read by the model, and an input_schema listing the parameters and their types. Together they are the tool definitions, the block that tells a model which of your functions it may ask for and what arguments each one takes. The model executes nothing itself. It reads that block, decides which entry fits the question in front of it, and emits a structured request naming the tool and its arguments; your service runs the real function and hands the result back as another message. Tool definitions exist because a model has no access to your systems and no way to discover them, so the only account of your API it will ever have is the one you put in the request. You meet them as the tools array in an API call, however it gets built — hand-written dictionaries, decorators over Python functions, a registry serialised at start-up, or a Model Context Protocol (MCP) server handing you the same structures over a protocol.

Because the API is stateless and nothing is retained between calls, that array is re-sent in full every time, and being re-sent every time is what puts it on a collision course with caching. Tool versioning — changing a deployed tool's shape, whether that is a parameter name, an enum value or the wording of a description, on a service already serving traffic — is therefore an edit to the part of the request that everything else is cached behind, made while conversations written against the old shape are still open. Topic 2.15, on prompt caching mechanics, established how a provider reuses the work it already did on the front of an earlier request and what that reuse is worth. This topic is about what it costs when the front of the request is the thing you need to change.

The comparison is over bytes, and there is no metadata#

A provider serving a cached prefix does one thing when a request arrives: it walks the serialised request from the first token forward, matching it against stored entries, and stops at the first position where the bytes differ. Everything before that position can be read from cache. Everything after it has to be prefilled. That forward walk has no resumption, no diffing and no alignment, and the consequence for tool versioning is the part that surprises people, because the walk does not know what a description is.

The absence of resumption is not a design choice a provider could revisit, and seeing why makes every figure in this topic follow rather than have to be memorised. What is stored is not the text of the prefix. It is the intermediate state the model computed while reading that text, position by position, and each position's state was computed with every earlier token visible to it. Change a token at some position and the stored state for that position is state for a different sequence, and so is the state for every position after it, because each of those also read the changed token on its way past. There is nothing to align to and nothing to patch: the saved work behind the divergence was manufactured out of the bytes in front of it. The halt is therefore forced by how the reusable work was made, not by a policy anybody wrote down, and what the provider is comparing is a flat byte sequence with no structure in it.

There is no field in the request that the matcher treats as annotation. The tools array is serialised into the same byte stream as the system prompt and the messages, and once it is serialised, a description string is exactly as load-bearing to the comparison as a parameter name, an enum value, a required-fields list, or the two spaces of indentation your JSON encoder chose to emit. The distinction between "part of the executable contract" and "helpful prose for the model" is real, and it lives entirely in your head and in the model's reading of the prompt. It has no representative in the cache key.

That is worth stating plainly because the opposite belief is both common and reasonable. Every other versioned interface a backend engineer works with does treat some of its content as metadata. An OpenAPI description field does not change the wire format. A docstring does not change a function signature. A comment in a Terraform file does not change the plan. Tool definitions look like all three of those things, and the habit transfers, and it is wrong here for a mechanical reason rather than a philosophical one: the model reads the description, so the description is in the prompt, so the description is in the prefix that gets matched.

Where the 88,800 comes from#

That agent's request is assembled in the order the SDK sends it — tools first, then the system prompt, then the messages — and the cacheable prefix runs to about 90,000 tokens before anything per-conversation begins. Laying it out with cumulative offsets turns a one-character edit into arithmetic.

Two properties of that table are worth checking rather than reading past, because they are what make it an instrument instead of an illustration. The Tokens column has no leftover row: twelve hundred, plus three hundred and ten, plus four thousand eight hundred and ninety, plus two thousand three hundred, plus eighty-one thousand three hundred, comes to exactly the ninety thousand the prefix was said to run to. Nothing is unaccounted for, which is what licenses the second property — that each row's start offset is the row above it plus that row's length, so a position in the table is a position in the byte stream rather than an estimate of one. The moment a residual bucket appears in a map like this, the offsets below it are guesses and the arithmetic that follows is not worth doing.

The arithmetic itself is subtraction, and the reason it is subtraction rather than anything more interesting is the forward walk. Everything from the first differing byte to the end of the prefix stops being readable, so the invalidated region is the whole prefix minus the part that still matches in front of it, and the part that still matches is exactly the cumulative offset at which the edited segment starts. Only one term in that subtraction is a choice anybody made. The edited tool's own length is the portion of the loss that somebody asked for; take it out of the total and what remains is the collateral, which on this layout is very nearly all of it.

The match now ends at token 1,200, so 90,000 − 1,200 = 88,800 tokens stop being readable on every request until the caches rewarm. Of those, 88,490 are bytes nobody edited. The 81,300-token carrier policy document is the expensive part and it is also the part with the least to do with the change: it is identical across every customer, every conversation and every request in the fleet, which is precisely what made it worth caching, and it sits behind a tool description that a typo fix moved.

The 1,200 tokens in front of the edit are not automatically salvage, either. Providers only serve an entry once the matched region clears a minimum cacheable length — the floor that topic 2.15, on prompt caching mechanics, puts in the low thousands of tokens and varying by model family — and a first tool short enough to leave the surviving prefix under that floor means the request is written fresh in its entirety rather than partially read. Two agents can therefore take the same edit to the same position and get different bills, because one of them still has a usable prefix in front of the divergence and the other has a fragment.

At roughly 50 requests per minute and a cache lifetime in the five-minute range, the recovery is not instant, because the entry has to be refilled independently wherever requests land. Each refill is a request that pays the write rate on 88,800 tokens instead of the read rate, and each one also pays the latency of prefilling them: at a prefill throughput on the order of 25,000 tokens per second, 88,800 tokens is about 3.5 seconds of work that had previously been a cache read. The window closes after about twelve minutes, because that is how long it takes for every serving node handling the endpoint's traffic to see a request and fill its own copy.

The belief that gets in the way here is that the first request after a deploy pays the write and every request after it reads. That is exactly what happens on one machine, which is why it is the natural thing to assume, and it is why teams expect the recovery to be a single expensive request rather than a window. What makes it false is that an entry is written where the work was done. A request that lands on one serving node fills that node's copy and leaves every other node holding nothing, so the count of requests that pay the write rate is not one but roughly the number of places the traffic can land, and the duration is however long it takes for traffic to reach all of them. That also explains why the window outlasts a single cache lifetime: entries filled early can expire before the last node has been reached at all.

The same edit, one position earlier#

Position is the entire lever, and it is worth seeing what the identical logical change costs at a different offset. Consider a second agent whose tool sits second in an array of thirty, ahead of a 40,000-token retrieved-context block that is byte-identical for every user in the system.

{
  "tools": [
    { "name": "list_accounts",  "description": "...", "input_schema": { } },
    { "name": "update_account", "description": "...", "input_schema": { } },
    { "name": "...28 more..." }
  ],
  "system": [
    { "type": "text", "text": "You are a billing assistant..." },
    { "type": "text", "text": "<40,000 tokens of retrieved context>",
      "cache_control": { "type": "ephemeral" } }
  ],
  "messages": [ ]
}

Editing the description of update_account invalidates from the middle of the second tool onward: the remaining twenty-eight tool definitions, both system blocks, and the 40,000-token retrieved-context block, all the way to the breakpoint. The retrieved-context block is the one that hurts, since it is shared by the entire user population and therefore had the highest reuse count of anything in the request. A change that touches a tool nobody thinks of as important costs exactly as much as a change to the block everyone agrees is expensive, because cost here is a function of what sits behind the edit rather than of what the edit is.

The cache_control marker in that request is a reminder of something worth keeping straight, since it is easy to read a breakpoint as a boundary that contains damage. It does not. A breakpoint declares where a reusable region ends and therefore what the provider is willing to store; it says nothing about where a match is allowed to begin, and a match always begins at token zero. Placing a marker after the retrieved-context block means the provider will store a 40,000-plus-token entry, and a divergence at token 900 makes that entry unreachable in full. Under a provider that caches automatically without markers the picture is identical, because the automatic behaviour is a prefix match too — the marker changes what gets written, never what gets compared.

There is a corollary that turns the whole problem into something you can manage. If the invalidation always runs from the changed byte to the end of the prefix, then two facts decide the price of any tool edit: how far into the prefix the tool sits, and how much stable, widely-shared content sits behind it. Neither of those is a property of the edit. Both are properties of how the request is laid out, which means they are things you control at assembly time rather than at edit time.

Cost and latency with no failure signal#

The agent kept working throughout. Every one of the 600-odd requests that landed in the cold window got a correct response, called the right tools, and returned in a time that no user would have flagged as anomalous — 3.9 seconds to first token is slower, but it is not an error, and nothing in the application's error rate moved. That is the characteristic that makes cache invalidation from tool edits hard to catch: it produces a cost and a latency event with no failure signal attached, and cost and latency both average away at the granularity most dashboards use.

Both figures in that window can be recovered from the layout, which is the check worth doing on any incident of this shape. The count of affected requests is the arrival rate multiplied by the duration, and fifty a minute for the twelve minutes the window ran gives the six hundred. The hit rate is the other one, and it is easy to misread because it did not go to nothing. It is defined here as cached read tokens over total prefix tokens, so it is a ratio of tokens rather than of requests, and a request that diverges partway through still contributes whatever matched in front of the divergence to the numerator. A floor rather than a zero is the expected reading, and it is set by two things at once: the surviving front of the prefix on every request, and the requests still landing on nodes that had not yet been reached by the new bytes.

It is also worth separating this from two neighbouring things it gets confused with. The first is ordinary expiry. Cache entries have a lifetime and they die on their own schedule regardless of whether you deploy, so a hit rate that sags during a traffic trough is the entry timing out between requests, and no amount of schema discipline touches it. The second is a functional break. Changing a tool's description can absolutely change what the model does — that is the subject of topic 5.11, on tool descriptions as prompt tokens the model conditions on — and that is a separate axis from the cache. An edit can be behaviourally significant and cache-expensive, behaviourally significant and cache-free, or neither, and conflating the two questions is how teams end up refusing to improve a bad description because they are afraid of the cache, or shipping a rewrite that changes tool selection because they only asked the cache question.

The third confusion is the one that produces the four-minute review. "I only changed a description, not the schema" treats the request as if it has a typed contract portion and a comment portion, and reasons that the cache key is computed over the former. Nothing in the request is metadata. The whole serialised prefix is what gets compared, descriptions included, which is why a one-tool edit invalidates from that tool onward and why bundling a tidy-up into the same pull request quietly converts a tail invalidation into a total one.

The cost of a tool change, then, is decided by where the change lands in the serialised request and by what sits behind it, and neither of those numbers is visible in the diff a reviewer reads. Almost nobody knows, for their own service, what the request actually looks like when it goes out on the wire or in what order the pieces are assembled — the assembler is read as a dictionary of named parts rather than as a byte sequence, and the two are related only by convention.

Finding out is mechanical. Capture one real request body, label the segments in the order they appear, and the price of every future schema edit becomes a lookup rather than an argument, which is the first thing worth doing before touching a tool definition at all.