Tool versioning without invalidating the prompt cache

Read these first

• Tool design — descriptions
• Prompt caching mechanics

What you will know

• Change a tool without invalidating the whole prompt cache

  A single byte in the serialized schema can force misses for every later token, even when runtime behavior is backward-compatible.

• Trace a request's cacheable prefix and locate the serialized tool definitions in it

  Ordering, whitespace, descriptions, and schema serialization can change the prefix key even when the callable interface appears unchanged.

• Version a tool so old conversations still make sense

  Use additive fields, aliases, or adapters and retain the old schema until stored transcripts and retries no longer reference it.

• Measure the cache impact of a tool change

  Compare hit rate, prefix-token spend, latency, and error rates with a baseline under representative traffic.

• Decide when the cache cost is worth eating

  A correctness, security, or compatibility fix may justify a cold-cache window, while a cosmetic description edit usually does not.

Chapters

1. 1

   Tool definitions in the cached prefix

   Tool definitions travel at the head of every request, which is exactly where a provider's cache starts matching. Fix a typo in the third of forty tool descriptions and roughly 88,800 tokens of a 90,000-token prefix stop matching until caches rewarm, because nothing in a serialised request is treated as annotation — descriptions included. What an edit costs is decided by where it lands and by how much stable content sits behind it.

   Read chapter
2. 2

   Tool-block placement in the prefix

   A forty-token preamble naming the customer, plan and region, placed ahead of the tools array and a 60,000-token policy document, explains a hit rate that never rose above 2%. The instrument that prices a tool edit is an offset map read off the serialised request: for one thirty-tool catalogue, rewording escalate_to_human at offset 3,940 invalidates 61,900 tokens. Also why sorting the tools array alphabetically for tidiness is a change at the very front of that prefix.

   Read chapter
3. 3

   Classifying a tool-schema edit

   Every proposed edit to a tool gets two questions rather than one: does the model see a different contract, and does the serialised prefix differ? Changing what a handler does behind an unchanged schema is invisible to both. A serialiser upgrade that emits a title field and reorders keys is invisible to the model and total to the cache — one such migration moved the divergence point to roughly token 30 and took hit rate to near zero for a cache window.

   Read chapter
4. 4

   Deprecated fields and conversation history

   A conversation that runs for days re-sends every prior assistant turn on each request, so tool calls written against the old schema keep being demonstrated to the model on exactly the turns where you want the new shape. That is why traffic on a deprecated field decays unevenly — 41% on day one, 1.4% at three weeks, with Monday spikes from transcript replay — rather than stopping on a deadline. Retirement is a signal you watch across live, retry and replay traffic, not a date you pick.

   Read chapter
5. 5

   Renaming a tool parameter

   Rename customer_id to account_id in place and the calls sitting in thirty days of replayable transcripts stop validating — 6,100 of one store's 74,000 — while a deprecation date changes nothing about what a stored history contains. The additive shape adds the new field, keeps the old one callable through an adapter, and retires it on a signal, at a price of two cold windows about fifty days apart. Where a separate v2 tool is the better answer, and what the alias cannot fix.

   Read chapter
6. 6

   Measuring a cold-cache window

   Spend up 3% and p95 up 40 ms, both inside normal variation, is an instrument incapable of detecting a twelve-minute event confined to one prefix — the $184 it cost is a fifth of one standard deviation of daily spend. Measure cache hit rate, the cached-versus-uncached split of prefix tokens and TTFT in one-minute buckets, against the same clock window on a comparable day, and watch for several cache lifetimes so a one-time window can be told from a permanent plateau.

   Read chapter
7. 7

   Batching and scheduling schema edits

   Three queued edits shipped separately cost three cold windows, roughly 1,800 degraded requests and about $550; shipped together at a 03:10 trough they cost one window, 130 requests and near $40 — and the window gets longer in wall clock, not shorter. The classification that decides which changes may wait is by what the change fixes. A raw SQL filter that security wants removed is the case the additive playbook gets wrong, because there compatibility is the thing being removed.

   Read chapter