Prompt caching mechanics

Read these first

• KV cache
• Token counting and budgeting before the call

What you will know

• Order a prompt so the stable part comes first and is cacheable

  A request-specific suffix can then change without discarding the reusable token prefix across users and turns.

• Place explicit cache breakpoints where the provider supports them

  A misplaced boundary can create an unnecessarily short segment or prevent the provider from reusing the intended prefix.

• State the minimum cacheable prefix length for your provider

  Providers enforce token-count and sometimes request-shape thresholds, so a nearly long-enough prefix may still produce zero reuse.

• Measure cache-write and uncached input costs across hit rates

  The write premium is amortized only by later hits; cold or low-reuse traffic can make caching more expensive than sending input normally.

• Reproduce cache misses caused by three silent invalidators, including a timestamp in the system prompt

  Prefix caches usually compare serialized inputs exactly, so metadata, tool-schema, or ordering changes can discard all downstream reuse.

Chapters

1. 1

   Prompt caching

   A provider's prompt cache is addressed by the content of a request's leading tokens, not by a session or a conversation, so any two requests in the same organization that begin with identical bytes read the same entry. This chapter separates it from the key/value cache inside a single generation, works the arithmetic on a one-shot endpoint where fifty callers a minute share a 15,000-token system prompt, and names the conditions that decide whether the discount arrives.

   Read chapter
2. 2

   Prompt block ordering

   Prefix matching walks forward from token zero and stops at the first byte where two requests differ, so the order of the blocks in a prompt decides how much of it is eligible for reuse. This chapter defines the divergence point, gives the rule that produces the right order — blocks sorted by how many requests carry an identical copy — shows how to measure the shared span from request logs, and works through an assembler whose 23,000 stable tokens were worth nothing because a 500-token profile sat in front of them.

   Read chapter
3. 3

   Automatic prefix caching and breakpoints

   Providers implement prompt caching under two designs. Automatic prefix caching hashes the leading tokens of every request without being asked, matches at a fixed increment, and can never make a request more expensive. Explicit breakpoints cache nothing until a marker says where the reusable region ends, and pay for a deeper read discount with a premium on the first fill. This chapter sets out what each regime asks of the caller, what the request looks like in both, and the two-call experiment that establishes which one you are in.

   Read chapter
4. 4

   Minimum cacheable prefix length

   Every provider publishes a token count below which a cacheable segment does not qualify, and the rule is enforced as a cliff: 1,024 tokens caches in full, 1,023 caches nothing, and the request that asked for it succeeds without an error. This chapter gives the thresholds by model family, shows how the usage counters distinguish a segment that never qualified from one that qualified and is unstable, and works through widening a short prefix with content the model needed anyway.

   Read chapter
5. 5

   The cache write premium

   Filling a cache entry bills the prefix above the ordinary input rate — commonly 1.25× — and that premium is recovered only by later requests that read the entry. What a prefix is worth is therefore its size multiplied by its reuse count, which puts the break-even at the second matching request regardless of size, and makes a 1,500-token classifier sent two million times a day worth roughly forty times a 40,000-token document read six times.

   Read chapter
6. 6

   Cache TTL and hit rate

   A reuse count is what remains of your traffic after the cache's time to live has been applied, so an entry's lifetime is what converts an arrival pattern into an expected number of reads per write. This chapter covers sliding against absolute expiry and why the difference is worth 40% on the same six questions, computes expected writes from a distribution of inter-arrival gaps, and gives the hit rate below which caching costs more than sending the prefix plainly.

   Read chapter
7. 7

   Layered cache breakpoints

   A cached segment is keyed on every byte up to its boundary, so the boundary decides which requests are able to read the entry it produces. Placing one marker to capture the largest possible span makes every entry specific to one user or one conversation and writes the shared tokens in front of it once per account. This chapter works through nested boundaries at the seams between readerships — shared, tenant, conversation — the traffic assumptions each layer rests on, and how to spend four markers on five seams.

   Read chapter
8. 8

   Silent cache invalidation

   A prefix that varies between requests turns every call into a cache write, which bills above the uncached rate and produces no error. Three separate causes give identical counters: rendered per-request metadata inside the prefix, serialization drift from a dependency upgrade, and request fields outside the prompt text that participate in the key. This chapter derives the cost from a hit rate, separates the three causes, and gives the two tests and one metric that catch them at the pull request rather than on the invoice.

   Read chapter