Measuring a cold-cache window

Every provider returns, on each response, the split between tokens that were read from cache, tokens that were written to cache and tokens that were neither. That split is what makes a schema deploy measurable at all, because a cold window is precisely a mass migration of one of those counters into another, and it is invisible in anything computed over a longer period. Chapters 1 through 5 have been putting numbers on tool edits — 88,800 tokens here, 118,000 there, fifty days of compatibility over there — and every one of those numbers has to come from somewhere before it can be argued about.

The failure this chapter is about is not a missing dashboard. It is a dashboard that answers the question, in the sense of returning a number, using an instrument that integrates over a period far longer than the event. Doing the arithmetic on what such an instrument could have shown matters more than asserting that it is too coarse, because "the metric is too coarse" is easy to say and easy to wave away in a room where the metric is the only one anybody has.

What daily spend and a daily p95 could have shown#

Take an endpoint running about 50 requests per minute with an invalidated region of 88,800 tokens. Twelve minutes of cold serving is around 600 requests each paying the cache-write rate on tokens they would otherwise have read from cache. At $3.75 per million tokens written versus $0.30 per million read, the delta is $3.45 per million, and 600 × 88,800 = 53.3 million tokens, which comes to about $184.

Each of those three operations is doing a specific job and the middle one is where the arithmetic usually goes wrong. Multiplying the request rate by the duration gives the population that fell inside the window, which is the only set of requests that behaved differently. Multiplying that population by the invalidated region gives the tokens that changed category, and the region is the same on every one of those requests because a cold prefix is cold in full. Pricing those tokens at the difference between the two rates, rather than at the write rate, is the step that gets skipped. The endpoint was always going to pay for those tokens; on a normal day it pays the read rate for them. What the deploy caused is not the whole write charge but the gap between what was paid and what would otherwise have been paid, so the write rate overstates the event by the read rate — close to a tenth of the figure here, and much more on providers whose read rate is a larger fraction of the write rate.

Daily spend on that endpoint averages $4,200 with a day-to-day standard deviation of roughly 7%, or $294. The event is smaller than one standard deviation of the measurement, so it cannot be separated from the noise even in principle, and a 3% reading on the chart is not a measurement of the deploy at all — it is Tuesday being slightly busier than Monday. Had the cold window cost nothing, the dashboard would have shown something in the same range.

The latency number fails for a different reason, which is dilution rather than noise. The endpoint serves about 72,000 requests a day, so 600 affected requests are 0.83% of the population. A p95 computed over the full day is the boundary of the slowest 3,600 requests; adding 600 requests that all land in the slow tail shifts that boundary by 0.83 percentile points, so the reported p95 becomes roughly what p94.2 was the day before. Where the distribution between p94 and p95 is flat — and on a well-behaved endpoint it usually is — that shift is tens of milliseconds. A reading of +40 ms is the expected result for an event where every one of those 600 requests took 3.9 seconds instead of 640 ms, which is a sixfold degradation in time to first token (TTFT), the wait before any of the answer appears, for the users who hit it.

What makes that calculation counter-intuitive is that a percentile does not report a magnitude. It reports the value sitting at a position in a sorted list, and the position is fixed as a share of the population — the slowest twentieth, in the case of a p95. Adding requests that all sort into the slow end does not raise that value by anything like their own slowness; it slides the boundary down the list by the share of the population they represent, and the reported number then changes by however much the distribution happens to rise across that short distance. Two facts follow, and both are usually missed. A p95 is insensitive to how slow the slow requests were, so an event three times worse would have moved the same reading by the same amount. And the size of the shift is a property of the distribution's shape near the boundary rather than of the event, which means a reading of tens of milliseconds is not a small measurement of the incident. It is not a measurement of the incident at all.

This is the same failure as photographing a lightning strike with a thirty-second exposure and concluding the sky was uniformly grey. The instrument integrated over a period far longer than the event, the event's energy got spread across that period, and the resulting image is not a weak signal that needs a better eye — it is an average, and the average is genuinely what the instrument was built to report. No amount of staring at daily spend recovers a twelve-minute window, and the correct response to being asked for a number is to say the dashboard cannot produce one and then go and get the data that can.

That comparison is an analogy and it stops being one at a specific place, which is worth marking because the place it stops is where the second half of this chapter lives. A long exposure averages a real signal and returns an honest average of it; the daily figures do that much and no more, and to that extent the picture holds. What the photograph has no equivalent of is the band drawn around the average. Daily spend is not merely blurred, it is being judged against a normal range built from a different phenomenon — weekday traffic drift — so the instrument does not just fail to resolve the event, it supplies a plausible alternative explanation for whatever it does show. A blurred photograph at least looks blurred. A reading of three percent against a band of seven looks like an answer.

The numbers that answer the question#

The usage block on each response distinguishes the three kinds of input token, and the whole measurement is built from it.

{
  "usage": {
    "input_tokens": 340,
    "cache_creation_input_tokens": 88800,
    "cache_read_input_tokens": 0,
    "output_tokens": 190
  }
}

A steady-state request on the same endpoint carries cache_read_input_tokens: 88800 and cache_creation_input_tokens: 0, so the deploy is visible as a mass migration of one number into the other. Logging those three fields per request, with the endpoint and the deploy SHA, is the entire instrumentation requirement, and it is usually a two-line change to whatever wrapper already logs token counts for billing.

SELECT
  date_trunc('minute', ts)                             AS minute,
  count(*)                                             AS requests,
  sum(cache_read_tokens)                               AS read_tokens,
  sum(cache_write_tokens)                              AS written_tokens,
  sum(cache_read_tokens)::float
    / nullif(sum(cache_read_tokens + cache_write_tokens + uncached_tokens), 0)
                                                       AS hit_rate,
  percentile_cont(0.5)  WITHIN GROUP (ORDER BY ttft_ms) AS ttft_p50,
  percentile_cont(0.95) WITHIN GROUP (ORDER BY ttft_ms) AS ttft_p95,
  avg(CASE WHEN tool_error THEN 1 ELSE 0 END)          AS tool_error_rate
FROM model_requests
WHERE endpoint = 'support-agent'
  AND ts BETWEEN '2025-03-11 13:30' AND '2025-03-11 15:30'
GROUP BY 1 ORDER BY 1;

Run against a deploy day, that produces twelve consecutive minutes where hit_rate collapses and written_tokens spikes, and it produces the cost directly: the sum of written_tokens over those minutes, minus what would have been read, priced at the difference between the two rates. That is a defensible $184 rather than an inferred one, because it counts the actual tokens that were actually written.

Cache hit rate alone is not sufficient, which is why the query pulls four things. Hit rate tells you the cache missed; the cached-versus-uncached token split tells you how much prefix each miss had to prefill, and those come apart — a hit-rate dip on an endpoint with a 4,000-token prefix costs almost nothing, and the same dip on a 120,000-token prefix is a serious event. TTFT converts the token number into the thing users experienced. The error rate is in there to establish that nothing broke, because a cache measurement that does not also check errors leaves open the possibility that you are measuring a partial outage rather than a cold window.

Comparing against the right thing#

Running the same query over the same clock window on a comparable day is what makes the comparison honest, and "comparable" carries weight. Comparing 14:00–15:00 on the deploy day against 13:00–14:00 the same day looks tempting because it is adjacent, and it fails, since 14:00 is on the ramp into the afternoon peak on most business-hours products, so the hour before has a different request mix, a different rate of new-conversation starts and a different steady-state hit rate. The same weekday one week earlier holds traffic shape roughly constant and is the default worth using; if a promotion or an outage makes that week unrepresentative, the median of the same weekday across three prior weeks is a reasonable substitute.

The observation window has to extend well past the visible recovery, for several cache lifetimes after the deploy, because the shape of the recovery is what distinguishes two very different outcomes that look identical in their first minute. A one-time invalidation shows hit rate returning to its prior level and staying there once every serving node has refilled. A structural regression — a per-request value that has crept into the prefix, a tools array whose order now depends on something non-deterministic, a serialiser that emits floats differently under load — shows hit rate settling at a new, lower plateau.

A plateau is not a window#

The recovery shape is worth more than the cost figure, because the two outcomes it separates differ by orders of magnitude. On a 120,000-token prefix at 72,000 requests a day, a hit rate that lands at 78% instead of 96% means about 12,960 requests a day moving from a cache read to a cache write on 120,000 tokens, at the $3.45 per million difference between the two rates — roughly $5,400 a day, indefinitely. That is not a $184 event with a long tail; it is a different category of problem that gets confused with a cold window constantly, because both begin with a cliff and only one of them comes back up.

The daily figure there is built the same way as the window's and differs in one term, which is what makes the comparison legitimate. A hit rate landing eighteen points below where it sat means eighteen percent of the day's requests are reading nothing they used to read, and against seventy-two thousand requests that is the twelve thousand nine hundred and sixty in the sentence above. Each of those carries the full prefix from cache creation instead of cache read, and each is priced at the same difference between the two rates as before. The structure is identical to the twelve-minute calculation — a population, a region, a rate difference — and the only thing that changed is that the population is now a day's worth rather than a window's, and it recurs tomorrow. That is why the two outcomes differ by orders of magnitude despite looking the same in their first minute: one multiplies the region by the requests inside a window, the other multiplies it by every request there will ever be.

A holdout is the measurement people reach for next and it deserves a specific warning here. Routing 10% of traffic to the old schema so you can compare directly does not give you a clean control, because the two populations now send two different prefixes and therefore fill two separate cache entries, so the holdout arm pays its own cold window at the moment you create it and both arms run at a lower hit rate than the unsplit endpoint did. The technique that works well for feature rollouts measures something other than what it measures here, and if you need a control, the same window on a comparable day is closer to one.

Twelve minutes of elevated TTFT also has a consequence that money does not capture. If the agent carries a p95 latency service level objective (SLO), a published target the team has committed to meeting, twelve minutes at 3.9 seconds is a question about the error budget — the amount of that target a team may spend before it has to stop shipping — rather than a cost question, and it belongs in whatever process already governs deploys that spend budget, the same category as a database migration that locks a table briefly.

Reporting $184 as a small number is the honest outcome, and known small quantities behave differently from unknown ones in two directions. They stop being an argument for freezing schema changes, which is the failure mode teams reach when someone finally notices the cliff and nobody can price it. And they start being an argument about frequency, because $184 is the cost of one deploy, and a team that ships schema edits eleven times a month as they occur to people is paying eleven of those windows — including for a reworded description that changed nothing the model does.

That is what measuring buys: it moves schema edits from "invisible, therefore free" into "priced, therefore schedulable". Scheduling is where most of the money actually is, and it is also where the one class of change that cannot wait for a schedule has to be identified and let through.