A saving figure that reconciles against the invoice

The quarterly infrastructure review has one slide on prompt caching and one number on the slide: platform hit rate, 71%, with a note that it was 70% last quarter and 72% the quarter before. The director's question is always the same — how much is prompt caching saving us? — and the answer has been some version of "a lot, it's a 90% discount on cached tokens" for four quarters running. Meanwhile the service runs eight cacheable routes, at least one of which has never produced a single cache read and one of which is above 95%, and neither of those facts has ever appeared on a slide.

Everything the previous five chapters built exists to produce two artefacts: a report a finance team can reconcile, and an alert that pages the right person. Both are mostly a matter of picking columns.

What each column is for#

A report column earns its place by supporting a decision. Eight routes, one row each, and every column below answers a question somebody actually asks.

Route is the unit of ownership, the unit of the alert, and the unit at which a fix gets scheduled. Prefix class is what turns a number into a judgment: 74% on a per-session prefix is close to that workload's ceiling, while 62% on a shared prefix that should be resident all day is a defect worth a ticket. Eligible lookups sizes the opportunity and, in the adhoc-search row, records the coverage question — 140,000 requests a day that no cache can serve because the prompt is under the minimum length, which is a fact worth carrying rather than a failure worth hiding.

Hit rate is the operational signal from chapter 1 and belongs next to the route's own ceiling rather than next to any global target. Write and read tokens are the raw material of the money column and are worth showing rather than folding away, because reads-per-write is the reuse figure that decides whether the route clears chapter 2's threshold: doc-qa at 0.11 reads per write is far below the 0.28 break-even, and bulk-reprocess at zero has never had a chance.

Net saving is the column the director reads, and it is the one with a wrong version in wide circulation.

Two columns are deliberately absent. There is no platform total row for hit rate, because chapter 1 established that the blend is a statement about traffic mix rather than about the cache, and a total that gets quoted in a meeting is a total that will eventually be operated on. And there is no target column, because a single target across routes with different prefix classes would be wrong for at least six of the eight — the comparison that means something is each route against its own ceiling and its own last quarter, both of which the reader can do from the rows that are there.

The rows are sorted by net saving rather than by hit rate, which changes what a reader takes away. Sorted by hit rate, bulk-reprocess at 0% is the alarming line and batch-summarise at 62% looks like the second problem. Sorted by money, bulk-reprocess costs $81 a day and is a tidy-up, while doc-qa at ten times that is the thing to schedule. Ordering a report is an editorial act, and ordering it by the quantity people are meant to act on saves an argument every quarter.

The saving figure that reconciles#

The tempting calculation is cached tokens times the 90% discount. It is clean, the finance team can follow it, and it is always too large. Write out both sides properly with p as the base input price per token:

Actual input spend is write_tokens × 1.25p + read_tokens × 0.1p + uncached_input × p. The counterfactual — the same traffic, the same prompts, caching disabled — is (write_tokens + read_tokens + uncached_input) × p. Subtract, and the uncached input cancels, along with every output token on both sides:

def net_saving_usd(write_tokens: int, read_tokens: int,
                   price_per_mtok: float = 3.0) -> float:
    """Actual spend versus what the same traffic would have cost uncached.
    Reads give back 0.9x base; writes cost an extra 0.25x that is paid
    whether or not a read ever follows."""
    return (read_tokens * 0.9 - write_tokens * 0.25) * price_per_mtok / 1e6

def naive_saving_usd(read_tokens: int, price_per_mtok: float = 3.0) -> float:
    """The version that ends up on slides. Prices every cached token as though
    the alternative were full price with no write ever having happened."""
    return read_tokens * 0.9 * price_per_mtok / 1e6

net_saving_usd(1_296_000_000, 144_000_000)    # -583.20  (doc-qa, per day)
naive_saving_usd(144_000_000)                 #  388.80  (same route, same day)

Across the eight routes the correct figure is $14,834 a day, about $445,000 a month. The naive figure is $16,288 a day, about $489,000 — an overstatement of roughly 10%, and a sign error on the two routes that are actually losing money. doc-qa appears on the naive report as a $389-a-day contributor when it is a $583-a-day cost, a swing of $972 a day that points the next quarter's engineering effort in exactly the wrong direction.

The overstatement is systematic and one-directional: it omits write_tokens × 0.25p, which is never negative. Any figure produced that way is larger than the truth, and the gap grows precisely on the routes with poor reuse — the ones that most need to be visible.

The headline, and the sentence that travels with it#

Asked "how much is prompt caching saving us", the answer is: about $445,000 a month, on a total model spend of $1.6 million.

The caveat is one sentence and it has to be attached every time, because the figure is a counterfactual — a comparison against a world that did not happen — rather than a reduction against a bill that did. It measures this quarter's traffic against what this quarter's traffic would have cost with caching disabled, on the same models and the same prompt layouts. It is not a fall against last quarter's invoice — volume grew 22%, so the invoice went up — and it touches only repeated input, which is why a 38% share of spend on output tokens sits entirely outside it. Someone will eventually try to reconcile $445,000 against a line on a bill and find nothing; the sentence is what stops that from becoming a credibility problem.

The two negative rows are worth a sentence of their own in the write-up rather than a footnote, because they are the part of the report a director has never seen before and the part that establishes it is honest. Saying "two of our eight routes cost us money and here is the plan" buys more credibility than any single positive figure, and the plan is short: bulk-reprocess gets its breakpoint removed this week, and doc-qa gets the restructuring from chapter 2 — cache the shared instruction block that sits ahead of every document, leave the document itself uncached — which turns a $583-a-day loss into a modest gain rather than merely stopping the bleeding.

Two more numbers belong under the headline. Total saving available at a hypothetical 100% hit rate across all eight routes is about $498,000 a month, using the ceiling calculation from chapter 3, so the programme has already captured 89% of everything it can ever capture. And the two negative routes together cost $664 a day, roughly $20,000 a month, which is the largest single improvement left and is a configuration change rather than an optimisation.

What the alert fires on, and what it deliberately ignores#

A report is a quarterly artefact and the wrong instrument for chapter 5's incident, where seventy points disappeared in ten minutes. The alert is per route and per model, compares each route against its own recent behaviour, and never looks at the platform figure. It is written below as Prometheus-style alerting rules, where expr is the condition evaluated on a schedule and for is how long that condition has to hold before anybody is woken.

- alert: PromptCacheHitRateCollapse
  # Per route+model, against that route's own 7-day median. A route whose
  # healthy value is 74% and one whose healthy value is 95% both work.
  expr: |
    (
      sum by (route, model) (rate(cache_eligible_hit_total[15m]))
      / sum by (route, model) (rate(cache_eligible_total[15m]))
    )
    < 0.7 * quantile_over_time(0.5,
        (
          sum by (route, model) (rate(cache_eligible_hit_total[15m]))
          / sum by (route, model) (rate(cache_eligible_total[15m]))
        )[7d:15m])
    and sum by (route, model) (rate(cache_eligible_total[15m])) > 0.5
  for: 10m
  labels: { severity: page }

- alert: PromptCacheSilentlyNotStoring
  # The third bucket from chapter 1: marked cacheable, provider stored nothing.
  # Almost always a prefix under the minimum length after a template edit.
  expr: |
    sum by (route, model) (rate(cache_eligible_miss_nocache_total[30m]))
    / sum by (route, model) (rate(cache_eligible_total[30m])) > 0.05
  for: 30m
  labels: { severity: ticket }

Comparing against the route's own rolling median is what lets one rule cover eight routes with ceilings from 62% to 95%. A fixed multiplier of that median — 0.7 here — expresses the thing you actually care about, which is a discontinuity rather than a level. It also self-heals after a legitimate change: ship chapter 4's prefix reorder, and the median follows the new plateau within a week, so the alert starts guarding the improved level without anyone editing a threshold.

The volume predicate matters more than it looks. Without it, a route doing two requests an hour pages someone every time both of them happen to miss, and an alert that cries wolf on low-volume routes is an alert that gets silenced for the high-volume ones too.

What deliberately does not fire is as much of the design as what does. The platform average never fires, because chapter 1's arithmetic guarantees it cannot move enough — a seventy-point collapse on one route registered as three points of blend, well inside normal week-to-week variation. Cost per request never fires either: chapter 4 showed it moving 30% on a mix shift with the cache behaving identically, so paging on it produces a false alarm every time a large customer onboards. An absolute threshold — "alert below 80%" — never fires, because session-assist lives at 74% by the arithmetic of its own workload and would page forever while a shared-prefix route falling from 95% to 82% would page not at all. And ineligible request counts never page; adhoc-search growing to 300,000 requests a day is a coverage conversation for the next review, not a 3 a.m. event.

Where to start on a service that has none of this#

The order that gets a result fastest is not the order this topic presented. Instrument first: the three counters from chapter 1, tagged with route, model, tenant and prefix class, which is one function and an afternoon. Run them for a full week without changing anything, so the medians the alert compares against are real. Then compute the ceiling from chapter 3 for your two highest-spend routes before writing any optimisation code, because that number tells you whether the work is worth funding at all — and on a workload dominated by output tokens, the honest answer is often that it is not.

The page that fires needs a runbook — the short written procedure the on-call engineer follows when it arrives — and the runbook here is chapter 5 compressed into four lines: check whether a deploy went out in the preceding fifteen minutes; compare prefix digests across two equivalent requests; if the digests match, inspect the routing hint; if they differ, find the first differing byte offset and see whose code owns it. An engineer who has never thought about prompt caching can follow that to a diagnosis, which is the only test a runbook has to pass.

After that the sequence is mechanical. Sort the routes by net saving, fix or disable anything negative, and take the largest ceiling with the largest gap between ceiling and realized. Ship each change behind a flag with a holdback so the write-up survives contact with a customer onboarding. And put the digest test in CI on the first day, before anyone has had the chance to break a prefix layout that nobody yet knows is load-bearing.