An answer that was correct on Monday decays silently

The refund window went from 30 days to 14 on Tuesday at 09:00, announced internally the previous week and published to the help centre by the content team on schedule. On Thursday afternoon a support lead pastes a customer transcript into the channel in which the assistant states, clearly and politely, that returns are accepted within 30 days of delivery. The cache entry behind it was generated on Monday at 16:20 and its last hit was six minutes ago.

The first reaction in the thread is that the TTL must be misconfigured, because it is set to 24 hours and the entry is 68 hours old. The TTL is working exactly as the library documents it: the client refreshes expiry on read, so every hit pushes the deadline out another day. Sliding expiration is the default in several popular cache clients and it is a reasonable default for the workload those clients were designed for, where the thing being cached does not go out of date so much as go unused. Applied to an answer whose correctness depends on a document, it guarantees that the entries which never expire are the ones being served most often. The most popular question on the surface is the one with the longest-lived wrong answer.

A time window is not connected to the thing that changed#

Switching to absolute expiry and cutting it from 24 hours to one is the proposal that follows, and it sounds like a control because it has a number in it. What it actually does is bound the duration of wrongness for a change nobody has told the cache about, in exchange for regenerating everything twenty-four times as often.

The refund policy changes roughly four times a year. At a one-hour TTL, each cache entry is regenerated 2,184 times per quarter, of which one regeneration lands after a policy change and does something useful, and 2,183 reproduce an answer that was already correct. The hit rate falls from 34% to 12%, taking most of the saving that justified the project, and after all of that a customer can still be told the wrong policy for up to an hour after it changes. Every number in the table is a different point on the same bad trade, and none of them is derived from anything about the refund policy, because a TTL cannot be — the cache has no idea that a policy exists.

The second failure of the TTL as a control is that it answers no question after the fact. On Thursday afternoon the outstanding item is not only stopping the wrong answer, it is finding out which customers were told the old window and over what period, and a shortened expiry has nothing to say about either. It changes future behaviour and leaves the past unreadable.

The source data drives the invalidation#

The mechanism that does connect is provenance. Every generated answer was produced from something — retrieved chunks, a policy document, a knowledge base article, a pricing table — and the identity and version of those inputs is available at generation time and thrown away by every naive cache implementation. Storing it turns invalidation from a timer into a lookup.

entry = {
    "vector":       q_vec,
    "question":     question,
    "response":     response,
    "generated_at": now,
    "sources": [
        {"doc_id": "kb/refund-policy",   "version": "v7", "chunk": 3},
        {"doc_id": "kb/returns-process", "version": "v2", "chunk": 1},
    ],
    "prompt_version": "sys-2026-05-11",
    "model":          "claude-haiku-4-5",
    "embed_model":    "text-embedding-3-small",
}
index.upsert(entry)
for s in entry["sources"]:                 # inverted index: doc -> entries
    redis.sadd(f"cache:by_doc:{s['doc_id']}", entry_id)

The inverted set is the part people leave out and then need. Vector stores are built to answer "what is near this vector" and are poor at "give me every entry mentioning kb/refund-policy", so a scan over 40,000 entries to find the 300 that used a document is slow enough that nobody runs it during an incident. A Redis set per document id makes the eviction a set read and a batch delete, in single-digit milliseconds.

The event is the piece that has to be built, and where it comes from depends on where the truth lives. A CMS with webhooks emits it directly. A database-backed policy table gives it up through change data capture, which is the stream of row-level inserts and updates a database can publish as they commit, or through a trigger writing to an outbox table that a worker drains. A pipeline that re-indexes documents nightly can emit one event per changed document at the end of the run. Whatever the source, the useful property is that invalidation now happens seconds after the change rather than at an interval unrelated to it, and it invalidates only the entries that were built from the thing that moved — 312 entries out of 40,000, leaving the other 99% of the cache intact and earning.

Two design questions come up immediately and both have a conservative answer. The first is whether to evict when only one of an answer's five source chunks changed, given that a typo fix in a chunk the answer barely used will throw away a perfectly good entry. Trying to decide materiality means diffing the source and reasoning about whether the changed sentence influenced the generated text, which is a judgement the cache is not equipped to make and which fails in the direction of keeping wrong answers. Evicting on any version change of any listed source is cheap — the regeneration costs about a cent and a half — and it fails in the direction of a miss. The second question is what to do about answers with no sources at all, which is what an unretrieved, parametric answer — one written out of the model's own weights, with no document behind it — looks like in this schema. An empty sources array is a signal that nothing can invalidate the entry, and on a surface where correctness matters the reasonable rule is to refuse to store those entirely, because they are exactly the answers whose factual basis you cannot audit later.

Warming the evicted head is a small optional refinement with a real benefit, because eviction of a popular document produces a burst of simultaneous misses on the same question. Regenerating the top twenty affected questions in the background before traffic arrives costs about thirty cents and avoids the thundering herd that would otherwise arrive at the model at once.

TTL does not disappear from this design; it stops being the mechanism and becomes the backstop. Some inputs cannot be tracked — a third-party status page the assistant summarised, a tool response you have no change feed for, a claim the model produced from its own parameters with no retrieved source behind it. A seven-day absolute expiry over the whole cache bounds how long an untracked input can persist, and at seven days it costs almost nothing in hit rate because it only bites on entries that would have been evicted by a source change anyway if the source were tracked.

There is an escape hatch worth having alongside the automated path, which is a manual eviction endpoint that takes a document id or a free-text pattern and drops everything matching. On Thursday afternoon, before any of the event plumbing exists, the thing the team needs is the ability to remove 312 entries in ten seconds without flushing a cache that took nine days to fill. A flush is the tool everyone reaches for under pressure, and it converts a policy problem into an hour of elevated latency and full-price traffic on top of the incident.

The requests that must never be served from a cache#

Provenance handles answers that were correct when written and became incorrect when a document changed. There is a class it cannot handle at all, and that class belongs on the denylist established in the previous chapter regardless of similarity score, provenance or freshness machinery: requests whose correct answer depends on when they are asked.

"How long do I have left to return this?" resolves against today's date. "Is the API currently degraded?" resolves against a status feed that changes by the minute. "When is my next payment taken?" resolves against a billing cycle. "Is this in stock?" resolves against inventory that moved while the answer was being written. None of these has a source version that changes — kb/refund-policy is still v7, the status page still says what it said — and the answer is still wrong, because the reader's now moved rather than the source. An eviction rule keyed on document versions will never fire for any of them.

Detecting them is a write-side rule in the same gate that filters tenant-dependent answers. If the generation read the clock, called a status or inventory tool, or produced text containing a resolved relative expression — "you have 12 days left", "as of today", "currently", a concrete date computed from the request time — the response is not eligible for storage. A regex over the answer text is a crude implementation and it catches most of the volume, and the intent classifier that already gates the allowlist can carry a time_dependent label for the rest. The default on ambiguity is not to store, which costs a hit and never costs a wrong answer.

Counting a leak that crossed no boundary#

The cross-tenant defect in the previous chapter was countable with a join, because the wrongness was a mismatch between two recorded identities. The stale-policy defect has no equivalent, and quantifying it is the harder of the two.

With provenance in place from the start, it is close to tractable. Every hit logs the matched entry, every entry lists kb/refund-policy at some version, and the change event has a timestamp, so the affected population is every hit after Tuesday 09:00 on an entry carrying v7.

SELECT count(*) AS bad_hits, count(DISTINCT r.user_id) AS users
FROM   responses r
JOIN   cache_entries e ON e.entry_id = r.matched_entry_id
CROSS  JOIN LATERAL jsonb_array_elements(e.sources) AS s
WHERE  r.cache_hit
  AND  r.ts >= timestamp '2026-05-12 09:00'
  AND  s->>'doc_id'  = 'kb/refund-policy'
  AND  s->>'version' = 'v7';

Without provenance, which is the situation on Thursday afternoon, the reconstruction is heuristic. You can find every hit after the change whose entry was created before it, which on this system is 2,140 responses, and then search the stored answer text for the old window — \b30[- ]day returns 611 of them. Both numbers are wrong in known directions: 2,140 is an over-count because most of those entries never mentioned the refund window at all, and 611 is an under-count because an answer can imply the old policy without stating the number, and because the text search misses phrasings nobody thought to grep for. The honest report to an incident review is a bounded estimate with the method attached, and the reason it is only an estimate is that the information needed to make it exact was discarded at write time.

That asymmetry is the argument for provenance that survives contact with a planning meeting. The two extra fields cost nothing to write and are the difference between naming 611 affected customers with a caveat and naming them with a query. What neither chapter has settled is whether a cache carrying all of this machinery — scoped keys, a write gate, source-driven eviction, a shadow evaluation and a holdout — is worth running at all, and that decision turns on numbers in units nobody has converted yet.