A corpus frozen at generation time has no freshness dimension

At 09:12 the eu-west API started returning 503s on about a third of requests. The status page was updated at 09:31 with "Investigating: elevated error rates in eu-west". The documentation assistant's retrieval index is rebuilt nightly at 06:15, and the status page is one of the sources it ingests, so the copy it held said "All systems operational — last incident 3 March". Between 09:40 and 12:30, 412 conversations asked some version of are you having problems right now, and the assistant answered nearly all of them with a confident no, followed by troubleshooting steps for a client-side misconfiguration. Sixty-one of those conversations turned into support tickets, several of them from customers who had spent an hour checking their own network.

The eval suite at that point was 500 cases, and it had run green on every commit for four months, including the commit deployed that morning. Nothing in it could have caught this, and the reason is worth stating structurally rather than as an oversight. Every case in the set has the form expected answer = f(question, corpus), where corpus is the dump the generator read on 28 February. There is no t anywhere in that function. Whether the case is run at 09:40 during an outage or at 04:00 on a Sunday, the correct answer is the same, so the suite has no way to express the sentence "this answer was right yesterday and is wrong now", which is precisely what 412 customers experienced.

The last note is the finding. The staleness was not detected and rejected; it was never represented. The chunk carried no age, the prompt asked for no age, and the eval had no case in which age mattered, so three independent parts of the system were consistent with each other and all wrong.

Why the generator was never going to produce this#

A frozen corpus dump presents every fact as timeless. "The eu-west region is operational" and "The default retry limit is 3" arrive in the same format, with the same authority, and a generator reading them cannot tell which one has a half-life of minutes and which one has a half-life of years. Asked to write questions and answers, it writes them for facts, and the resulting cases inherit the corpus's implicit claim that all of its contents are equally current.

Prompting harder does not help, which is worth demonstrating because it is the first thing anyone tries. Ask the generator for time-sensitive questions and it produces questions about time: how long does provisioning take, when does the trial expire, what is the retention period. Those are questions whose answers are stable facts about durations. What was needed is a question whose correct answer changes depending on when it is asked, and that cannot be generated from a corpus alone, because the case is not a property of the corpus at all — it is a property of the relationship between two corpus states and a clock. The generator has one corpus state and no clock.

The shape of a case that has a clock#

The repair is a new case class with a fixture format the other eleven do not need. A freshness case carries two corpus states, a time at which the question is asked, and an expected behaviour that is a constraint on the answer rather than a reference string.

id: freshness/status-during-incident-3h
class: freshness
seeded_from: INC-2291 (2026-03-14)
corpus_at:
  t0: "2026-03-14T06:15Z"     # indexed snapshot the system will retrieve
  docs:
    - id: status-page
      text: "All systems operational. Last incident: 3 March."
      source_ts: "2026-03-14T06:14Z"
      freshness_slo: 300        # seconds; this source is expected to be near-live
ground_state_at:
  t1: "2026-03-14T09:31Z"     # what is true when the question is asked
  docs:
    - id: status-page
      text: "Investigating: elevated error rates in eu-west."
asked_at: "2026-03-14T09:40Z"
question: "are you having problems right now?"
expect:
  must_not: assert_current_operational_state_without_qualifier
  must_one_of:
    - qualify_with_as_of_timestamp     # "as of 06:15 there were no reported issues"
    - decline_and_link_live_source
    - trigger_live_fetch_tool

Running it requires the harness to lie about the time and about the index, which is a small amount of plumbing that pays for itself across the whole class.

def run_freshness_case(case, system):
    index = build_index(case.corpus_at.docs)          # what the system can retrieve
    with frozen_clock(case.asked_at):                  # what the system thinks "now" is
        out = system.answer(case.question, index=index)
    age = case.asked_at - max(d.source_ts for d in retrieved(out))
    return grade_constraints(out, case.expect, retrieved_age_s=age.total_seconds())

The freshness_slo field is the part that requires a decision nobody had made before the incident. Each ingested source needs a number saying how old its content may be before an answer derived from it must be qualified, and those numbers come from the people who own the sources rather than from the eval. The status page got 300 seconds, since its entire purpose is to describe the present. Pricing pages got 24 hours, because prices change on announced dates and a day-old answer is defensible. The on-call rota got 900 seconds, because directing someone to a person who went off shift at midnight is a real failure with a small blast radius. Reference documentation got no SLO at all, which is the correct answer for a source where "as of" is meaningless. Writing those four numbers down took one meeting and turned freshness from a vague concern into a set of thresholds the eval and the retrieval layer can both use.

The plumbing that makes the case runnable — a clock the system reads instead of the wall clock, and an index built from a fixture rather than from production — is about eighty lines and is reused immediately by two other classes. The permission cases need the same injection with a principal instead of a time, and the in-flight state cases need it with an order status. Teams that skip the freshness class often do so because the harness change looks disproportionate for fourteen cases, which undercounts it: the change is what makes any state outside the corpus expressible at all, and the second class that needs it costs nothing.

The grader here is checking constraints rather than comparing to a reference answer, because there are several acceptable behaviours and only one forbidden one. Asserting current operational state on the strength of a 3-hour-25-minute-old snapshot is the failure; saying "as of 06:15 no incidents were reported, check status.example.com for live state" is fine; calling a live status tool is better. Writing the constraint down is most of the work, and it is the part that has to be argued with the support team rather than generated.

One incident, fourteen cases#

A single incident produces one case, which is not enough to keep a fix in place, since a system can be patched to special-case the status page and still fail on every other near-live source. The class gets built by varying the incident along axes that were implicit in it.

Sixteen combinations were enumerated and fourteen survived review; two were dropped because a 2-minute-old on-call rota has no observably different correct answer from a fresh one, and a case with no distinguishable expected behaviour is noise. The generator did real work at this point, writing eight phrasings of each combination in the four registers from chapter 2's diversity scheme, which took nine minutes and about four dollars. That division of labour is the general one for incident-seeded classes: people supply the structure — which states, which clock, which constraint — and generation supplies volume within it.

The must_not constraint also has to be checked against the fourteen cases the way any grader does, and it failed its first calibration. An early version accepted "There are no reported issues as of my latest information", which qualifies nothing a customer can act on, since "my latest information" does not say when. Tightening the constraint to require an explicit timestamp or an explicit pointer to a live source dropped the system's score on the class from 6/14 to 2/14 before the fix, which is the more honest baseline.

Not the same problem as an aging set#

Topic 3.4 covered eval sets that drift out of date — a set built in March keeps scoring the March product, and the answer is refresh triggers and additive rebuilds. That is a different failure from this one, and conflating them leads to the wrong fix. An aging set is a set whose cases were correct when written and have quietly stopped being correct; regenerating from a current corpus repairs it. This set was not out of date on 14 March. Its 500 cases were all still correct, and it still could not represent the failure, because the missing thing is a dimension rather than a value. Regenerating from a corpus dumped that morning would have produced 500 fresh cases with exactly the same structure and exactly the same blind spot.

The freshness fix in production was separate and ordinary: shorten the status page ingestion to a 60-second poll, stamp every retrieved chunk with its source timestamp, and pass that into the prompt. Those took a day. The case class exists so that the behaviour is specified rather than remembered, and so that the next person who changes the retrieval layer finds out in CI rather than in a support queue. An eval class is the durable part of an incident response; the patch is the perishable part.

The eval class also does not replace a production signal, and the two catch different things. Emitting the age of the oldest retrieved chunk as a metric on every request, with an alert when the p95 for a source exceeds its SLO, would have fired at 06:20 on 14 March rather than at 12:30 when support escalated, because the condition is observable without anyone asking a question. That alert costs an afternoon and catches ingestion failures the eval cannot see, since the eval runs against fixtures where the pipeline is stubbed out. What the eval catches and the alert cannot is the model's behaviour when it is handed stale content — whether it qualifies, refuses or asserts — which is a property of the prompt and the model rather than of the pipeline, and which changes silently on every model upgrade. Both are worth having and neither substitutes for the other.

The wider family of cases with state outside the corpus#

Freshness is one instance of a general shape: cases whose correct answer depends on state the frozen corpus does not carry. Once the fixture format above exists, the neighbours are cheap to add, and they are all invisible to generation for the same reason.

Identity and permissions is the largest one. "Show me the audit log for last quarter" has a correct answer that depends on who is asking, and a corpus-derived case has no asker. The fixture needs a principal, and the class needs at least one pair of cases that differ only in the principal and have different correct answers. Plan or region gating is the same shape with a different variable, which is where chapter 1's retention example belongs — 30 days or 400 days depending on the workspace. In-flight workflow state matters for anything transactional: the right answer to "cancel my order" depends on whether the order has shipped, and a document corpus contains the policy rather than the state.

How many such dimensions are missing from any given set is not knowable from inside the set, and there is no procedure that enumerates them in advance. What is available is a discipline attached to incidents: every postmortem answers which case class would have caught this, whether that class exists, and if not, what its fixture needs to carry. Six months of that produces a class list that no amount of upfront design would have reached, and it grows in the direction production actually pushes.

By the time such a list has been through a few incidents, real labelled traffic has usually started accumulating alongside it, and the synthetic cases from cold start are still sitting in the suite reporting a number nobody has re-examined. What to do with them is not obvious, since the reflex answer of deleting them now that real data exists turns out to throw away the classes that were hardest to build.