A version label means nothing until something restores it

The request came out of a customer escalation: reproduce the March result on support-golden that the team had cited when they signed off on the retrieval change. The dataset hash was in the run record, the manifest resolved on the first try, and the restore job died forty-one seconds later:

[00:00:03] resolved manifest sha256:71bb3e9c… (2000 entries)
[00:00:04] fetching 2000 blobs, concurrency 32
[00:00:41] ERROR NoSuchKey: evals/blobs/sha256/c4/c41f0a9e…
[00:00:41] ERROR NoSuchKey: evals/blobs/sha256/7b/7be2d130…
[00:00:41]   … 10 more
[00:00:41] restore failed: 12/2000 blobs unresolvable
                 (aborted before scoring)

Twelve blobs, out of two thousand, referenced by a manifest that was perfectly intact. A lifecycle rule on the blob prefix with a 270-day expiry had removed them; the twelve were the oldest examples in the set, added when it was first assembled, and untouched since. The versioning was correct in every respect that a code review would check. It described a set of objects that no longer existed, and nothing had noticed for nine months because nothing had ever tried to read them.

This is the ordinary first result of a first restore, and it is worth dwelling on why: the failure is in the storage layer, not the version scheme, and no amount of hashing prevents it. A content hash is a claim about what the bytes were. It says nothing about whether the bytes are still there.

A backup nobody restores is not a backup#

Operations teams settled this argument decades ago. A nightly pg_dump, the standard export of a whole Postgres database to a file, that has never been loaded back into a running instance is a file of unknown value, and the discipline that emerged is the restore drill: on a schedule, take a real backup, restore it into a scratch environment, run assertions against it, and treat a failed drill as a production incident. The reason the practice stuck is that backup pipelines fail silently in ways their own logs report as success — the dump ran, exit code zero, file written, and it happens to be truncated, or encrypted with a key nobody has, or missing the extension the schema depends on.

Versioned evaluation data has the identical failure profile with one addition. A database restore either loads or it does not, so the drill's assertion is close to binary. An eval restore can complete, score every example, and produce a number that is quietly wrong, because the examples came back but the thing that scored them did not come back the same. That extra failure mode is why the drill needs assertions past "it ran."

The failures fall into three families, and they surface in a predictable order because each is a precondition for the next.

The middle family is the one that survives longest undetected, because most of its members are not part of the dataset at all. A RAG faithfulness eval — retrieval-augmented generation, where the system searches a corpus of documents at question time and answers from the passages it gets back — scores answers that were produced against a retrieval corpus, and if the run record pinned the dataset but described the corpus as prod-index, restoring the dataset restores half the experiment. The corpus has been re-embedded twice since March with a newer embedding model, meaning the model that turns each passage into the vector the search runs over has been swapped, its chunk boundaries moved when the chunker's default overlap changed, and the retrieved context for every example is different. Every example is present, every hash matches, and the number is about a different system.

The third family looks like an implementation detail and is the most embarrassing to discover late. A harness that resolves a version and then does random.sample(examples, 500) for the nightly run has a dataset version that pins two thousand examples and a run that scored five hundred unrecorded ones. If the seed comes from the wall clock, no amount of restoring gets that run back, and the recorded score is not reproducible even in principle.

Make the restore a job, not an intention#

A restore that happens when someone remembers is a restore that happens after the challenge, which is the one time it is guaranteed to be inconvenient. Schedule it, pick versions by age so that the retention cliffs get exercised, and make each assertion explicit and separately reportable so a failure names its own family.

Choosing which run to restore matters more than it looks. Sampling uniformly from the run store means almost every drill picks a recent run, and recent runs are the ones least likely to fail, since nothing has had time to expire, rebuild, or be deprecated underneath them. Stratifying by age puts a drill on each side of the retention cliffs that actually exist in your infrastructure — a 30-day run to catch outright breakage in the restore path, a 90-day run to catch anything tied to a quarterly index rebuild, and a 270-day run to catch lifecycle expiry and provider deprecations. A second stratum worth adding is provenance: runs that were cited in a decision, linked from a design document, or quoted to a customer are the ones whose failure is expensive, and they are worth drilling on their own schedule regardless of age.

The gates are ordered so that a failure is diagnostic on its own. Checking digests before membership means a corrupt blob does not present as membership drift; checking membership before dependencies means a set that quietly grew does not present as a score change. Writing the same checks as a single boolean at the end produces a red build and a two-hour investigation.

def restore_and_verify(run: dict, store: BlobStore, bands: BandStore) -> Report:
    manifest = store.get_manifest(run["dataset_hash"])          # gate 1

    examples = []
    for entry in manifest["examples"]:                          # gate 2
        blob = store.get_blob(entry["sha256"])                  # raises Missing
        assert sha256(blob) == entry["sha256"], entry["id"]
        examples.append(decode(blob))

    assert manifest_hash(rebuild_manifest(examples, manifest["preprocessing"])) \
        == run["dataset_hash"]                                  # gate 3

    deps = resolve_pinned(run)                                  # gate 4
    result = score(examples, deps)                              # gate 5
    assert result.n_scored == run["example_count"]
    assert result.n_errors == 0

    band = bands.for_config(run["evaluator"]["config_hash"])    # gate 6
    delta = abs(result.faithfulness - run["result"]["faithfulness"])
    return Report(passed=delta <= band, delta=delta, band=band,
                  restored=result.faithfulness,
                  recorded=run["result"]["faithfulness"])

bands.for_config is the piece that makes the last gate mean anything. The tolerance is looked up by the evaluator configuration hash the original run used, so a run scored under a rubric and judge that have since been replaced is compared against the band measured for that configuration, not the current one. A band stored as a single global constant drifts out of correspondence with the pipeline it describes within a couple of releases.

The cost is not an argument against doing it. One end-to-end restore of a 2,000-example set costs roughly $30 and seventeen minutes, so a weekly drill across three age strata is about $4,700 a year — comparable to a single engineer-day spent reconstructing one disputed result by hand, and it buys the guarantee rather than a one-off answer.

0.83 against a recorded 0.85#

The first restore of a nine-month-old version passes every structural gate and comes back at 0.83 where the record says 0.85. The reflex answer is that evaluation has inherent variance, LLM judges are nondeterministic, two points is well inside normal noise, and the restore passes. Chapter 3 has already supplied what that answer is missing: for this pipeline, the end-to-end run-to-run standard deviation on this set is 0.0089, so the band at two standard deviations is 0.018. The observed delta is 0.020. It sits outside the band, and the restore fails.

The arithmetic is the point rather than the verdict. A different pipeline with a noisier judge and a band of 0.025 would pass the same delta, and it would pass for a stated reason rather than by assertion. Without a measured band, "inside normal noise" is a claim about the world made by someone who has not measured the world, and it has the property that it can absorb any delta anyone is uncomfortable investigating. The characteristic way a versioning system stops meaning anything is not a dramatic failure; it is a sequence of two-point discrepancies each individually waved through, until the recorded history and the reproducible history have separated by fifteen points and nobody can say when.

A failed drill needs somewhere to go, or it becomes a red check that people learn to skip. Treat it the way a failed backup restore is treated: it has an owner, it is triaged the same day, and its outcome is written back into the run store rather than into a chat thread. The write-back is the part teams omit and the part that compounds. When gate 2 fails because twelve blobs expired, the runs that referenced that manifest are marked unreproducible with the reason and the date, so that anyone who later cites one of those numbers sees immediately that it cannot be re-derived. When gate 6 fails, the affected run is marked score_unverified with the observed delta and the band it was compared against. Neither mark deletes the historical number, because the number was really produced and the record is append-only, and both stop it from being quoted as though it still stood.

A failing score gate is also not a verdict about the data. It is an instruction to find out which of the surviving explanations holds: an unpinned dependency that gate 4 could not check because it was never recorded, a judge model whose provider silently updated behaviour behind a name the record trusted, a scoring path with an ordering dependency, or a genuine bug in the original run. All of those are findable; none of them are findable from a dashboard.

What a green restore does not certify#

It says the run is recoverable, not that it was right. A dataset with 200 mislabelled examples restores perfectly and reproduces its wrong number to four decimal places, because reproducibility and correctness are orthogonal properties and this machinery only addresses the first. It also says nothing about whether the version is still relevant: a set assembled before the product added a whole category of user intent will restore, score, and describe a system that no longer exists.

Two errors that cancel will also pass. If the restored retrieval corpus is slightly better and the restored judge is slightly stricter, the aggregate can land inside the band while both halves have moved, and only a per-example comparison of the kind chapter 3 set up would show the churn underneath a stable mean. Adding a second assertion on the paired per-example agreement rate — the fraction of examples whose verdict is unchanged — catches most of that, and it is the cheapest upgrade to a restore test that already exists.

The gate the restore cannot pass at all is the one where a pinned dependency is simply gone. A judge model the provider has retired cannot be resolved by any amount of hashing, and gate 4 fails with dependency_unavailable and no remedy. That failure is unavoidable, it is common on any timescale longer than a few quarters, and it is the situation the next chapter has to work in: attributing a score change between two releases when several things moved at once and at least one of them can no longer be put back.