A generated set is only usable alongside its generator and its gaps

The set got shared the way these things always get shared. Somebody on the platform team posted a link to docs-eval-v2.jsonl in a channel with the message "500 cases for RAG over product docs, we score 0.92, feel free to run it". Three teams picked it up over the next month. The billing assistant team scored 0.88 and decided their retrieval was adequate. The onboarding bot team scored 0.71, spent two weeks tuning chunk size and reranking, got to 0.83, and shipped. A fourth team put it in CI as a merge gate.

Every one of those decisions rests on facts that were not in the file. The 500 cases had been generated from a snapshot of docs/ taken on 4 February, which meant 63 of the reference answers described behaviour that had changed by the time the onboarding team ran them — their 0.71 was partly a measurement of documentation drift. The billing team's corpus does not overlap the product docs at all, so their 0.88 was a score on questions about a different corpus, answerable only when the retriever happened to find something adjacent. And nobody outside the originating team knew that the set contained no questions requiring two documents beyond the 60 built for it, no time-dependent cases, and 40 cases explicitly marked as low-fidelity.

The fourth team's use is the one that ages worst. A merge gate is a promise that the number will keep meaning the same thing across hundreds of runs by people who were not in the channel and will never read the original message, and by month three the person who added the gate had changed teams. Somebody eventually raised the threshold from 0.85 to 0.90 because the score had been sitting at 0.93 for weeks, which made the gate fire on a change that improved real-traffic behaviour and slightly reordered retrieval for the generated questions. Nobody in that argument had any way to establish what the gate was supposed to be protecting.

The examples were the least valuable part of the handover. What the recipients needed was everything that determined what those examples could and could not fail on.

The dependency graph behind a reported score#

A score computed on a generated eval set is a function of more inputs than the examples, and most of the inputs are usually undeclared.

Change any node and the number changes. The recipients of that file saw D and N and inferred a relationship between them that holds only when the other five nodes are the ones you had. This is the same reason a build artifact travels with a lockfile: the bytes do not tell you what produced them, and a dependency resolved differently produces a binary that looks identical and behaves differently. The practice of writing this down for datasets has a name and a lineage — Datasheets for Datasets proposed it for corpora in 2018, model cards did the equivalent for trained models, and software supply chains converged on the same idea under the name SBOM. Generated eval sets need it more acutely than any of those, because a corpus can at least be inspected by reading it, while a coverage gap is invisible by construction: the cases that would reveal it are the ones that are not there.

What actually travels#

The manifest that shipped with v3 of the set is a single YAML file next to the JSONL, and it is short enough to read in a meeting.

set: docs-eval
version: 3
built: 2026-03-02
owner: platform-search@   # a person answers questions about this

generation:
  model: gen-v3-2026-01
  decoding: {temperature: 1.0, top_p: 0.95, seed: 20260302}
  prompts:                      # hashes, so a silent edit is detectable
    single_hop: sha256:4f2ab1c9…
    unanswerable: sha256:9c07de31…
  cost: {tokens_in: 1.9M, tokens_out: 410k, usd: 11.40}

seed_material:
  corpus: git@docs.git#e91b774            # exact commit, not "latest"
  corpus_date: 2026-02-28
  real_seeds: support-export-2026-02.jsonl (sha256:1de4…), 84 conversations
  excluded: nothing under docs/internal/

coverage:
  classes: eval/classes.yaml@v3
  produced: {single_hop: 120, unanswerable: 58, contradiction: 40,
             stale_fact: 40, multi_hop: 60, multi_intent: 40, …}
  verified_by: eval/verify_classes.py   # each case passes its class assertion

known_absences:
  - No case has a time dimension; a correct answer never depends on when it is asked.
  - No case depends on the identity or permissions of the asker.
  - multi_intent cases are synthesised, fidelity: low; report separately.
  - Non-English input absent (0.4% of traffic, out of scope by decision).

fidelity_check:
  method: joint-cell comparison, 8 features, vs 2,000 real questions 2026-02
  result: 19 cells >=0.5% of real are >10x under-represented, covering 31.4%
  discriminator_auc: 0.97   # separability, not validity — see below

validity:
  expires: 2026-06-01        # rebuild seeds if the corpus commit is >90 days old
  not_valid_for: corpora other than product docs; systems trained on this corpus

Each of those blocks exists because of something that went wrong without it. The corpus commit is there because of the 63 drifted reference answers; with e91b774 recorded, a recipient can diff their corpus against it and get a count of affected cases in a minute. The prompt hashes are there because a teammate improved the unanswerable prompt in place, three weeks of scores moved by four points, and the change was invisible in git history since the prompt lived in a notebook. The decoding seed is there so a regeneration produces a comparable set rather than a new one. The not_valid_for line is there because of the billing team.

The validity block is the least conventional field and the one that changes behaviour most. An expiry date on an eval set looks strange, since a test suite is normally something you keep, and the justification is that a generated set's reference answers are derived from a corpus that keeps moving while the answers do not. Ninety days was chosen by measuring: over the previous year, docs/ accumulated about 110 changed paragraphs a month, and mapping those to cases showed roughly 4% of the set going stale per month, so at ninety days something like one case in eight is describing behaviour that has changed. The expiry does not delete anything; it makes the CI job print a warning next to the score and opens a ticket to re-seed. Teams that skip this end up with the situation from the opening, where a score drifts downward for reasons that have nothing to do with the system.

Where the manifest lives matters as much as what it contains. Next to the data, in the same repository, versioned together, and referenced by every report the set produces — a score emitted without a pointer to the manifest version that produced it is the same failure at a smaller scale. The reporting job writes docs-eval v3 (manifest 2026-03-02) — 0.92, known absences: 4 into the same line as the number, and the count of declared absences next to the score does more to calibrate a reader than any amount of documentation filed elsewhere.

The known_absences block is the one that does the most work and is the hardest to write, because it requires stating things you would rather not have on record. It is also the only part of the manifest that stands in for a measurement. Chapter 1 ended on that point: a training-augmentation team has an independent number that will move when their data is bad, and an eval team has no such number, so a written statement of what the set cannot fail on is the only artifact a reader can check the score against. Someone reading 0.92 next to no case has a time dimension will not conclude that the assistant handles ongoing incidents well.

The two records are not the same document#

Both kinds of generated data need provenance, and the field lists diverge in a way that follows from what each record protects.

Team A's critical fields are mechanical and mostly enforceable in code: a CI check can assert that no row in the holdout shares a seed_id with any training row, and the ratio is a number in a config. Team B's critical fields are claims in prose, which nothing can verify automatically, and that is uncomfortable but it is the honest situation. The nearest thing to enforcement is verified_by, where each case must pass its class's assertion from chapter 2, so at least the coverage counts are not aspirational.

Retrofitting one onto a set you inherited#

Most teams meet this practice while holding a set that has no manifest and whose author has moved on, and the reconstruction is worth doing even though it will be incomplete. Four fields are usually recoverable. The generator model can be inferred from the commit that added the generation script, or from the API logs for the week the file was created, and even "one of gen-v2 or gen-v3, generated in the week of 4 February" is more useful than nothing. The corpus snapshot can be approximated from the file's creation date and the docs repository history, which gives you a commit range and therefore a diff. The coverage claim can be reconstructed by reading a sample of the cases and labelling them into classes — a hundred cases at fifteen seconds each is half an hour, and it converts an undifferentiated set into a table of counts.

The fourth is the fidelity comparison, which does not need any historical information at all, since it only requires the set and some real traffic; chapter 3's cell comparison can be run today against a set generated two years ago. What is genuinely unrecoverable is the prompts and the sampling settings, and their absence has a practical consequence worth writing into the manifest as a limitation: the set cannot be extended in a way that is comparable to its existing cases, so any new cases form a separate cohort and get reported separately. A retrofitted manifest with four filled fields, two marked unknown and an explicit note that the set cannot be extended comparably is a genuinely useful document, and writing it takes an afternoon against a set that half the organisation is already running.

What the manifest is not#

It is not a data card written for compliance, filed once and never read. The test of whether the practice is working is whether anybody has ever changed a decision because of it, and the observable version of that in this team was the onboarding bot's second run, where somebody diffed their corpus against e91b774, found 63 affected cases, excluded them, and reported 0.79 instead of 0.71 with a note about the exclusion.

It is also not a substitute for the independent real holdout. A perfectly documented synthetic set with an honest absences list still cannot fail on the situations it does not contain, and the manifest's job is to stop a reader from believing otherwise. Topic 3.4 covered what a case row must freeze — input, retrieval context, tool fixtures, ground truth — for a case to be reproducible at all, and none of that is replaced here; the manifest sits one level up, describing the set rather than the row.

There is one field in the manifest that turns out to be load-bearing in a way the others are not, and the version above under-specifies it. generation.model says gen-v3-2026-01. The system under test in the originating team was also built on gen-v3-2026-01, and so was the judge model in the grading rubric. Three of the nodes in the dependency graph are the same model, and the next chapter is about what a suite can and cannot detect when that is true.