A synthetic set is built from a list of situations it must contain

The proposal that came out of Team B's retro was to generate 1,500 cases instead of 500, with a better prompt. It is the natural response to a set that missed things, and it would have produced a larger set that missed the same things, because the loop that made the first 500 — take a chunk, ask for a question and an answer, repeat — cannot produce a question the corpus does not answer. No prompt fixes that, since the generator is conditioned on a passage and asked to write something answerable from it. Three times as many draws from the same process gets you three times the coverage of what the process can reach and nothing at all outside it.

What the team did instead took an afternoon and produced no data. They wrote down the situations the set had to contain, one per line, with the sources of those lines named: four came from the retro's reading of real conversations, three from the support team's list of recurring escalations, two from the documentation writers who knew where the docs contradicted themselves, and two from an engineer who had built the retrieval layer and knew which queries it handled badly. Eleven classes. Only then did anyone write a generation prompt, and there ended up being eleven of them rather than one.

The list is the coverage claim#

An engineer testing a payments API does not generate five hundred random JSON bodies and report the pass rate. They enumerate the situations: amount zero, amount negative, amount above the per-transaction ceiling, currency mismatch between customer and merchant, idempotency key replayed with a different body, idempotency key replayed with the same body, expired card, card that requires 3DS. The count of tests is a consequence of that list; the list is what somebody reviews. Partition-based test design has worked this way since the 1970s, and the reason it survived is that "500 cases" is not a statement anyone can check, whereas "these eleven situations, at these counts" is.

Generated eval sets lost that discipline because generation makes volume cheap and volume feels like coverage. Five hundred cases produced in eleven minutes for about nine dollars of tokens looks more thorough than eleven hand-written ones, and on the axis of how many phrasings of an answerable question does this set contain it is. Coverage is a claim about situations rather than about rows. Here is the list Team B wrote, in the form it ended up living in the repository.

# eval/classes.yaml — the situations the set must contain
- id: single_hop
  desc: Answerable from one chunk, plainly stated.
  seed: random chunk
  target: 120
  verify: gold chunk in top-3 retrieval

- id: unanswerable
  desc: Plausible product question the corpus does not answer.
  seed: chunk deleted from the index after generation; competitor docs; roadmap items
  target: 60
  verify: no chunk scores above 0.62; expected behaviour is refusal

- id: contradiction
  desc: Two indexed documents give different answers to the same question.
  seed: doc pairs with cosine > 0.85 and an NLI contradiction; v1 pages never unpublished
  target: 40
  verify: both chunks retrieved in top-5; expected behaviour names the conflict

- id: stale_fact
  desc: Correct answer changed; the superseded statement is still in the index.
  seed: git history of docs/ — paragraphs edited in the last 12 months
  target: 40
  verify: old and new text both present; expected answer is the current one

- id: multi_hop
  desc: Requires combining two chunks that share an entity.
  seed: entity co-occurrence graph over chunks, path length 2
  target: 60
  verify: neither chunk alone contains the answer

- id: multi_intent
  desc: One message carrying two requests, the second one subordinate.
  seed: real conversations only; marked low_fidelity when synthesised
  target: 40
  verify: two distinct answerable spans, second not in the leading clause

Five further classes followed the same shape: conditional_answer for statements gated on plan or region, permission_scoped for questions whose answer depends on who is asking, identifier_missing, hostile_tone, and out_of_scope for questions about a different product entirely. The target numbers are not equal, because the classes are not equally important or equally hard to generate, and the counts came from arguing about which ones would hurt most in production.

Those arguments are short once the axis is named. A class gets a larger target when a failure in it is expensive and when the class has internal variety worth sampling, and a smaller one when the behaviour is close to binary. contradiction got 40 rather than 120 because there are only eleven real contradictions in the corpus and the interesting variation is in how the question is posed rather than in the underlying conflicts. single_hop got 120 despite being the easy class, because it is the regression floor: if plain answerable questions start failing, something structural has broken in retrieval and you want that to show up loudly. unanswerable got 60 because refusing correctly is the behaviour most likely to regress silently when someone tunes the retrieval threshold, and 60 gives enough resolution to notice a ten-point move in it.

Each class needs a different way of getting seeds#

The reason the eleven prompts could not be one prompt is that the classes differ in what they are conditioned on, and some of them cannot be conditioned on the corpus at all.

The unanswerable class is the clearest case, and the trick that produces it is worth stating precisely because it inverts the usual loop. Generate a question from a chunk in the ordinary way, then remove that chunk from the index the eval runs against. The question is now well-formed, plausible, in-domain, and unanswerable, and you know exactly why — which is a stronger position than trying to prompt a model into inventing questions it believes the docs do not cover, since a model asked for that produces either absurdities ("what is the CEO's shoe size") or questions the docs quietly do answer.

def build_unanswerable(chunks, n, index):
    picked = random.sample(chunks, n)
    cases = []
    for ch in picked:
        q = gen(UNANSWERABLE_PROMPT, chunk=ch)          # ordinary question generation
        index.remove(ch.id)                              # the corpus no longer contains the answer
        top = index.search(q, k=5)
        if top[0].score > 0.62:                          # a sibling chunk still answers it
            index.add(ch); continue                      # reject: not actually unanswerable
        cases.append(Case(question=q, expect="refuse_or_state_unknown",
                          klass="unanswerable", removed_chunk=ch.id))
    return cases

Roughly a third of attempts get rejected by that threshold check, because documentation repeats itself and the neighbouring chunk often covers the same ground. The rejection is the useful part: without it, a fifth of the class would be mislabelled cases where refusal is the wrong behaviour, and the eval would be penalising the system for answering correctly.

The contradiction class needs pairs, so the seeding runs over the corpus rather than over single chunks: embed everything, take pairs above 0.85 cosine, run a natural-language-inference model over them and keep the pairs it scores as contradictory, then have a human confirm about forty of them. Team B found eleven genuine contradictions this way, mostly old pages that were superseded and never unpublished, and generated three or four questions from each. The stale_fact class came out of git log on the docs directory, since every paragraph edited in the last year is a fact that used to be different, and the superseded version is sitting in the history in a form that can be re-indexed deliberately.

The multi_intent class is where honesty costs something. Topic 3.4 established that a model asked for messy input produces a stylised imitation of messiness, and that applies here in full: prompt for two intents in one message and you get two clean sentences joined by "also", which is not what real multi-intent traffic looks like. Team B's rule was that this class is seeded from real conversations wherever any exist, and when a synthesised case has to stand in, it carries fidelity: low in its metadata and its results are reported separately. A class you cannot generate well is still worth having in the list, marked as weak, because the list is also the record of what you know you are bad at.

The verify field in the class list is what stops all of this decaying. Each class carries an assertion that any case claiming membership must satisfy, and those assertions run in CI over the stored set rather than only at generation time, because a case's class membership is a property of the case and the corpus, and the corpus moves. When a documentation writer restored an old page in April, four unanswerable cases stopped being unanswerable, and the verification job failed with the case ids rather than the eval quietly starting to penalise correct answers. The assertions are cheap — a retrieval call and a threshold for unanswerable, a top-5 membership check for contradiction, a two-chunk necessity check for multi_hop implemented by running retrieval with each chunk removed in turn — and between them they took an afternoon to write and have caught something roughly every six weeks since.

Diversity is a per-class problem too#

Generating within a class does not automatically produce variety inside it. Running 120 single_hop cases as one batch at temperature 1.0 gave 120 questions of which the first four words fell into 31 distinct patterns, and deduplicating at 0.92 cosine removed 34 of them as near-identical. Mode collapse of this kind is well documented and it gets worse with batch size, since a model producing a long list tends toward its own centre.

The fix is to vary the conditioning rather than the temperature. Team B rotated three axes inside each class: the seed chunk (never reused within a class), a persona drawn from a list of six real user types recorded by the support team, and a phrasing register drawn from four options — terse, formal, narrative, and error-message-first. That produced 4.2 times more distinct opening patterns at the same count, and the dedupe pass then removed 7 of 120 rather than 34.

The token cost of the third row is 8% above the second, which is the kind of margin that never shows up in a budget conversation. The real cost is that the third row requires you to have written down six personas and four registers, and to defend them.

Where the list stops helping#

The classes are not a taxonomy of intents and they are not traffic shares. Topic 3.4 already covered stratifying an eval set around failure modes, and a case-class list is the same instinct applied before the data exists rather than after; the difference worth holding onto is that a stratum in a sampled set is discovered by slicing traffic you have, while a class here is asserted by people who think they know what will break. Assertion is weaker. Team B's eleven classes did not include anything about time, which is why the outage in chapter 6 walked straight through the set, and no amount of care about the eleven would have produced the twelfth.

There is a governance benefit that is easy to overlook and turns out to matter more than the technical one. classes.yaml is 60 lines of English and is the only part of the eval that a support lead, a documentation writer or a product manager can read and argue with, which means the people who know what goes wrong can contribute to the eval without touching a generation prompt. Team B's contradiction class exists because a documentation writer said, in a meeting about something else, that half the old integration pages were never unpublished. Reviewing 500 generated question-answer pairs would never have surfaced that, since nobody outside the team can hold 500 cases in mind, and the value of the list is partly that it compresses the set into something reviewable by the people who have the relevant knowledge.

The list also does nothing about fidelity within a class. Sixty unanswerable cases with verified absence of a supporting chunk still contain sixty questions written in the generator's voice, and if the system fails on unanswerable questions specifically when they arrive misspelled and half-punctuated, the class will report a clean pass. Enumerating situations is orthogonal to whether the instances of each situation resemble production, and the next chapter is about how thoroughly the usual checks miss that.

What the list buys is that both remaining moves in this topic become possible. When an incident exposes a situation the set never contained, the response is a new class with its own seeds and its own verification, added beside the existing ones without touching them, rather than a vague resolution to regenerate everything with a better prompt. When real labelled traffic starts arriving, the question "can we drop these synthetic cases yet" is answerable per class — the single_hop cases are the first to go, because real traffic is full of single-hop questions, while the eleven contradictions will not appear in a thousand reviewed conversations and stay. A batch of 500 undifferentiated questions supports neither operation, and that is the practical cost of skipping the afternoon.