Synthetic data generation for evaluation sets and cold start

Read these first

• Building a golden dataset

What you will know

• Generate synthetic examples for an eval set and state their limits

  Record the generator, prompts, sampling assumptions, and intended coverage so reviewers can identify artifacts and unsupported claims.

• Show synthetic data missing a real failure mode

  Use a production incident or held-out real example involving an unmodeled workflow, edge case, or interaction that the generator fails to reproduce.

• Use it for cold start and then replace it

  Synthetic cases should have an explicit retirement or down-weighting plan as representative labeled traffic accumulates.

• Measure distribution mismatch with a concrete comparison

  Compare synthetic and real data by relevant slices, labels, lengths, intents, and joint patterns; matching aggregate averages can still hide severe gaps.

• Refuse to validate a system solely on generated data

  Generated test cases share assumptions with the system that produced them, so an independent real-data holdout is needed to expose correlated blind spots.

Chapters

1. 1

   Synthetic data as an ingredient versus as the measuring instrument

   Team A generates 50,000 paraphrases of real labelled tickets to augment classifier training; Team B generates 500 question-answer pairs from the product docs and uses them as its eval set. Only one has a backstop. A's labels are inherited from reviewed tickets, and a held-out real eval reveals style contamination if it bites. B's synthetic data is the instrument, so an error there does not degrade a number you can see — it corrupts the number you would see it with.

   Read chapter
2. 2

   A synthetic set is built from a list of situations it must contain

   Before anything is generated, enumerate the case classes the set has to hold: a question the corpus cannot answer, two documents that contradict each other, a stale fact, a multi-intent message, an answer that requires combining two sources. Seed selection, prompting and diversity then work per class, and coverage is checked back against that list rather than discovered afterwards. A set produced as an undifferentiated batch of 500 questions cannot be extended by class when an incident exposes a gap, and cannot be retired by class when real traffic arrives — which is exactly what the last two chapters ask you to do.

   Read chapter
3. 3

   Every average can match while every joint pattern is absent

   Two thousand synthetic support questions match 2,000 real ones on mean length, intent mix and entities per message, and a teammate concludes the set is distributionally faithful. Those are marginals. Real messages carry two intents at once, paste raw error text and misspell product names, and each of those joint patterns can be entirely missing while every average lands on target. You will also see why a real-versus-synthetic classifier reaching 0.97 AUC is not the verdict people read it as.

   Read chapter
4. 4

   A generated set is only usable alongside its generator and its gaps

   When another team runs your synthetic eval set and reports the score, the examples are the least important thing you hand over. They need the generator and its version, the prompts and sampling settings, the seed material it drew on, a written statement of the coverage it was built to have, and the failure modes it is known not to contain. What a training-augmentation team must record differs from what an eval team must record, and the difference is not cosmetic.

   Read chapter
5. 5

   One model generating, answering and grading cannot fail visibly

   A RAG system built on model M, evaluated on 500 questions M generated and graded by M-as-judge, reports 94%: anything M consistently misreads is invisible at all three stages. This chapter separates the three couplings and shows why replacing the judge is often the least consequential of them, since documents written by M contain the invented details there was nothing real to get wrong. The same reasoning decides what you insist on when a vendor offers to generate your evaluation.

   Read chapter
6. 6

   A corpus frozen at generation time has no freshness dimension

   Users asking about an ongoing outage received confident answers assembled from a status-page snapshot three hours stale, and the synthetic eval set had never surfaced anything like it. The generator worked from a frozen copy of the corpus, so not one case in the set depends on time. The repair is not more cases of the same kind but a new case class whose correct answer turns on data freshness, seeded directly from the incident.

   Read chapter
7. 7

   Retire synthetic cases class by class, not all at once

   Eight months after cold start you hold 800 synthetic cases and 3,000 real labelled examples; the synthetic set says 91% and the real one says 74%. Report the 74 and do not delete the 800. The rare situations they were built for — a question the corpus cannot answer, two documents that contradict each other — are precisely what 900 reviewed items will not contain. Retire by class where real coverage is now thick, and treat regenerating from real patterns with suspicion. The real set stays the gate even at 300 cases, but there you need its confidence interval to know what it can resolve, and the synthetic set earns a second job as a screening signal for large breakages between real measurements.

   Read chapter