Synthetic data as an ingredient versus as the measuring instrument

The quarterly model review had two teams presenting back to back, and both of them opened with almost the same sentence: we lean heavily on synthetic data. Team A owns the ticket triage classifier. They have 5,200 human-labelled support tickets spread over 14 intents, each ticket read by two people, and they had spent the quarter generating ten paraphrases of every training ticket to get 42,000 additional rows. Team B owns the documentation assistant, a retrieval system over 1,800 chunks of product docs. They had no labelled traffic at all, so they prompted a model to read the chunks and write 500 question-and-answer pairs, and that set had become the release gate: ship at 85% or above.

Both slides showed a number going up. Team A's macro-F1 had moved from 0.81 to 0.84. Team B's suite had gone from 0.89 to 0.92 across two prompt revisions. The two numbers do not mean the same kind of thing, and the difference is not about how carefully each team generated its data. It is about where the generated text sits relative to the measurement. For Team A, generated text is an ingredient in the artifact being measured, and the measurement happens somewhere else, on tickets that real customers wrote and two humans read. For Team B, the generated text is the measurement. An error in Team A's synthetic data degrades a number they can see. An error in Team B's synthetic data corrupts the number they would use to see it.

What Team A can find out and Team B cannot#

Here is the shape of Team A's pipeline, reduced to the lines that carry the argument.

seeds = load_reviewed_tickets()                    # 5,200 rows, 14 intents, 2 reviewers each
train_seeds, holdout = group_split(seeds, holdout_n=1000, group_key="ticket_id")
para = paraphrase(train_seeds, n_per_seed=10, model="gen-v3", temperature=0.9)
clf = fit(train_seeds + para)                      # 4,200 real + 42,000 generated
print(macro_f1(clf, holdout))                      # 1,000 real tickets, no paraphrase of them anywhere in train

The last line is the whole difference. It runs against a thousand tickets that no generator touched, and it will move if the generated rows hurt. Team A found out that it does. The paraphrase model, asked for ten variants of each ticket, produced text that was cleaner and more uniform than the tickets it came from, and at ten variants per seed the classifier started preferring that register.

The drop at ten was concentrated in two intents. Reading the paraphrases for billing_dispute explained it: a seed reading "I was NOT charged twice, the second one is my wife's card, please stop the refund" had produced variants where the negation quietly dropped out, and those variants carry the original's label while describing the opposite situation. That is label noise manufactured by the generator, which is a genuine failure and exactly the kind people warn about. It cost Team A five points of macro-F1 for one sprint, and then it cost them nothing, because the holdout number moved, somebody went and read rows, and the ratio came down to three. The failure was expensive in the way a red test is expensive.

Team B's pipeline has no equivalent of that last line.

cases = generate_qa(chunks, n=500, model="gen-v3", temperature=1.0)
results = [grade(rag_answer(c.question), c.reference) for c in cases]
score = mean(results)                              # 0.92
assert score >= 0.85, "release gate"

Nothing downstream of score disagrees with it. If the generator systematically wrote questions that the retrieval system happens to be good at, score goes up and stays up, and there is no second number anywhere in the system whose job is to contradict it. Team B eventually got a contradicting number the way most teams do, which is by shipping. In the first month the assistant handled about 4,000 real questions; a sample of 120 was reviewed by two support engineers, and 61% were judged good. The same system, the same weights, 0.92 on one collection and 0.61 on another — topic 3.4 established why those two collections differ, so the point here is narrower. Team B could not have known the size of the gap, or that a gap existed, from anything inside their pipeline.

The mechanism was legible once someone looked. Of the 500 generated questions, 86% were answerable from a single chunk, because a generator handed one chunk at a time writes questions that chunk answers. In the real sample, 40% of the questions either needed two documents or could not be answered from the corpus at all, and those were where the assistant failed. A calibration analogy is close enough to be useful here: a scale calibrated against a reference weight of unknown mass will give you readings that are precise, repeatable, internally consistent and wrong, and no amount of weighing reveals it, because every measurement passes through the same error. The closer software analogy is a test suite generated by reading the implementation — every test passes on the first run, and the suite's pass rate is a statement about the generator's reading comprehension rather than about the code.

Team A is not safe, only observable#

None of this makes training augmentation harmless. The negation-dropping paraphrases were real label noise. Style contamination is real; there are published results where a classifier trained mostly on generated text degrades on the natural inputs it will actually see. And there is a failure mode in Team A's code that would be silent if the second line were written slightly differently. group_split runs before paraphrase and splits on ticket_id, so no holdout ticket contributes a paraphrase to training. Reverse those two lines — generate first, split the pooled rows at random — and roughly 19% of the holdout becomes paraphrases of training rows, the holdout number climbs by several points, and the backstop is gone while continuing to print a number. Team A's protection is not that their data is real. It is that a real, independent number exists and is wired to the decision, and that wiring can be broken by an ordinary refactor.

What the asymmetry does say is that the two teams' errors have different blast radii. Team A can be wrong about their generated data and find out from a measurement they already run. Team B can be wrong about theirs and have every downstream signal agree with them.

The line between the two uses is not always as clean as these two teams make it look, and the case that blurs it is worth recognising early because it is common. A team that generates preference pairs to fine-tune a grading model has produced training data by every structural test — the generated rows are an ingredient, the trained artifact is the grader, the ratio is a dial. What they have also done is manufacture a measuring instrument out of generated text, so the errors in those pairs propagate into every score the grader ever emits, and the backstop has to be a set of human-adjudicated grading decisions rather than a held-out eval of the downstream system. The question that decides which regime you are in is whether an error in the generated data ends up in something that produces numbers or in something that produces behaviour, and a single pipeline can do both.

Why the wrongness is correlated rather than random#

There is a sharper version of Team B's problem than "the questions were too easy". The generator and the system under test are not independent sources of error. Both read the same corpus, and where the corpus is ambiguous, both resolve the ambiguity the same way.

One of Team B's chunks contained the line "Retention: audit logs are kept for 30 days unless the workspace is on the Enterprise plan." The generator produced this case:

{
  "question": "How long are audit logs retained?",
  "reference": "Audit logs are retained for 30 days.",
  "source_chunk": "docs/security/retention.md#L12"
}

The conditional disappeared. The assistant, built on the same model family, read the same sentence the same way and answered "30 days", so the case passed, and it passed on every release for four months. Enterprise customers, whose logs are kept for 400 days, were being told 30. The eval set could not fail on this, because the failure is the shared reading, and a case authored by one of the two parties to a misunderstanding cannot arbitrate it.

There is a tempting objection to this, which is that a careful human reviewing the generated cases would have caught the missing conditional. Team B did review them, in the sense that two engineers read a sample of 40 cases and judged them reasonable. Reviewing a generated case for correctness means checking each reference answer against the source passage, which takes two to four minutes for a documentation question and is the same work as writing the case by hand; sampling 8% of them and skimming is what actually happens, and skimming is exactly the mode in which a plausible answer to a plausible question passes. The retention case is not a conspicuous error. It reads as a correct summary of the sentence it came from, unless you notice the subordinate clause that both the generator and the reader's eye slid past.

This is the reason to refuse to validate a system solely on generated data, and it is a stronger reason than distribution mismatch. Mismatch means the generated cases are unrepresentative, which is a statistical problem you can partly characterise. Correlated blind spots mean the generated cases are systematically absent exactly where the system is systematically wrong, which no amount of generating more of them addresses. The remedy is an independent holdout of real cases — small is fine, and small is usually what is available. A hundred and fifty adjudicated real conversations would have surfaced the retention error, because at least one Enterprise customer asks about log retention every few weeks and the answer they were given is on record.

What each team has to write down#

Both teams owe a record of how their data was produced, and the two records are not the same document, because they are protecting different things.

The last row is the one worth arguing about in a review. Team A's critical record is mechanical: which paraphrase came from which seed, so that grouping survives every future re-split, plus the generator version and the ratio, so the F1 movement can be attributed when it happens. That record exists to protect a measurement they already have. Team B's critical record is a written claim: what situations the 500 cases were built to contain, and which known failure modes they do not contain. That record exists because there is no measurement to protect, and a stated coverage claim is the only thing another engineer can check the number against. Chapter 4 works out what such a record has to include before the set can safely leave the team that made it.

Team B's harder problem shows up first, though, and it is upstream of any record. When the retro asked what the coverage claim for those 500 cases was, there was no answer to write down, because the set had been produced as one undifferentiated batch: a loop over 1,800 chunks, temperature 1.0, stop at 500. The gap in it — questions needing two documents, questions the corpus cannot answer — could only be described afterwards, by reading real traffic and noticing what the set did not have. A set built as a batch can only be audited that way. The alternative is to decide what situations the set must contain before generating anything, which changes how the generation is run, and is where the next chapter starts.