Every average can match while every joint pattern is absent

The comparison arrived as a table in a pull request description, and it was a good table. Somebody had taken 2,000 synthetic support questions and 2,000 real ones from the same month and lined up the summary statistics, and the review comment underneath said the synthetic set was distributionally faithful and could stand in for the real one while the real one was being re-labelled.

Everything in that table is a marginal: a statistic computed over one dimension at a time, with every other dimension summed out. The set that produced it can be a perfect match on all eight rows and still contain none of the message shapes that break the system, because what breaks systems is usually a combination.

Eight matching numbers constrain almost nothing#

Take eight binary properties of a support message: carries two intents, includes a pasted stack trace or error string, misspells a product name, omits the identifier needed to answer, is a correction of an earlier turn, mixes two languages, is written in all caps, exceeds 600 characters. Any message is a point in a space of 2⁸ = 256 cells. Matching the eight marginals fixes eight numbers. The joint distribution over those cells has 255 free parameters, so agreement on the marginals leaves 247 degrees of freedom completely unconstrained, and the generator is free to place all of its mass in a handful of cells as long as the column sums come out right.

That is not a theoretical concern; it is the ordinary behaviour of a generator. Here is the same pair of sets compared on cells instead of columns.

The first three rows are the marginals, and they match. The next four are cells, and they are empty. The generator produces messages that have one interesting property at a time, because each property was requested by a prompt and prompts were applied one at a time. Real users combine properties, since a person confused enough to misspell the product is also the person pasting a log they do not understand, and the correlations between these properties are the reason the hard cases are hard.

The check is a dozen lines and worth having in the repository next to the generator.

FEATURES = ["two_intent", "pasted_error", "misspelled_product",
            "missing_id", "correction", "mixed_lang", "allcaps", "long"]

def cells(df):
    return df[FEATURES].astype(int).astype(str).agg("".join, axis=1).value_counts(normalize=True)

real_cells, synth_cells = cells(real), cells(synth)
missing = [(c, p) for c, p in real_cells.items()
           if p >= 0.005 and synth_cells.get(c, 0) < p / 10]
print(f"{len(missing)} cells at >=0.5% of real are >10x under-represented")
print(f"they cover {sum(p for _, p in missing):.1%} of real traffic")

On the two sets above that printed 19 cells at >=0.5% of real are >10x under-represented / they cover 31.4% of real traffic. Thirty-one percent of real messages had a combination of properties that the synthetic set effectively did not contain, while every summary statistic agreed within a few points. Feature detectors do not have to be sophisticated for this to work — two_intent was a regex for a second question mark or a sentence starting with "also" or "and can you", and pasted_error was a look for three or more consecutive lines with no sentence-ending punctuation, or the substring Traceback, or a hex string of length 8 or more. Cheap detectors under-count in both sets, roughly equally, and the ratio survives.

The two thresholds in that code are doing real work and are worth choosing deliberately. With 2,000 real messages, a cell at 0.5% holds about ten examples, and below that the comparison is reporting sampling noise: a cell with three real instances and zero synthetic ones is not evidence of anything. The 10× ratio rather than a fixed difference is there because the interesting failure is a cell that is an order of magnitude rarer in synthetic data, not one that is 30% rarer, and at these sample sizes a 30% difference is inside the noise anyway. Sets of 500 a side, which is the usual situation early on, can only support a threshold of about 2% per cell, which means the comparison sees perhaps eight cells instead of nineteen, and that is a real limitation rather than a reason not to run it.

There is a cost to this comparison that the marginal table does not have: you must decide in advance which properties matter, and you can only find missing cells over properties you thought to detect. Chapter 2's class list is the natural source of the feature vocabulary, and the incident-driven additions in chapter 6 are how the vocabulary grows. Nothing here finds a property nobody has named.

The output is a decision list rather than a verdict. Each under-represented cell gets one of three dispositions: it becomes a new case class with its own seeds, which is what happened to two intents and missing identifier because 11.2% of real traffic is too much to leave unmeasured; it gets declined explicitly and written into the known-absences list, which is what happened to all-caps corrections at 3.1%, since the behaviour there is the same as for ordinary corrections and the register does not change what a good answer looks like; or it gets covered by perturbation of existing cases, which suits the surface-form cells like misspelled product names, where the situation is already in the set and only the input is too clean. Nineteen cells produced three new classes, eleven declines and five perturbation rules, and the declines are as valuable as the additions because they are the ones a reader of the score needs to know about.

What a 0.97 AUC discriminator does and does not say#

Topic 3.4 introduced the real-versus-synthetic classifier as a cheap way to put a number on the gap between two collections. Run it on these sets and it reaches 0.97 AUC, which people read as a verdict: the two are trivially distinguishable, so the synthetic set is worthless. That reading is wrong in a way that leads somewhere expensive.

Look at what the classifier is using. On a logistic regression over character n-grams, the highest-weight features pointing at "synthetic" were the em dash, the word "Additionally", a trailing question mark preceded by a complete sentence, and — carrying more weight than any single token — the absence of typos. The features pointing at "real" were the newline character, lowercase sentence starts, and digit runs of length 6 to 9. Those are stylistic fingerprints. They tell you the text was written by a model, which you already knew, since you generated it.

The distinction that matters is between two questions that separability conflates. Whether a case looks like production is a question about surface form. Whether a case exercises the behaviour you care about is a question about the situation it puts the system in, and a case can be obviously machine-written while putting the retrieval layer in exactly the position where it fails. A well-formed, em-dashed, perfectly spelled question about a fact that two indexed documents disagree on is a real test of contradiction handling. The discriminator will flag it with 0.99 confidence and the flag says nothing about the test.

An ablation makes the point concrete. Strip the features that only carry authorship — normalise dashes and quotes, lowercase everything, remove all whitespace structure, correct the spelling in both sets — and refit. The AUC falls from 0.97 to 0.74, and the features that remain are about content: the real set still has digit runs that look like order ids and error codes, and the synthetic set still has a characteristic distribution of question openings. The 0.23 that disappeared was typography. The 0.24 above chance that remains is the part that overlaps with the cell table above, and it is the only part worth acting on, but no single AUC figure tells you which of the two you are looking at.

The expensive move is to treat the AUC as a target. Once "get the discriminator below 0.7" is on a roadmap, the generator gets tuned — inject typos at the empirical rate, strip em dashes, lowercase the first letter 12% of the time, add newlines — and the AUC comes down while the cell table above does not move at all, because none of those edits create a message that carries two intents and omits the identifier. You have optimised for fooling a classifier, which is a different objective from containing the situations where the system fails, and the two are only loosely related. It is Goodhart's law in its plainest form, and it is worse than a wasted sprint, because the falling AUC is read as evidence of improving fidelity.

The discriminator earns its keep used as a locator rather than a gate. Score the real examples, sort by how confidently the model calls them real, and read the top fifty by hand. Those are the real messages least like anything in the synthetic set, and they are a free draft of the next case classes. On this data, 22 of the top 50 contained pasted error text, 14 were mid-conversation corrections with no restatement of the original request, and 9 were in a mix of two languages. Two of those became classes; the third was deliberately declined, since the product had 0.4% non-English traffic and nobody was going to be paid to handle it.

p_real = clf.predict_proba(featurize(real))[:, 1]
for text in real.text[np.argsort(-p_real)[:50]]:
    print(text[:200])   # read these; they are the gaps, in the users' own words

Which set decides whether the change ships#

None of this is an argument for measuring on the synthetic set. When a model change is up for release and you hold 5,000 synthetic cases and 5,000 real ones, the real set is the gate, and the synthetic set reports alongside it per class. The reason is chapter 1's asymmetry rather than anything in this chapter's tables: the real set can fail for reasons neither the generator nor the system anticipated, and the synthetic set cannot.

The interesting version of the question is when the real set is 300 cases rather than 5,000, which is the ordinary situation a few months after launch. The pull toward the larger number is strong, because 5,000 cases feel like they resolve more than 300. What 300 real cases resolve is a matter of arithmetic — a 74% pass rate on 300 items carries a 95% Wilson interval of roughly 69% to 79%, so a three-point change is invisible and a ten-point regression is not — and chapter 7 works through what to do with that. What 5,000 synthetic cases resolve is a much tighter interval around a quantity that is not the one you want. Precision on the wrong estimand is the trap, and it is more seductive at 5,000 rows than at 300 because the confidence interval looks so much better.

The comparison in this chapter is a property of a particular pair of sets at a particular time, computed with a particular list of feature detectors, and every one of those choices affects what it says. That matters because the number it produces rarely stays with the person who computed it. Eval sets get shared, and the team that runs yours next quarter sees 500 cases and a score, and has no way to know that 31.4% of real traffic falls in cells the set does not cover — unless that fact travels with the data.