One model generating, answering and grading cannot fail visibly

The summarisation feature launched with a faithfulness score of 96% on 1,500 cases, and the way that number was produced took four days. Model M was prompted to write 1,500 source documents in the target domain — supplier compliance memos, the kind of thing the customers upload. The feature, built on M, summarised each one. M-as-judge scored each summary for faithfulness against its source on a three-point rubric. Ninety-six percent scored faithful, the launch review accepted it, and two weeks later support escalated a summary that cited "the retention obligations under Clause 14.3(b)" for a supplier memo whose text contains no clause 14.3(b), no clause numbering of any kind, and no retention section. Four more escalations followed within a week, all of the same shape: invented clause numbers, stated with the flat confidence of a citation.

The two reasons are independent, and the smaller one gets all the attention#

The first reason the 96% missed this is that the corpus could not contain the failure. M wrote those 1,500 documents, and when M writes a compliance memo it invents clause numbers, because that is what compliance memos have. A summary that reports "Clause 14.3(b) requires 7-year retention" against a source document that says "Clause 14.3(b) requires 7-year retention" is faithful, and it is correctly graded faithful, and the grade is meaningless because there was nothing real for the summary to be wrong about. The generated corpus also made the extraction easy in a way real documents do not: M's memos state each clause number once, adjacent to the obligation it governs, in a consistent format. Real supplier memos carry cross-references to clauses in other documents, defined terms introduced twelve pages earlier, amendments that renumber sections, and footnotes that qualify the body text. The behaviour that broke in production — resolving a citation under ambiguity — was not exercised anywhere in the suite, because the suite's documents had no ambiguity to resolve.

The second reason is the judge. M-as-judge asked to check faithfulness performs a plausibility judgment rather than a verification: does this summary read like an accurate account of that document. A well-formed clause reference reads accurate to a model that itself produces well-formed clause references, and there is a documented tilt on top of that, since judges score text from their own family higher than text from other families on the same rubric. Those two effects are independent of the corpus problem. Fix the corpus and keep this judge, and it will still wave through an invented citation that looks right; fix the judge and keep this corpus, and there are no invented citations to catch.

The dotted edges are the two blockages, and they sit at different points. Almost every discussion of this setup concentrates on the judge, because LLM-as-judge is the piece with a literature, a vocabulary and a known failure name. Here it is the less consequential of the two. Replacing the judge produces a suite that grades the same 1,500 fabricated memos slightly more harshly and certifies a system that still cannot exhibit the bug.

Which one to replace, if you replace one#

Three uses of M are in play. The system under test is not really a candidate, since that is the product. The choice is between the corpus and the judge, and they cost very different amounts.

The two-week estimate on the corpus row is not padding, and it explains why teams choose the judge swap. Getting 300 real supplier memos into an eval store means a legal review of what the customer contract permits, a pass to replace company names, contact details and reference numbers with consistent surrogates so the documents stay internally coherent — topic 3.4's pseudonym design, applied to documents rather than conversations — and then human faithfulness labels on the summaries, which for a compliance memo runs about eight minutes a document and is not work a support engineer can absorb on a Friday. Call it fourteen person-days spread across three functions. Against a four-point score movement it looks unaffordable, and against the eighteen-point movement it actually produced, and the escalations it would have prevented, it is the cheapest thing on the list. The difficulty is that you cannot know which of those two it is until after you have done it.

The middle row is the one worth dwelling on. The deterministic check is a genuinely good instrument, and on the generated corpus it reports almost nothing.

CLAUSE = re.compile(r"\bClause\s+\d{1,2}(\.\d{1,2})*(\([a-z]\))?", re.I)

def unsupported_citations(summary: str, source: str) -> set[str]:
    cited = {m.group(0).lower().replace(" ", " ") for m in CLAUSE.finditer(summary)}
    present = {m.group(0).lower().replace(" ", " ") for m in CLAUSE.finditer(source)}
    return cited - present

Run that over the 1,500 generated memos and their summaries: 4 summaries out of 1,500 cite something absent from their source, a rate of 0.3%, which is a transcription slip rather than a hallucination. Run the identical function over 300 real customer memos and their summaries: 34 summaries out of 300 cite something absent, a rate of 11.3%. Same code, same system, same prompt, two corpora, a rate that differs by a factor of thirty-eight. The instrument was never the problem. The material being measured was.

That ratio is also the answer to how much of the 96% was a statement about the system. On the real memos with the same rubric and the same judge, faithfulness came out at 78%, and with an independent judge at 74%. The judge swap moved the number by four points; the corpus swap moved it by eighteen and was the only change that made the escalated bug reproducible in the suite.

What would make the replacement suite believable#

A new number is not evidence of a better suite by itself, since a suite can produce a lower score and still be unable to fail for the right reason. Three things are worth requiring before the replacement is trusted.

The escalated production cases go in first, and the suite must fail on them before the fix and pass after. Five real memos, five real summaries, five real invented clause numbers — a suite that scores those faithful is disqualified regardless of what it does elsewhere. This is the cheapest and most decisive check, and it is available the moment an incident exists.

Then the judge is calibrated against people on the material it will actually grade. Take 120 of the real memo-summary pairs, have two people label faithfulness against the rubric, adjudicate the disagreements, and compare the judge against those labels. What you want to see is not agreement in aggregate but agreement on the positive class: of the 34 pairs a human called unfaithful, how many did the judge catch. The first version caught 19, which is a recall of 0.56, and the number that had been quoted as "96% faithful" was being produced by an instrument that missed nearly half of the failures it was pointed at.

Finally, the suite is checked for whether it can fail at all, by feeding it known-bad output. Take fifty correct summaries and corrupt each one in a specific way — renumber a clause, invert an obligation, move a deadline by a year, attribute a requirement to the wrong party — and confirm the suite flags them. This is mutation testing applied to an eval, and it costs an hour. A suite that scores the corrupted summaries at 88% faithful is telling you its 78% on real data is not measuring what its name says.

It is also worth being precise about how independent a different-family judge really is. Two models from different labs share a great deal of pretraining data and a great deal of prior, so they agree that a well-formed citation is probably a real one, that fluent text is probably accurate, and that a confident register signals correctness. What changes when you swap families is the self-preference component and some idiosyncratic scoring habits, which is a genuine but bounded improvement. The only fully independent grader in this chapter is the twelve-line regular expression, which has no priors at all and cannot be talked into anything, and that is the general shape of the trade: where a property can be checked by extraction and comparison, a deterministic check beats any judge on both reliability and cost, and the judge's proper job is the residue that genuinely requires reading.

The vendor version of the same question#

A vendor arrives with a system that scores 96% on a 10,000-case benchmark their own pipeline generated, and offers to generate 10,000 more from your documents so you can evaluate before signing. Every structure in this chapter is present in that offer. Their generator and their system share assumptions, and their benchmark score is a statement about the internal consistency of their pipeline rather than about your workload. Generating from your documents fixes the corpus content and leaves the case shapes — which questions get asked, which situations get constructed — under the control of a process tuned alongside the product it will be scoring.

What to insist on is a few hundred cases drawn from your own traffic, with ground truth established by your people, that neither their generator nor their system has seen, and to make the purchase decision rest on that score. Three hundred is enough to distinguish a system that works from one that does not, which is the decision actually on the table. Their 10,000 generated cases are worth running as a secondary signal — they are free, and a system that fails on its own vendor's generated benchmark has told you something — and worth nothing as the basis of the decision. The same rule applies internally, which is what chapter 1's refusal to validate solely on generated data amounts to in practice: some quantity of real, independently labelled cases sits underneath every claim, and its size is a budget question rather than a methodology question.

None of this is an argument that judges must always come from a different family, or that generated corpora are never appropriate. Where the object being summarised has no ground truth outside itself — style transfer, tone rewriting, format conversion — a generated source document is a perfectly good input, since there is nothing external for the output to contradict. The axis that matters is whether the property you are measuring is a property of the relationship between the output and something real. Faithfulness is. Fluency is not.

Replacing the corpus with 300 real memos fixes the coupling and introduces a limitation of its own, and it is the one the next chapter opens on. Those 300 memos are a snapshot: documents as they stood on the day they were exported, questions answered against a corpus frozen at that moment. A system whose correctness depends on how current its information is has a whole dimension that no frozen corpus — real or generated — can express.