Every citation correct, and the conclusion supported by none of them

A composition failure is an answer whose individual claims are each supported and whose conclusion is not. Nothing is wrong with any citation in it. The spans exist, the quotes match, and each span entails the sentence it is attached to; what fails is the step from those sentences to the one drawn at the end. Every check built in chapters 3 and 6 iterates over citations or over sentences, so a defect that lives in the relation between sentences is outside all of their scopes by construction rather than by oversight.

This matters most in features that summarise, and it is worth saying why before saying what to do about it. A summary is a genre that asks for conclusions. Its value to a reader is precisely the sentence that says what all the material amounts to, and that sentence is where the reader's attention lands. So the class of feature that most needs its conclusions to be checkable is the class whose conclusions are hardest to check, and the citation stack that behaves well on pricing questions can behave badly here without any component of it changing.

Naming the failure#

Being precise about which metric should have caught it is the fastest way to see why none did. Citation precision asks, for each emitted citation, whether the span entails the claim it is attached to. A figure of 0.96 is answering that question correctly. Take a summary ending on a claim about the main driver of churn risk with three markers under it: test each of those three citations against the sub-claim it actually supports and each one passes, which is what a precision of 0.96 is made of. Citation recall asks whether each claim is fully supported by its citation set, and this is where a conclusion sentence should fail. It does not, and the reason is a detail of how recall is computed. Recall is normally measured by decomposing the answer into claims using the same generator that wrote it, and that generator decomposes a sentence about the main driver of churn risk into sub-claims about billing complaints — each of which is supported. The decomposition dissolves the quantifier before anything measures it.

The failure has two mechanisms, and they need different responses.

The first is aggregation over an unrepresentative set, and it is a sampling argument rather than a citation argument. "Main driver" is a claim about a distribution: which category accounts for more of the churn risk than any other, across everything the account filed. The evidence for a claim of that shape would have to be a count over all of the tickets. Suppose the system saw 8 of 47 the account had filed, selected by a retriever optimising for relevance to a query about churn risk. Then the selection rule and the conclusion share a criterion — both are about churn risk — so the sample was drawn in a way that favours exactly the material the conclusion rests on. It is not merely a small sample. It is a sample biased toward the conclusion it is then used to support, which is the one defect no amount of extra sampling from the same rule can fix.

There is a second, deeper reason no span can help here, and it is the one to carry away. The claim is not about any passage. It is about the corpus: about how many tickets of each kind exist. Entailment relates a premise to a hypothesis, and no premise made of prose contains a fact about how many other prose passages there are. That is why chapter 3's aggregation family scored 0.12 under every checker configuration, flat across premise widths and models — the score was not measuring a weak checker, it was measuring a category of claim with no possible premise in the corpus.

The second mechanism is quantifier and comparative language, which appears everywhere in summaries: most, increasingly, the primary, unlike last quarter, has not been resolved. These are claims whose truth conditions involve absence, comparison or counting, and evidence of presence never settles them. Three tickets about billing establish that three tickets about billing exist. They cannot establish that no larger group exists, because a passage's existence is evidence about that passage and silent about every passage not retrieved. A summary invites this language by its nature, which is why a summarisation feature has a structurally higher rate of unsupportable conclusions than a question-answering feature does.

Five ways forward, and what each costs#

Read the first two columns as a pair rather than reading the rows as a ranking. The rows are not six qualities of the same solution; each one covers a disjoint region of the failure, and the gap between what a row catches and what it misses is the shape of that region. The quantifier ban catches every conclusion that announces itself with a trigger word and misses every implicit aggregation phrased without one. The data path catches anything expressible as a count over a structured field and misses everything else entirely. The entailment pass catches genuine non-support and misses precisely the three configurations chapter 3's own numbers say it handles worst. Because the regions are disjoint, the rows compose rather than compete, and the question to ask about any of them is which region of the failure it removes rather than how good it is.

The obvious fix is the fourth row, and it is where most teams go first: keep the citations, add a second pass that checks whether the conclusion follows from the union of the cited spans. The mechanism is correct. The confidence usually attached to it is not, and the argument against that confidence is available before the pass is built. A multi-sentence conclusion, containing a comparative, evaluated against evidence spread across several passages, is the exact configuration where entailment models are least reliable — chapter 3 measured conditionality at 0.61 and aggregation at 0.12 against 0.91 on simple scope. A churn conclusion of the kind above is an aggregation case with a comparative in it, which is to say it sits at the 0.12 end and not the 0.91 end. Shipping a checker on that basis substitutes a green tick for a visible defect, and a team that has replaced "several reviewers noticed a problem" with "the conclusion checker passed" is more confident and no safer.

That does not mean do not build it. It means the checker is a thing to be measured before it is a thing to be trusted, and the measurement is specific: label conclusions by hand, on this feature's own output, for this failure. Three hundred conclusion sentences, two annotators, adjudication on disagreements, is about fifteen hours of work.

Then look at the checker's performance on the unsupported class rather than its overall accuracy, because the base rate — the share of conclusions that are unsupported before any checker looks at them — makes overall accuracy meaningless. If 6% of conclusions are unsupported, a checker that passes everything is correct on the other ninety-four percent and scores 0.94, which is a better-looking number than most real checkers achieve while detecting nothing at all.

The numbers that do mean something are worth working through once, because their arithmetic decides how the checker may be used. Over 1,000 conclusions at a base rate of 6%, sixty are genuinely unsupported. At a recall of 0.7 — meaning it flags seven of every ten conclusions that genuinely are unsupported — the checker catches seven tenths of sixty, which is forty-two. At a precision of 0.5 — meaning half of what it flags turns out to be fine — those forty-two true flags are exactly half of everything flagged, so the flagged set is eighty-four and the other forty-two are false alarms. The queue is exactly twice its yield, and that identity is what settles the design question. A list where one entry in two is a real problem is a good triage queue, because a human reading it spends half their time on genuine defects, and it is an unusable gate, because half of everything blocked would have been fine. Knowing which of those you have is the difference between routing conclusions to a human and blocking them automatically, and it is not a judgement call once the two numbers are measured.

Removing the citations, stated at full strength#

The argument for removal is stronger than it first sounds, and it deserves to be made without hedging before it is answered.

Citations change how the text is read, and the effect has been measured rather than assumed. In an internal study of 60 people reading the same six summaries, half with citations and half without, cited summaries were rated 4.3 out of 5 for trustworthiness against 3.1, and readers of the cited version challenged 58% fewer individual statements in a follow-up questionnaire. Read those two figures as a gap rather than as levels: markers buy a bit over a point of trust and remove more than half the challenges a reader would otherwise raise. That effect is the point of citations when the citations bear on what matters. In a summarisation feature the sentence that matters is the conclusion — it is the line that goes into the account review deck, and several reviewers admitted they read the last paragraph and skipped the body. So the trust premium is being collected on the one sentence where the evidence is weakest, and the citations that earn the premium are attached to sentences nobody is relying on.

Removal is also cheap, immediate, and has no failure mode of its own. There is no checker to validate, no model to fine-tune, no threshold to pick. It cannot drift, it cannot silently regress after a reindex, and it does not require anyone to keep measuring anything. Against a proposal that involves building a conclusion-level checker of unknown reliability, "delete the feature that is misleading people" is a real engineering argument and not a defeatist one.

Where it fails is a step rather than a sentiment. Removal takes away the evidence and leaves the claim. The summary still ends with a statement about the main driver of churn risk, still goes into the deck, and now nothing in the product connects that sentence to anything. Readers who wanted to check it have lost the only route they had, which chapter 4 measured at 8 seconds per claim. And the reviewers who found the problem found it because the citations were there — they read the conclusion, followed the markers, and saw that the spans did not add up. So removal deletes the detection mechanism for the failure that motivated it, while the error rate itself is untouched: exactly as many conclusions are unsupported after the deploy as before, and the system has moved from a state where that error is discoverable to one where it is not. The unsupported conclusion is the thing to remove, and citations are the instrument that makes its absence enforceable.

Composing the other rows is what removal was reaching for and does not achieve. Banning unsupported quantifiers removes the largest and most damaging family for two days of work. Computing the aggregates properly — labelling tickets by category and citing a query result with its row count, which is a verifiable citation of a different kind — converts a claim about the main driver from an unsupportable prose claim into a supportable data claim, and it is the only option that lets the feature keep saying the useful thing rather than saying less. Marking synthesis explicitly costs almost nothing and is honest about what the last paragraph is. The entailment pass goes on top, in flag-and-route mode, once you know its numbers on your own labels.

The thread running through these seven chapters is that a citation is an assertion the system makes, and every assertion a system makes needs something that can find out whether it is false. Chapter 1 specified the record precisely enough to be checkable, chapter 2 produced it, chapters 3 and 6 built the checks, and chapters 4 and 5 made the human check cheap enough to happen. This chapter is the boundary of that programme rather than a gap in it: the assertions are individually sound and the composition is not, and no per-citation machinery reaches a property of composition, however good each citation gets.

Bringing this into a system you own has a natural order, because each step makes the next one measurable. Start by emitting the record — chunk id, character range, stored quote, document version — even if nothing consumes it yet, because everything downstream is impossible without it and it costs a schema change. Add the three inline checks, which are under a millisecond and will immediately tell you what fraction of your model's evidence strings are paraphrases. Build the near-miss suite next, sixty cases across the families that occur in your corpus, and get the per-family numbers before you believe any aggregate. Then ship the inline span render, and instrument the reviewer events, because that is where a correct citation turns into a checked one. The sweep after every reindex and the entailment pass come after, in that order, since drift is systemic and per-claim entailment failures are not. Whatever you build last, keep the conclusion sentence in view: it is the one people read, and it is the one none of this machinery is watching.