A fluent wrong answer gives no warning, so the friction must exist already

At 14:19 on a Tuesday, a handler working claim CLM-88421 asked Atlas about the appeal window for a declined subsidence claim and got two sentences back. Every sentence came back entailed. The support label read supported by your documents. The citation resolved to clause 14.2 of the Household Claims Handling Manual, version 7, which was the current version in the store, and the highlighted passage did say sixty days. The handler did not open it, because the answer was clean and they had opened enough citations that month to know what they would find.

At 14:21 the handler drafted a decision letter, ran the rewrite, read a three-hunk diff, rejected one hunk about tone, and applied the rest. At 14:23 they pressed send. The letter sat in the outbox for two minutes and went out at 14:25.

The answer was wrong. A 2025 endorsement, circulated by the underwriting team as an emailed PDF and never added to the document store, had shortened the appeal window on subsidence claims from sixty days to thirty. The manual in the store was internally consistent, current as a file, and superseded as a policy. Every instrument in the product reported green, because every instrument in the product measures the relationship between the answer and the corpus, and the corpus was the thing that was wrong.

That is the moment worth designing for. It is not an error state — there is no error state, and no signal, and no hesitation from the handler. Whatever protection exists at 14:21 was paid for in engineering weeks and user friction weeks earlier, or it does not exist.

Where the protections sit, and where the gap is#

The dotted attachments are the mechanisms of the preceding eight chapters, and the diagram shows the thing that is easy to miss in a design review: between Display and Action there is nothing. No mechanism fires at 14:21. Every control either happened earlier, and therefore had to be built before anyone knew this answer would be the wrong one, or happens later, and therefore only limits the damage. The design question is never what do we show when it goes wrong, it is what did we already spend so that the wrong case is survivable.

Pricing the mechanisms#

Each control has a price in four currencies: user time, conversion, latency, and engineering. Naming all four is what turns a design argument into a decision someone can make.

On Atlas the whole set came to roughly 19 engineering weeks, about 530ms of added answer latency, eight points of acceptance on the rewrite feature, two minutes of delivery delay, and four hours a week of a senior handler's time indefinitely. That last line is the one that gets cut in the second year and is the only row that would have caught CLM-88421 before a policyholder did.

Against the shallow version of this design — a banner saying outputs may be inaccurate and a button to report problems — the comparison is not close. The banner was present on all 22,400 answers that quarter, including every wrong one, and its information content is therefore zero; it does not distinguish the 14:19 answer from the 14:18 one. The report button requires the user to have noticed, and this user did not notice, which is the defining property of the case being designed for. Both are real controls with real uses — the banner sets a legal expectation, the button catches the loud failures — and neither of them is a protection.

The control that would have caught this one#

Every mechanism in the table measures the answer against the corpus, so the only ones with any purchase on a corpus that is missing a document are the ones that describe the corpus itself. That is why the freshness statement is in the list despite being the least glamorous line in it.

What Atlas added to layer 0, after the incident, was a single sentence: Answering from the Household Claims Handling Manual, last revised 14 months ago. No endorsements are indexed for this policy. Both halves came from the document store rather than the model. The first is a date. The second is a coverage check: for the policy on the claim in context, does the store contain any document of type endorsement with an effective date after the manual's revision date. A handler with local knowledge reads the second sentence and remembers the email.

def coverage_note(policy_id: str, cited_docs: list[Doc]) -> str | None:
    primary = max(cited_docs, key=lambda d: d.revised_at)
    endorsements = store.find(policy_id=policy_id, doc_type="endorsement")
    indexed = [e for e in endorsements if e.indexed]
    if not indexed:
        return (f"Answering from {primary.title}, last revised "
                f"{months_since(primary.revised_at)} months ago. "
                f"No endorsements are indexed for this policy.")
    stale = [e for e in indexed if e.effective_from > primary.revised_at]
    if stale:
        return (f"{len(stale)} endorsement(s) postdate {primary.title}. "
                f"Check them before relying on this answer.")
    return None

It fired on 31% of subsidence answers, which is a high enough rate that it would have become furniture if it had been shown as a permanent banner; shown only where the coverage check actually finds a gap, it stayed meaningful. Nine weeks after launch, handlers had used it to flag four more unindexed endorsements, and the underwriting team's email workflow was changed so that they went into the store.

What happens after it happens#

Some fraction always ships, and the second half of designing this moment is how fast the organisation finds out and how much it can undo when it does.

Discovery came from the labelled review, four days after the letter went out. Recovery depended entirely on the assisted_by_answer link between actions and the answers that produced them, which turned how many letters used the sixty-day figure from a manual audit into a query, and returned 1,140 in under a minute. Nothing in the interface produced that link; it was a field in the action record from chapter 6, added for a different reason, and it was the difference between a bounded remediation and a general apology.

The other half of recovery is a sweep. Once the endorsement was indexed, the team re-ran the 22,400 logged questions from the quarter against the corrected corpus and diffed the new answers against the stored ones at the level of the support envelope, which cost about $90 in inference and six hours of wall time.

changed = []
for row in logged_answers(quarter="2026Q1"):
    new = answer(row.question, corpus_version="current", acl=row.acl_scope)
    if materially_differs(row.envelope, new.envelope):   # numbers, dates, negations
        changed.append((row.answer_id, row.actions, diff(row.envelope, new.envelope)))

Seventy-four answers changed materially, of which 61 were the subsidence window and 13 were unrelated defects nobody had reported. The sweep is only possible because the questions, the envelopes and the corpus versions were all retained, under the retention and consent rules of chapter 8, which is a reasonable argument for keeping the envelope rather than only the rendered text.

Not everything deserves all of it#

Spending the full nineteen weeks on every surface produces a product that is exhausting to use, and the mapping that decides where to spend has two axes: how expensive a wrong answer is, and how long it takes anyone to notice. A wrong autocomplete is cheap and instantly visible, so it gets none of this. A wrong internal summary is cheap and slow to surface, so it gets the label and nothing else. A wrong letter is expensive and slow to surface, and it gets everything, which is what the composer eventually had. The quadrant that catches people out is expensive-and-slow, because the absence of complaints reads as success right up until the point where six weeks of them arrive at once.

The friction is also not evenly spendable within a surface. Chapter 5's risk-ordered hunks put the extra click on the 9% of rewrites touching a number, a date or a negation, and that principle generalises: a control applied to everything becomes furniture, and a control applied to the narrow slice where the damage is concentrated stays legible. The same is true of the freshness note firing on a third of subsidence answers and none of the motor ones.

What none of this does is make the model right. The mechanisms in the table buy three things — the wrong claim is localised rather than spread across a fluent paragraph, the committed action is reversible for a defined window, and the blast radius is queryable after the fact — and they leave the underlying error rate exactly where it was. A team that reports improved trust metrics after shipping all of it has improved the accuracy of the trust, which is the honest goal; users who correctly distrust an answer they should distrust are a success, and a product that has made distrust cheap enough to act on has done its job.

The place to take this next is your own surface. Pick the action a user takes on the output, work backwards to what fires between display and that action, and price what is missing in the four currencies above. For further depth, the CHI 2024 paper Design Principles for Generative AI Applications is the most rigorous treatment of designing for imperfection and user control, and Patterns for Building LLM-based Systems & Products is the most practical catalogue of how teams have implemented the uncertainty and feedback halves in production.