Fluent output wears the surface features of a checked answer

A claims handler at a mid-size insurer types how long does a policyholder have to appeal a declined subsidence claim into Atlas, the assistant embedded in the claims console, and gets three sentences back:

A policyholder may appeal a declined subsidence claim within 60 days of the
decision letter. The appeal must be submitted in writing and include any new
structural evidence obtained since the original survey. If the 60-day window
has lapsed, a handler can grant a one-time 30-day extension where the
policyholder can show that the decision letter was delayed in transit. [1][2]

The first sentence is a near-verbatim restatement of clause 14.2 of the Household Claims Handling Manual, which was retrieved and sits in the model's context. The second is a fair summary of two paragraphs spread across that manual and a separate appeals procedure note. The third is not in the corpus at all. There is no one-time 30-day extension anywhere in the insurer's documentation; the model produced a plausible provision because appeals processes in its training data usually have one, and because the sentence before it created a slot that a sentence like that fits. The handler granted the extension four times that week before someone in compliance noticed.

The easy reading of that incident is that the model failed, and the more useful one is that the screen let the failure through. Retrieval put the right clause in front of the model and the first sentence used it correctly; the third sentence is the kind of thing language models have produced since they were built. What carried it as far as a policyholder was a pane that rendered all three sentences in the same typeface at the same weight under the same pair of citation markers, with nothing in it that would let the handler see which of the three the insurer's own documents stood behind.

That pane is what this topic is about. AI product UX — UX standing for user experience, meaning everything the person actually sees, touches and decides from — is the layer sitting between a model's output and the action a user takes on the strength of it. Its job is to carry the information the prose cannot: how well supported an answer is and on what evidence, which words came from which source, what has not finished arriving yet, what a button is about to commit on the user's behalf, and how to get back if that commit turns out to have been wrong. It exists because the fluency of generated text is unrelated to its correctness, which strips out the cues a reader normally uses to gauge a document — the hedge, the hesitation, the roughness around a claim its author was unsure of — and leaves the interface to supply them from somewhere else.

You meet this layer wherever a model has been put in front of a person: the chat pane inside a support console, the grey suggestion sitting ahead of the cursor in a code editor, the generated summary above a page of search results, the improve this paragraph control in a document editor. The same handful of decisions recurs in all of them, which is what makes them worth studying as one subject rather than as a set of unrelated features: what goes on screen first and what is kept one click away, what gets cited and how precisely, whether generated text replaces the user's work or sits beside it awaiting approval, how long a reversal stays reachable after the user has moved on, and what the screen shows during the seconds before an answer exists at all. The nine chapters here work through those decisions roughly in the order a request travels through them, and this first one sets the standard the rest implement.

Nothing in those three sentences distinguishes them. They share a register, a sentence length, an absence of hedging and a declarative mood. The citation markers sit at the end of the paragraph and cover it as a block, so they attach to the invented sentence exactly as firmly as to the quoted one. When the team ran a span-level entailment check over 500 logged answers — entailment being the question of whether a passage actually implies a sentence, so each sentence was tested against the chunks that were in context for that answer — 61% of sentences came back fully entailed, 24% partially entailed, and 15% unsupported. Reading those answers by eye, the three groups were indistinguishable. The unsupported sentences were, if anything, slightly cleaner, because a fabricated provision has no awkward source phrasing to work around.

Prose has no channel for epistemic status#

The reason this happens is structural rather than a bug in any particular model. A language model is trained to produce text that looks like the text it was trained on, and the text it was trained on is overwhelmingly written by people who knew what they were talking about. Confident, well-formed, complete prose is the default output because that is the shape of most published writing. Grounding is a property of where the tokens came from, and that property is not encoded anywhere the reader can see.

The obvious fix — ask the model to hedge when it is unsure — does not survive contact with measurement. Atlas shipped a prompt revision instructing the model to mark uncertain claims with qualifying language. The share of sentences containing a hedge went from 8% to 54%, which looked like an enormous improvement on a dashboard. The correlation between the presence of a hedge and the sentence actually being unsupported by retrieval moved from 0.11 to 0.14. Hedges were distributed across the answer roughly uniformly, because "sound appropriately uncertain" is a stylistic instruction and the model applied it as one. Users noticed something else: after two weeks, handlers reported that the assistant had become "wishy-washy", and the rate at which they read past the first sentence dropped, which meant the hedging was making the good sentences less usable without making the bad ones any more visible.

That leaves the interface. If the answer text cannot carry the difference between a supported claim and a filled-in one, the surrounding chrome has to, and it has to do so with information the system actually possesses. Everything else in this topic — anchored citations, span-level marks, diffs, undo, staged progress — is a mechanism for putting that information somewhere a user can act on it.

Three mechanisms, and the difference between them#

The standard for this topic has three parts, and it is worth separating them, because teams frequently ship one and believe they have shipped all three.

Calibrated labels are coarse statements about the epistemic status of an answer or a part of one: supported by your documents, partly supported, not found in your documents. They are calibrated in the sense that the label is derived from something the system measured, not from something the model asserted about itself.

Visible source boundaries mark where in the answer the documents stop and the model starts. A label on the whole answer says the answer is 80-something percent grounded; a boundary says these words came from clause 14.2 and those words came from nowhere.

Honest partial states are what the interface shows while it does not yet know. Streaming means prose is on screen before retrieval attribution has resolved, so for some hundreds of milliseconds the interface is displaying a fluent sentence whose status is genuinely unknown. Rendering that sentence identically to a verified one is a claim the system cannot back.

The three interact. A label is only as good as the boundaries underneath it, and both are only honest if the partial state does not quietly assert them early.

What each uncertainty surface actually claims#

Product teams reach for uncertainty affordances in a fairly predictable order, and the order is close to the reverse of how useful they are.

The bottom three rows are what the rest of this topic builds. The top three are what gets shipped in week one, and the reason they persist is that they are cheap and they look like diligence. A banner reading Atlas can make mistakes — check important information costs one afternoon and satisfies a compliance reviewer. It also has no effect on the handler reading the subsidence answer, because it was equally present on the two hundred correct answers that came before it.

The numeric confidence row deserves its own caution, because it is the one engineers most want to build. A percentage is a strong claim. It says that across all answers the system labels 87%, roughly 87% are correct, and unless you have measured that against a labelled set and can produce the reliability diagram, the number is decoration with a spurious precision that makes it worse than no label. Where a team does have that data, the honest presentation is usually still a coarse bucket, because users do not distinguish 87% from 79% in any behaviour you can observe, and the bucket boundary is the only part of the number that changes what they do.

A label the system can defend#

The operational test for any uncertainty affordance is whether you can answer, six weeks later and in front of a regulator or an angry customer, why a specific answer carried the label it carried. That means the label has to be a function of logged inputs, computed by code you can point at, and reproducible from the log.

For Atlas that came out as an answer envelope, meaning a structured record that travels with the text and holds everything the interface is entitled to claim about it: the model produces the answer, a second pass computes per-sentence support against the chunks that were in context, and the UI renders from the envelope rather than from the raw text. Later chapters refer back to this record by that name.

{
  "answer_id": "ans_9f3c2b",
  "question": "how long does a policyholder have to appeal a declined subsidence claim",
  "sentences": [
    {
      "id": "s1",
      "text": "A policyholder may appeal a declined subsidence claim within 60 days of the decision letter.",
      "support": "entailed",
      "entailment": 0.94,
      "sources": [{ "chunk_id": "hchm_14_2_c3", "doc_version": 7 }]
    },
    {
      "id": "s2",
      "text": "The appeal must be submitted in writing and include any new structural evidence obtained since the original survey.",
      "support": "partial",
      "entailment": 0.61,
      "sources": [
        { "chunk_id": "hchm_14_3_c1", "doc_version": 7 },
        { "chunk_id": "apnote_2_c4", "doc_version": 2 }
      ]
    },
    {
      "id": "s3",
      "text": "If the 60-day window has lapsed, a handler can grant a one-time 30-day extension where the policyholder can show that the decision letter was delayed in transit.",
      "support": "unsupported",
      "entailment": 0.08,
      "sources": []
    }
  ],
  "label": "partly_supported",
  "label_rule": "v3:min_sentence_entailment<0.35 => partly_supported"
}

Two properties of that envelope matter more than the exact scoring method. The label carries the rule that produced it, versioned, so an answer labelled under rule v3 can be re-examined after the rule changes to v4 without pretending the old answers were scored the same way. And every sentence carries its sources with a document version, which is what makes the citation checkable at all — the subject of chapter 3, and the reason a source that resolves to the right document at the wrong version is not a source.

The second pass is not free. On this product, per-sentence entailment against up to six chunks with a small cross-encoder ran at 40–70ms per sentence on the batch, so a five-sentence answer added roughly 250ms and about $0.0004 in compute. That cost buys the difference between an interface that asserts and one that reports, and it is the cheapest of the mechanisms in this topic by a wide margin.

What this standard does not do#

It does not make the model more correct. The subsidence answer still contains a fabricated provision after the envelope is computed; what changes is that the third sentence is marked, and the handler's attention lands on the one claim worth checking rather than being spread evenly across three. The goal is to move the reader's scrutiny to where the risk is, and to stop the interface from spending trust it has not earned.

It is also distinct from the model-level work of chapter 17.4, which dealt with designing a system around a component that is wrong some known fraction of the time. That chapter was about thresholds, fallbacks and where to put a human. This one is about what the human sees when they get there, and the two can be got badly wrong independently: a system with an excellent routing threshold can still present its low-confidence path in prose indistinguishable from its high-confidence path.

Neither is this a safety or content-policy surface. Refusal, harm filtering and abuse handling are a different set of decisions with a different failure cost, and conflating them produces the worst of both — an interface that treats "I could not find this in your documents" and "I will not answer this" as the same state, which teaches users that the first means the second and that both mean the product is being difficult.

Finally, coarse labels are worthless if they are always the same. A system that labels 96% of its answers supported has a label users stop reading within a fortnight, exactly like the disclaimer banner. The label has to vary in a way that tracks something real, which means the entailment pass has to be good enough to disagree with the model sometimes, and the product has to be willing to show an answer that admits it is partly unsupported.

Having decided that the interface carries epistemic status, the immediate question is where. A label under 600 words of retrieved context and reasoning is not a label anyone reads, and the most common way teams destroy the work above is by putting the machinery first and the answer last. That is the composition problem, and it has a specific and frequently misapplied name.