A correct citation that costs a minute to check does not get checked

A friction metric measures what verifying a claim costs the person doing it: how long a check takes, and whether the interface was sufficient or the reader had to go and find the evidence themselves. A correctness metric measures whether the system's citations are right. Both are about citations and they answer different questions, so they can move independently, and a dashboard carrying only the second will show nothing at all when the first is the reason nobody is using the product.

The distinction matters because the two are routinely presented side by side and read as though one bounded the other. An accuracy figure is computed from the answer record and an entailment model; a friction figure is computed from events emitted by an interface while a human works. Nothing in the first computation reads anything a reader did, which is worth holding onto, because it means the relationship between the two numbers is not weak or noisy — it is absent by construction.

Two quantities that cannot be converted#

Take an accuracy figure of 0.94 against an entailment checker and a report from reviewers that they do not trust the answers and are reading the source documents themselves before responding to a customer. Both can be exactly true, and the reason is that they are measurements of different things over different objects. The checker measures, over the citations the system emitted, the share whose span entails its claim. The reviewers are describing what it costs them to establish that for themselves, one claim at a time.

Put the two on the same scale and the invariance becomes visible. Raising accuracy from 0.94 to 0.99 changes the fraction of citations that are wrong from 6 in 100 to 1 in 100. It changes the time to find out which by exactly zero, because finding out which one is wrong requires reading the evidence, and reading the evidence costs whatever the interface makes it cost. The first quantity moved by five points and the second did not move at all — not slightly, but not at all, since accuracy is not an input to any step of the reading.

The reviewers' behaviour follows from the second quantity alone, and the arithmetic closes tightly enough to leave no room for a story about anything else. A reviewer facing 40 claims in an hour-long queue, at 55 seconds a claim, spends forty times fifty-five seconds on verification, which is two thousand two hundred seconds, or a little under thirty-seven minutes. That leaves twenty-three minutes of the hour for the work the hour was for. Thirty-seven and twenty-three sum to sixty exactly: the entire hour is accounted for by verification plus everything else, with nothing left over. Against that, reading the source document once takes eight minutes and covers every claim in the ticket at once, because the cost of reading a document is paid once per document while the cost of checking citations is paid once per claim. A reviewer who chooses the eight minutes is not refusing to use the citations. They are correctly pricing them.

That closure is the reason to resist the instinct the two figures always produce in a room, which is to treat the reviewers' report as evidence of hidden inaccuracy — they have been burned by the 6%, so raise the number. That explanation predicts something specific and checkable: if reviewers had been burned, they would be describing particular wrong answers, and the remedy would show up as fewer disputes. What they describe instead is time. A system with 99% correct citations that each cost a minute to confirm produces exactly the same behaviour, which means the observed behaviour carries no evidence about the accuracy figure in either direction.

The belief being corrected here is worth naming as a belief rather than as a mistake, because it is a reasonable one. Accuracy is the quantity that stays high and steady while the dangerous property — what a check costs the person performing it — varies freely underneath it and is never measured. A dashboard where accuracy is the only citation metric is a dashboard that cannot report the failure at all, and its silence reads as an all-clear.

Instrumenting the check as an act#

Nothing on such a dashboard could have shown this, because every metric on it was computed from the answer record and none from what a human did with it. The events you need are small and they are all about the reader.

{"e":"claim_rendered","answer_id":"a91f3c","claim_id":"c3","cites":1,"ts":1730374101.2}
{"e":"citation_opened","answer_id":"a91f3c","claim_id":"c3","cite":0,"surface":"inline","ts":1730374106.8}
{"e":"context_expanded","answer_id":"a91f3c","claim_id":"c3","direction":"after","ts":1730374109.1}
{"e":"viewer_search","answer_id":"a91f3c","claim_id":"c3","query":"overage growth","ts":1730374131.7}
{"e":"source_opened","answer_id":"a91f3c","claim_id":"c3","target":"full_document","ts":1730374140.0}
{"e":"claim_resolved","answer_id":"a91f3c","claim_id":"c3","verdict":"supported","ts":1730374152.4}

Two design choices in that stream do most of the work. claim_resolved requires an explicit action from the reviewer — a supported/unsupported/unclear control on each claim — which turns verification from something you infer out of dwell time, meaning how long a panel stayed open before the reader moved on, into something you record. The difference is not fastidiousness. Dwell time cannot distinguish a reader who read carefully from one who left the tab open, and a metric built on it will report the two identically, which is the same defect this chapter is about arriving one layer down. Reviewers will use an explicit control if it is one keystroke and if the verdict is visibly useful to them, for instance by marking the claim in the ticket draft they are about to send.

The second choice is what viewer_search and source_opened mean. They are logged as escapes: each one means the interface failed to give the reviewer what they needed, and they are the negative signal the whole dashboard is built around. An escape is observable in a way that a reviewer's opinion is not, and it is attributable to a particular claim, a particular document and a particular moment.

Metrics computed from what the reader did#

Read down the target column rather than the before column, because the gaps between the two are not all the same kind of gap. Four of the five rows have a number in the target: a state the interface is supposed to reach, with the distance to it measurable today. The last row does not, and neither does the third. Those two are not goals; they are quantities to be read against something else, and mixing them into a table of targets without saying so is how a dashboard acquires a metric nobody knows what to do with.

Unassisted verification rate is the metric that names the reviewers' complaint. It asks whether the interface, on its own, gave the reader enough to settle the question: if they had to open a search box or leave for the source document, the citation resolved but did not do its job. At 0.21 the complement is just under four fifths, so four checks in five required the reviewer to go and find the evidence themselves, which is the same statement as "we are reading the source documents", expressed in a number that fits on a chart.

Median time to confirm prices the check. It is the quantity that determines whether verification survives contact with a queue, and it should be reported alongside the ninetieth percentile, written P90 and meaning the duration that nine checks in ten come in under. A median of 10 seconds with a P90 of 90 seconds means one claim in ten is still unaffordable, and reviewers will remember those ten percent rather than the ninety, because the expensive checks are the ones that interrupt the work.

Citation open rate is the one to be careful with, and the care it needs is the care a two-worlds measurement always needs. A falling open rate is consistent with two opposite situations — reviewers trust the system and no longer feel the need, or reviewers have given up on citations as a route to verification — and the rate itself cannot separate them, because it counts the same event in both. Distinguishing them takes a deliberate probe, which means creating a population where the reviewer's choice has been removed. Put 5% of answers into an audit mode where the reviewer is asked to resolve every claim, and compare the dispute rate on audited claims against the dispute rate on claims they chose to check. If audited claims dispute at roughly the same rate as self-selected ones, the reviewers' selection was carrying no information and the falling open rate is trust. If audited claims dispute much more, the claims reviewers were skipping were disproportionately the wrong ones, which is abandonment.

Dispute rate is the only one of these that touches correctness, and it is valuable mostly as a calibration signal for the automated checker. Human verdicts on real traffic are the labels the checker's 0.94 is trying to approximate, and disagreements between the two — claims the checker passed and a reviewer marked unsupported — are the highest-yield input to the next round of near-miss cases.

What to expect when the interface ships#

Shipping the inline span render from the previous chapter is the first move, ahead of any further work on the checker, because the friction metrics are the ones sitting at a fifth of where they need to be. The prediction worth writing down before the deploy, so that you are not reinterpreting afterwards, is roughly this: unassisted verification rate rises sharply, median time to confirm falls by most of the difference between hunting through a PDF and reading a highlighted paragraph, escape rate falls, and citation accuracy does not move at all.

That last part is the one people find unsatisfying, and it is the invariance from the start of the chapter restated as a prediction. Accuracy is computed from the citation record and the entailment model; nothing in the interface touches either input, so the number is invariant under everything you just built. If it does move, something is wrong with your measurement pipeline rather than right with your interface — which makes the prediction useful in a second way, as a check on the instruments.

Two second-order effects usually do show up. Dispute rate rises in absolute terms, often substantially, because reviewers are now inspecting many more claims than before and finding the errors that were always there. That rise is a denominator effect and has to be read as one: 11 disputes per 1,000 resolved when only 31% of claims were opened is not comparable to 14 per 1,000 when 80% are opened, because the two rates are computed over populations selected by different rules. The claims a reviewer chooses to open are the ones they already suspect; the claims they open when everything is cheap to open are a much broader set. A denominator whose selection rule has changed is not a denominator you can compare across, and the sensible framing is disputes per 1,000 rendered claims alongside disputes per 1,000 resolved, since the rendered count is fixed by the system rather than by the reviewer. And time-to-first-response on tickets, the interval between a customer writing in and a human answering them, tends to fall by more than the verification saving alone accounts for, because reviewers who stop context-switching between an assistant tab and a document viewer keep the answer in view while they draft.

Segmenting these numbers by document type is where the actionable detail lives. Aggregated, unassisted verification of 0.62 after a launch reads as a partial success; broken out, it is often 0.88 on markdown policy documents and 0.24 on scanned PDFs whose text layer is poor enough that the highlight lands a few words off, which is a specific defect with a specific owner in the ingest pipeline. The two segments are not a good result and a bad one averaged together — they are two different systems, since the second has a broken input to the offset arithmetic of chapter 2 and the first does not. The same split by claim type separates numeric claims, where a highlighted figure settles the question outright, from claims about obligations and conditions, where reviewers reliably expand the context and often still leave for the source. A dashboard that only carries the aggregate will show a slow drift as the corpus mix changes and give nobody anything to fix.

Friction metrics as an optimisation target#

None of these numbers say the answers are right. A system could reach 0.95 unassisted verification with citations that are uniformly wrong, provided they are wrong in an easy-to-check way, and reviewers would efficiently reject them. Friction metrics and correctness metrics have to be read as a pair, and the pair is the dashboard rather than either half.

The metrics are also gameable in ways that damage the product if anyone starts optimising them directly, and the two obvious exploits both work by removing information from the reader. Reducing time to confirm is achievable by showing less context, which raises the share of quotes whose qualifier is off-screen — the clipped-qualifier failure of chapter 4 arriving as a deliberate optimisation. Raising the unassisted rate is achievable by removing the search box, since the metric counts escapes and an escape the interface makes impossible is an escape that does not get logged. Both would look like wins on the chart. The guard is that dispute rate and the audit sample sit on the same chart, so a friction improvement bought by hiding evidence shows up as reviewers agreeing with claims the checker later fails.

There is a limit on what user-side measurement can see at all, and it is the reason the next chapter exists. Every number here is produced by a person looking at a claim, which makes it a sample, and a biased one — weighted toward the claims reviewers find interesting and the tickets they happen to work. Reviewers are also comparing the rendered span against the claim, which means they are checking the relation the checker checks. What they are not in a position to check is whether the rendered span is the text the system actually cited when the answer was written, because they never saw the original and the interface is showing them the current one. A failure that changes what the offsets resolve to, uniformly and quietly, across a whole archive at once, is invisible to every instrument in this chapter, and it is invisible for a structural reason rather than a resourcing one: a person spot-checking one claim at a time is not positioned to notice it, because they have no baseline to compare against.