Sampling fifty answers a week cannot see eight percent offset drift

Drift, in this chapter, is a stored citation that still resolves and no longer resolves to the text it quoted. It is produced by any change to how documents are parsed or split: a new extractor, a different chunk size, a re-run of ingest. The span was recorded against one segmentation of a document and is being read against another, so the offsets now index into a different string. Nothing about drift announces itself. The identifiers still exist, the ranges are still numerically valid, and the rendered highlight still lands on a real sentence.

A sweep is the countermeasure: a pass over stored citations that re-derives what each one resolves to now and compares it against what was recorded then. The reason a sweep is possible at all is the quote field from chapter 1 — the text as it read at answer time, copied into the record. Without it there is nothing to compare against, and drift is undetectable by any means short of reading the document. Everything in this chapter is downstream of that one redundant field, and a corpus whose citations were stored without it has no cheap route to any of it.

Why a weekly sample cannot find this#

The instinct when citations might be wrong is to have a person read some. Fifty answers a week, hand-checked, sounds like a reasonable audit until you put numbers through it. At an average of 2.6 citations per answer, fifty answers is a hundred and thirty citations, and if eight percent of stored spans are affected then about ten of those hundred and thirty are, spread over roughly four answers. Four affected answers in a sample is not a small signal, so it would seem the process catches the problem in week one.

It does not, because of what an affected citation looks like to a reviewer. The span resolves to the paragraph before or after the true one, in the same document, under the same heading, on the same topic, phrased in the same register. A reviewer checking whether the evidence is topically appropriate — which is what checking looks like at speed, especially when the reviewer already believes the answer — will pass most of them. Call the per-citation detection probability 0.25, which is roughly what an audit of the screen recordings from chapter 4 suggested for near-topic mismatches.

With about ten affected citations in a week, the chance that nobody flags any of them is the chance of missing all ten independently, which is 0.75 raised to the tenth power: about 0.056, or roughly one week in eighteen. So a flag is likely — the arithmetic says the sample will produce one. That is the trap, because a flag is not a detection. It is one reviewer saying a citation looked off, on a system where individual bad citations are already expected at 6%, and it gets triaged as an isolated miss. Recognising a systemic 8% drift needs enough flags to form a pattern, someone to notice that they all postdate the same deploy, and someone to connect the deploy to a chunking change. In practice that is three to six weeks, and the recurring version of this story is that it gets found when a customer disputes an answer and somebody goes back through an archive.

The tempting fix, once the cause is known, is a post-reindex assertion that stored offsets still fall within their chunk boundaries. It is cheap, it runs in seconds, and it catches almost none of the affected citations. This is the stumble worth naming carefully, because the check is not lazy or badly written — it is a correct implementation of the wrong predicate. Range validity asks whether the numbers address positions that exist. Text identity asks whether those positions hold the same characters. A boundary shift leaves the first true and the second false, and it does so systematically rather than occasionally: a chunk that grows from 400 tokens to 512 is strictly longer, so a range that was valid in the shorter chunk is almost always still valid in the longer one. The bounds check is a quantity that stays green while the dangerous property varies underneath it, and it gets greener as the chunks get bigger.

Scaling the sample is the obvious response, and it is worth pricing before anyone proposes it in a meeting. Suppose a system answers about 10,000 questions a week; at 2.6 citations an answer that is roughly 26,000 citations, and a reviewer checking one citation properly takes about a minute — the figure chapter 5 measured when the resolution interface was good.

Every cell in the first four columns of the first three rows follows from four numbers, and it is worth doing one row in full to see that nothing else is in there. Take the middle row. Two hundred and fifty answers at 2.6 citations each is six hundred and fifty citations read. Eight percent of six hundred and fifty is fifty-two affected. A quarter of fifty-two is thirteen flags. Six hundred and fifty minutes at a minute each is a little under eleven hours. The bottom hand-audit row works the same way: a thousand answers is two thousand six hundred citations, eight percent of that is two hundred and eight, a quarter of that is fifty-two, and two thousand six hundred minutes is a little over forty-three hours.

What that closure licenses is a claim about the table's status. It contains no measurement of this system beyond the four rates it started from — citations per answer, share affected, detection probability, minutes per check. It is arithmetic, not evidence, which means two things at once: nobody needs to trust the numbers to check the argument, and the argument transfers directly to any deployment where those four rates can be named. It also means the table cannot be improved by better sampling practice, since sampling practice is one of the four inputs and its best plausible value is already in there.

The third row is a full-time job and a half, spent reading citations that were correct, and it still finds the drift a week after it started. The fourth row costs less than a minute of a machine's time and reports every affected citation with its offset shift attached. Sampling remains useful for the questions only a person can answer — whether the evidence is the evidence a lawyer would have chosen, whether the answer is worth reading — and it is the wrong instrument for a systematic, mechanically detectable defect that touches thousands of records at once. A human audit is a measurement of quality, and this is a fault.

Six checks and what each asserts#

The assertion column is the one to read, because the six rows are not six degrees of strictness. They are six different predicates, and the interesting relationships between them are the implications that do not hold. In bounds does not imply quote match, which is the whole of the previous section. Quote match does not imply entailment, since text can be exactly where it was recorded and still fail to establish the claim. And none of the first five say anything at all about a sentence that carries no citation, for a reason given below.

Quote match is the check that catches drift, and it exists only because chapter 1 put the quote in the record. The comparison is not raw equality, because the same normalisation problems from chapter 2 apply — a re-extracted PDF may legitimately produce a different dash — so it runs through the same folding. The 0.98 threshold is chosen to sit in a gap rather than at a considered midpoint: typography differences move a similarity ratio by a percentage point or two, and a paragraph shift produces two strings with almost nothing in common, which scores near zero. There is nothing much between those two regimes to be wrong about.

from rapidfuzz import fuzz

def quote_match(chunk_text: str, cit: dict) -> dict:
    start, end = cit["chunk_span"]
    if not (0 <= start < end <= len(chunk_text)):
        return {"ok": False, "why": "out_of_bounds"}

    found, _ = normalise_with_map(chunk_text[start:end])
    stored, _ = normalise_with_map(cit["quote"])
    ratio = fuzz.ratio(found, stored)
    if ratio >= 98:
        return {"ok": True, "ratio": ratio}

    # The quote is the durable artifact; try to find where it moved to.
    relocated = locate(chunk_text, cit["quote"])
    if relocated and relocated["how"] != "review":
        return {"ok": True, "repaired": True, "ratio": relocated["score"],
                "chunk_span": [relocated["start"], relocated["end"]],
                "shift": relocated["start"] - start}
    return {"ok": False, "why": "quote_not_found", "ratio": ratio}

The repair path is the reason storing the quote earns its keep twice. Because the text is in the record, a drifted citation is not merely detectable, it is mechanically fixable: run chapter 2's matcher against the current chunk and rewrite the offsets. The shift field falls out of the same call for free, as the difference between where the quote is now and where the record said it was, and a distribution of shifts is a far more legible object than a list of failures — a systematic cause produces a tight cluster of similar shifts in one direction, while scattered individual errors do not.

Coverage is the check that closes a hole the others cannot see, and the hole is a consequence of how the other five are written rather than of anything they get wrong. Every check in the table except coverage iterates over the citations an answer emitted. A claim asserted with no citation emits none, so it never enters any of those loops, and passing five checks by not participating in them is indistinguishable in the output from passing them on the merits. An answer that states four facts and cites two has a citation precision of 1.0 — both of its citations entail their claims — and a hole in the middle of it that no per-citation number can express. Implementing coverage needs a classifier over the answer's sentences that separates claim-bearing sentences from framing ones ("here is what I found", "let me know if you need the full policy"), which is a small fine-tuned model or a cheap LLM call, and a rule about what happens to the uncovered ones. Rendering them visibly as uncited is the honest default; silently dropping them changes the answer, and leaving them unmarked is what produced the hole.

Placing the checks#

The split follows cost, and the cost comparisons are arithmetic anybody can repeat. The three inline checks together take about half a millisecond per citation, which on an answer with three citations is under two milliseconds against a generation that took four seconds — invisible, so there is no reason to run them anywhere else. Entailment through a judge is a different order of magnitude: at 120 ms per citation, the same three citations add three times a hundred and twenty milliseconds, which is 360 ms on the p50 response, the one in the middle of the latency distribution. Paying that inline buys a check whose action is rarely to block, so it runs behind the response, where a failure produces a flag and, for high-severity topics, a retroactive suppression of the claim in the stored answer.

The sweep is the part teams skip, and it is the one that catches a chunking change before anybody reads a drifted answer. Over an archive of 180,000 stored citations, quote match at 0.3 ms each is a hundred and eighty thousand times three ten-thousandths of a second — about 55 seconds of CPU, so it is affordable to run over everything after any change to parsing, chunking or the document store. Entailment over the same archive is the same multiplication with a much larger factor: a hundred and eighty thousand at 120 ms is twenty-one and a half million milliseconds, which is six hours, and at $0.0004 each it is $72. That ratio, rather than any judgement about which check matters more, is what assigns them different regimes. Quote match is swept exhaustively because exhaustive is cheap; entailment runs over a stratified sample of 2,000 — a sample drawn so that every group you care about is represented in it rather than left to chance, stratified here by document, by age of the answer, and by the near-miss families from chapter 3 — because exhaustive is six hours and seventy-two dollars every time somebody redeploys a parser.

Failure policy needs to be decided per check rather than globally, and the table's last column is the short version. Dropping the citation is right when the span is invalid and the rest of the answer stands. Degrading to a document link is right when the document is intact and only the location is untrustworthy, and it should be labelled as a degraded citation so the reader knows they are back in the chapter 1 world. Suppressing the claim is a heavy action reserved for cases where a claim without evidence is actively dangerous, which in a support context means pricing, security commitments and anything with a legal effect. Paging someone is a rate-based decision rather than an instance-based one: a single quote-match failure is routine at a baseline of about 0.3%, and the alert fires when the rate exceeds 2% over a fifteen-minute window or when a post-reindex sweep reports more than 1%, because both of those mean a systemic change rather than a bad answer.

Content changes, and the two queues a sweep produces#

The sweep has one failure of its own, and it arrives the first time it runs after a content change rather than an infrastructure change. Two months after one chunking fix, a vendor reissued 4,100 policy documents with revised wording, and the sweep reported 71% quote-match failures across that source in a run that had been reporting 0.3%. Nothing had drifted. The quotes no longer existed because the sentences no longer existed, which is a correct citation to a document that has since been superseded, and auto-repairing those offsets would have been actively wrong — the matcher would have found the nearest surviving wording and silently re-pointed a citation at text the answer never saw.

Separating the two cases is what doc_version is for, and the separation is exhaustive by construction: every failing citation has a stored version that either equals the version currently in the store or does not, with no third possibility. A quote-match failure where the two versions are equal is drift, and drift is repairable, because the document has not changed and therefore the text the citation quoted is still somewhere inside it. A failure where the versions differ is history, and the right action is to mark the citation as pointing at a superseded version and to keep serving the archived copy the answer was actually generated from, rather than to relocate anything.

def classify_sweep_failure(cit, store):
    live = store.version_of(cit["doc_id"])
    if live == cit["doc_version"]:
        return "drift"            # same text, moved: repair offsets
    return "superseded"           # different text: pin to archived version

A sweep that omits that two-line branch fails in both directions, and which direction it fails in depends only on what its operator chose to do with failures. If failures go to a review queue, every vendor content refresh floods it with thousands of citations that are behaving correctly. If failures go to the repairer, the repairer works its way through a content update rewriting citations to point at sentences that did not exist when the answers were written, which is a fabrication produced by maintenance. Rate alerts need the same split for the same reason, since a single threshold over the combined failure rate fires on every content refresh and trains whoever is on call to acknowledge it without looking.

Every check in this chapter is scoped to a citation or, in coverage's case, to a sentence. None of them looks at the answer as a whole, and that scope is a property of how they are written rather than an omission that a longer table would fix. An answer can pass all six — every span resolves, every quote matches, every claim is covered, every citation entails its sentence — and still assert something in its final paragraph that no cited passage establishes, because the final paragraph is drawing a conclusion from the claims rather than from the documents. A checker that iterates over citations cannot reach a property of how the claims were composed, and that is the subject of the last chapter.

The checks also do not validate themselves. The entailment component carries the error rate measured in chapter 3, which was 0.61 on conditional language and worse on aggregation, so a flag from it is a prompt to look rather than a verdict, and an absence of flags from it is weaker still. Treating a checker's output as ground truth is how a team convinces itself that a class of failure has been eliminated when what has actually happened is that the failure stopped being reported.