Dropping, merging and marking lose three different things

By the end of a detection pipeline every chunk in a corpus carries a group label saying which other chunks are copies of it, with a calibrated error rate attached. That label is knowledge and it is not yet an action. Something still has to happen to the copies, and the three available somethings are not variations on one idea — they differ in what they permanently destroy.

Three terms are needed before the comparison. Provenance is the trail from a statement in an answer back to the system, document and revision it was taken from. Divergence is what happens when two copies stop agreeing, because one of them was edited and the others were not. And an access domain is a set of documents a given group of users is permitted to read, which matters here because copies of one passage frequently sit in different ones. Each of the three policies below is fine on some corpora and catastrophic on others, and which is which follows from those three properties rather than from taste.

Deleting copies deletes the answer to where it came from#

Dropping is the policy everyone reaches for first, because it is the only one that makes the index smaller and it requires no changes anywhere else in the system. Retrieval, ranking, prompting and citation all carry on exactly as before, over a corpus 12% smaller. The costs are all deferred and all arrive as someone else's problem.

Provenance is the obvious one. A duplicate cluster is a record of the fact that four systems carried the same content, and that record is often the interesting thing — it tells you the knowledge base has an uncontrolled copy problem, it tells you which connector is generating the copies, and it is what an audit trail needs when the question is which system a statement came from. Deleting the members without recording them destroys the record along with the copies.

The step that is easy to skip is why the deletion is irreversible in the specific way it is. A deduplication job that keeps the winner and removes the losers writes down the identifier of the copy it kept. It does not, unless somebody thought about it, write down the identifiers of the copies it removed. So afterwards a citation names a surviving document, and there is no stored relation from that document back to the set it stood in for. The mapping is not corrupted or hard to query; it was never recorded, and the only remaining copy of it was the index rows the job deleted.

Access control is the less obvious cost, and on any corpus with per-document permissions it is a security defect rather than a documentation gap. The payments corpus had 4,100 near-duplicate clusters after thresholding, of which 1,180 spanned two or more access domains — a Confluence space restricted to finance and a Zendesk article visible to every support agent, say. When the canonical member, the one copy kept to stand for the whole cluster, is chosen by retrieval score, which domain it lands in is arbitrary. In 190 of those clusters the highest-scoring copy sat in a space fewer than 15% of users could read, which produces one of two failures depending on where permission filtering happens. Filter after retrieval and those users get an empty result for content they were fully entitled to read in its other copy. Filter by rewriting the ACL — the access-control list, the set of groups a document may be read by — onto the survivor as a union of the cluster's permissions, which is a tempting simplification, and support agents inherit read access to finance content.

Those two failures point in opposite directions and come from one cause, which is worth stating plainly because it is what rules out a clever fix. Selecting a survivor by score selects its permissions too, and a passage's permissions are not a property anybody intended the score to carry. Whichever way the filter is then arranged, it is being asked to reconcile a set of entitlements with a single document that no longer represents them. Restrictive filtering under-serves the entitled and permissive filtering over-serves the unentitled, and neither is a bug in the filter.

Reversibility is the third. If the source system is still live, deleted rows can be re-ingested; if the connector has been turned off, or the copy was itself the archive of something already gone, the index was the last place that content existed. Deduplication jobs are usually written as maintenance and run unattended, which is not the operational posture for an irreversible delete over an eighth of a corpus.

None of this makes dropping wrong everywhere. For the exact-hash tier from chapter 2 — byte-identical text after normalisation, no ACL divergence, source system still connected — dropping is fine, because there is nothing to lose that is not recoverable in one re-ingest. The trouble starts when the same reflex is applied to the embedding-derived clusters, where the members are not identical and where the differences may be the point.

Two copies that stopped agreeing#

Merging is the more sophisticated version: rather than deleting the extra copies, fold the cluster into a single record that carries a union of the metadata — all four URLs, the earliest creation date, the latest update — and one body of text chosen from the members. It keeps provenance, which is exactly what dropping loses, so it looks like the strict improvement.

What it destroys is divergence. Cluster 8841 on the payments corpus had four members:

cluster 8841  (4 members, min pairwise cosine 0.981)

zendesk/articles/4402                 updated 2024-11-02
  "Invoice disputes must be raised within 60 days of issue."

confluence/BILLING/invoice-faq        updated 2023-06-14
  "Invoice disputes must be raised within 30 days of issue."

drive/exports/billing-faq-2023.pdf    created 2023-07-01
  "Invoice disputes must be raised within 30 days of issue."

mediawiki/Billing/Invoice_disputes    updated 2022-09-30
  "Invoice disputes must be raised within 30 days of issue."

These four were genuine copies of each other when they were made, and the detector is correct to call them a cluster at pairwise cosine 0.981 — a two-character difference in a 400-token chunk moves the embedding almost not at all, which is the property that makes embeddings good at finding paraphrases and blind to a changed number. One of the copies was then updated and the others were not, so the cluster now contains a contradiction on the only fact anyone queries it for.

That single property is doing two opposite jobs at once, and separating them is the teaching in this cluster. An embedding summarises a passage's direction, and one number inside four hundred tokens contributes almost nothing to that direction. Insensitivity to a small edit is exactly why the same model recognises a PDF export whose footer changed, or a runbook somebody reworded — it is the reason the detector works at all. It is also why the detector cannot warn you that the edit which did happen was the one edit that mattered. The strength and the blindness are not two facts to be traded off; they are one fact read from two sides.

Every reasonable merge rule picks the wrong text here. Choosing the longest member, the highest-scoring member, or the member from the most authoritative connector are all uncorrelated with which one is current. Choosing by majority vote picks 30 days, three copies to one, which reproduces exactly the corroboration failure of chapter 1 — except that instead of misleading the model on one query it writes the wrong answer permanently into the index, and the correct copy is no longer there to be retrieved by anything.

A merge rule is therefore the kind of defence worth naming as one. It is a written procedure, it is applied identically to every cluster, and it produces the same shape of output whether the cluster is four true copies or three stale copies and one that was corrected last month. Nothing in its inputs varies with the thing it would need to discriminate, so no amount of care in choosing between longest, highest-scoring and most authoritative improves its behaviour on the case that costs you something. What changes the situation is not a better rule but an extra input.

The mitigation is to look for divergence before merging anything, and the check is cheap because the members are near-identical by construction. Diff them and inspect only what differs:

import difflib, re

NUMERIC = re.compile(r"\b\d[\d,.]*\s*(?:day|days|hour|hours|%|EUR|USD)?\b", re.I)

def divergences(members: list[str]) -> list[tuple[str, str]]:
    base = members[0]
    out = []
    for other in members[1:]:
        sm = difflib.SequenceMatcher(None, base.split(), other.split())
        for tag, i1, i2, j1, j2 in sm.get_opcodes():
            if tag == "equal":
                continue
            a, b = " ".join(base.split()[i1:i2]), " ".join(other.split()[j1:j2])
            if NUMERIC.search(a) or NUMERIC.search(b):
                out.append((a, b))
    return out

The reason this is affordable is the same property that caused the problem. Members of a cluster are near-identical, so a diff between two of them produces very few differing spans, and the filter then keeps only the spans containing a number or a unit. The expensive part of understanding a document — reading it — is skipped entirely, and what a human sees is a handful of word pairs.

Across the 4,100 clusters this flagged 260 with a numeric or date-bearing difference between members, of which manual review found 41 substantive — genuine policy contradictions rather than a changed page number or a reformatted date. About one in six of the flagged clusters survived review, which is a good rate for a filter whose job is to make a human's attention affordable rather than to be right on its own. Forty-one is a small number and it is also the set of clusters where a merge would have silently written a false statement into the index, so it is the set worth an hour of someone's attention. The check misses non-numeric divergences, such as a changed escalation owner or an added exception clause, and catching those needs a natural-language inference pass over the differing sentences, which is a model asked whether one sentence contradicts, follows from or says nothing about another, which costs a model call per flagged cluster and is worth running on the clusters that back high-traffic queries.

Merging also flattens time. Once four dated copies become one record, the question of when a claim was first made and when it was last confirmed no longer has an answer, and recency is one of the few signals available for adjudicating exactly the contradictions above.

Keeping everything and deciding late#

The third policy leaves the index alone and adds knowledge to it. Every chunk keeps its row, its own source, its own permissions and its own timestamps, and gains a cluster id. Retrieval then over-fetches and collapses each cluster to one representative before the results are handed on.

CREATE TABLE dup_cluster (
  cluster_id     uuid PRIMARY KEY,
  method         text        NOT NULL,   -- 'exact-sha256' | 'ann-selfjoin'
  model          text        NOT NULL,   -- 'bge-base-en-v1.5'
  threshold      real,                   -- NULL for the exact tier
  run_id         uuid        NOT NULL,
  created_at     timestamptz NOT NULL DEFAULT now()
);

ALTER TABLE chunks
  ADD COLUMN cluster_id uuid REFERENCES dup_cluster(cluster_id),
  ADD COLUMN cluster_rank int;            -- 1 = preferred representative

CREATE INDEX chunks_cluster_idx ON chunks (cluster_id)
  WHERE cluster_id IS NOT NULL;

Three things in that schema matter more than they look. The cluster row records the model and threshold that produced it, so when the embedding model changes it is obvious which clusters are stale rather than which are wrong. The run_id makes the whole assignment revertible with one UPDATE, which turns a decision about 12% of the corpus into an experiment rather than a migration. And cluster_rank separates the question of which copy to prefer from the question of which copies exist, so preference can be recomputed — by recency, by source authority, by score — without touching cluster membership.

Because nothing is deleted, per-copy permissions survive and the collapse can be applied after the permission filter, so each user sees a representative drawn from the copies they are allowed to read. That is the property the drop policy cannot have at any price, since the deleted rows are not available to be selected for anyone.

Read down the columns and that table looks like eight independent considerations. It is not. Four of those rows — provenance, divergence, per-copy access control and reversibility — are four readings of one underlying fact, which is whether the other copies still exist as rows. Marking answers Yes to all four for that single reason, and the two destructive policies answer No to all four for the same single reason, which is why the pattern in those rows is so uniform. Index size is the only row where a destructive policy wins, and it is also the only row that does not depend on that fact. Once the table is read that way the decision stops being a weighing of eight criteria and becomes one question with a size saving on the other side of it.

The last row is the one that decides it in practice, and it is a multiplication rather than an opinion. Chapter 4 put the false-merge rate at a calibrated cutoff at 3.8%, and there are 4,100 clusters, so roughly 160 of them group documents that should not be grouped. A rate on its own invites the response that it sounds acceptable; a count says what will actually happen. Under the marking policy those 160 mistakes make some result lists slightly worse and are undone by reverting a run id. Under either destructive policy they are 160 documents that no longer exist.

Treating the choice as a single global decision is what makes it feel like a matter of taste. In practice the corpus splits into tiers whose risk profiles differ enough that different policies are correct for each. The exact-hash tier — identical text, identical permissions, source system still connected — can be dropped, and dropping it recovers most of the index saving without any of the exposure, because 470,000 of the payments corpus's chunks were in that tier against 503,000 in the embedding-derived one. Those two figures are close enough to be the whole argument for tiering: the cheap, certain tier is nearly as large as the expensive, uncertain one, so almost all of the saving is available at almost none of the risk. Clusters whose members span more than one access domain should never be dropped or merged at any similarity, since permissions are the one attribute where an approximation is a vulnerability. Clusters flagged by the divergence check are held for review rather than processed automatically. What remains, which is the bulk, goes to marking.

Where marking is paid for#

Writing the tiers down as explicit rules has a benefit beyond getting each one right: it makes the pipeline's behaviour auditable in advance. A reviewer can read four conditions and know what will happen to any given cluster, which is a very different conversation from being told that the job merges things above 0.96.

The honest accounting for the recommended policy has three entries. Storage stays where it was, so none of the infrastructure saving that motivates many of these projects materialises — the vectors, the text and the index structure all remain. Ranking gains a stage, and stages have bugs; the collapse interacts with re-ranking, with pagination, and with any downstream component that assumed the retriever returns exactly k results. And every query now pays for work that used to be done offline, in both latency and in the extra candidates fetched.

That per-query cost is not incidental. Retrieving k results and collapsing them yields fewer than k, so the retriever has to ask for more than k to begin with, and how much more depends on how duplicated the corpus is — which varies by query. Getting this wrong is not a performance problem but a correctness one: the assistant quietly starts answering from six passages when the prompt was designed for ten, and nothing reports it. Building the collapse so that it delivers a full window, choosing which member represents its cluster when the candidates disagree about recency and authority, and fitting the whole thing inside a latency budget, is the next chapter.