Two clusters of high-similarity pairs are not duplicates at all

A self-join over a corpus of embedded passages returns pairs and a score for each, and the scores at the top of that file are very high. The instinct is to read a high score as a strong claim about duplication, and the arithmetic supports that reading, so the next move is usually to pick a cutoff and start merging. What the arithmetic cannot report is the reason two passages are alike, and there are reasons that have nothing to do with one being a copy of the other.

Three of those reasons account for most of a candidate file. Two consecutive chunks of one document overlap because a chunker was configured to make them overlap. Two different pages of a documentation site share a template, so most of their words are furniture rather than content. And a chunk of five words carries almost no content at all, so it lands near every other chunk of five words. Each of these produces scores at or above the level a genuine copy produces, which means a threshold placed anywhere admits them, and each is identifiable from information the pipeline already holds. This chapter is about that information and about what happens to a clustering step when one wrong pair gets through.

Excluding by document identity, not by connector#

Start with the pair that dominates the top of a candidate file:

A (0.9812)  ...refunds are processed to the original payment method. Card
            refunds settle in 3–5 business days depending on the issuing
            bank. SEPA refunds settle next business day. If the original
            method is closed, the finance team issues a bank transfer
B (0.9812)  refunds settle next business day. If the original method is
            closed, the finance team issues a bank transfer against the
            IBAN on file. Transfers require a completed W-8BEN for
            non-EU beneficiaries and are batched twice weekly...

Those are chunks 7 and 8 of confluence/BILLING/refund-processing, produced by a chunker configured with 512-token windows and 64 tokens of stride overlap. Two things drive the score and it is worth separating them. The stride overlap means a run of text at the end of A is literally present at the start of B, so part of the similarity is shared words by construction. The rest is that consecutive prose about one subject uses one subject's vocabulary throughout, so even the non-overlapping halves point the same way. The chunker was told to produce this and it did.

Adjacent chunks are the largest family and the easiest to remove, because the information needed to remove them is already in the row. A pair whose two members carry the same document identifier is not a cross-source duplicate and never was, whatever it scores.

DELETE FROM candidate_pairs cp
USING chunks a, chunks b
WHERE cp.a_chunk = a.chunk_id
  AND cp.b_chunk = b.chunk_id
  AND a.doc_id = b.doc_id;

On a payments corpus of several million chunks that single predicate took the 41,000 pairs above 0.97 down to 6,900, and the full 61M candidate file down to 22M. Read those two reductions against each other rather than separately. The predicate removed roughly five sixths of the top band and roughly two thirds of the whole file, which says same-document pairs are concentrated at the high end of the score range. That concentration is the reason a sample drawn in score order is dominated by them, and it is the reason the first look at a candidate file is so much more discouraging than the file deserves. It is worth applying the exclusion at the candidate stage rather than filtering it later, because every pair you carry forward is a pair someone may have to look at.

The subtlety is what doc_id is allowed to mean, and getting it wrong breaks the pipeline in one of two directions. Excluding by connector — treating everything from Confluence as one source and refusing to compare within it — removes the intra-connector duplicates that are usually the most common kind, since the same runbook pasted into two Confluence spaces is exactly the case you are hunting. Excluding by URL is too narrow in the other direction whenever the ingest splits one logical document into several: an 80-page PDF handbook loaded page by page arrives as 80 documents with 80 distinct identifiers, and its consecutive pages then look like cross-document duplicates for the same reason consecutive chunks did.

Excluding these pairs is not the same as saying they are harmless. Two adjacent chunks that share 64 tokens of stride can both land in a top-10 and consume two slots for one and a half passages' worth of evidence, which is the crowd-out mechanism of chapter 1 arriving from inside a single document. The remedy is different, though, and it belongs to the retrieval layer rather than to deduplication: collapse consecutive chunks from the same document into a single contiguous passage before assembling the prompt, or set the stride to zero and rely on a sentence-aware splitter. Treating them as duplicates instead would mean choosing one chunk and discarding the other, which throws away half of a passage that was split precisely because it did not fit in one.

The identifier that behaves correctly is the one that names the thing a human would call a document: the wiki page, the ticket, the whole PDF. If the ingest pipeline does not already record it, it has to be added, and adding it is cheaper than every downstream consequence of not having it. Where a corpus genuinely has no such notion — a stream of chat messages, for instance — the workable substitute is a positional exclusion: drop pairs from the same ingest batch whose chunk ordinals are within a small distance of each other, which catches the overlap seam without silencing real duplicates from elsewhere in the stream.

A template can carry most of a similarity score#

The 6,900 surviving pairs contained the second family, and it does not yield to any filter written in terms of identity, because these pairs genuinely are different documents.

A  Release 3.13.2 — 14 March 2024
   This release contains security and stability fixes for all supported
   platforms. Upgrade instructions are unchanged; see the upgrade guide.
   Fixed in this release:
     - CVE-2024-2961 — buffer overflow in the iconv conversion path
     - CVE-2024-3094 — malicious payload in an upstream build artefact
   Known issues: none.
   Support policy: the 3.13.x line receives security fixes until 30 June 2025.
   Questions go to support@example.com or the #platform-releases channel.

B  Release 3.14.0 — 2 May 2024
   This release contains security and stability fixes for all supported
   platforms. Upgrade instructions are unchanged; see the upgrade guide.
   Fixed in this release:
     - CVE-2024-21626 — container escape via a leaked file descriptor
   Known issues: TLS session resumption is disabled on the 3.14.0 line.
   Support policy: the 3.14.x line receives security fixes until 30 Sept 2025.
   Questions go to support@example.com or the #platform-releases channel.

Cosine 0.962. Around 70% of the tokens are identical because they come from a template, and the tokens that differ are the version numbers, the dates and the CVE identifiers, CVE being the public catalogue that gives each disclosed vulnerability a number people search by — which is to say, everything anybody queries these pages for. The query was CVE-2024-21626 fixed has exactly one correct answer among these two documents, and a pipeline that merged them at 0.962 would have deleted or hidden one of the two, half the time the wrong one.

The inversion in that paragraph is worth holding onto, because it is what makes this family dangerous rather than merely annoying. The tokens carrying the score and the tokens carrying the answer are disjoint sets. Similarity is computed over all the text, so the seven-tenths that is furniture dominates it, while the query a user actually types touches only the three-tenths that differs. A number computed over the whole passage cannot report on a distinction that lives in one part of it.

Merging templated siblings is worse than leaving duplicates in place, because a duplicate wastes a slot and a false merge removes evidence. It also fails silently and permanently: nothing downstream notices that the 3.13.2 page stopped being retrievable, and the person who eventually notices is a customer asking why the assistant said a CVE was unfixed.

The signal that separates this family from real duplicates follows directly from the inversion. If the score is being carried by shared scaffolding, then removing the scaffolding should collapse it, and if the score is being carried by shared content, removing scaffolding should leave it alone. The score computed after that removal is the pair's residual similarity, and the gap between it and the raw score is what the rest of the pipeline uses. Boilerplate here means the lines a document carries because of where it lives rather than because of what it says: a navigation trail, a support footer, a legal notice, the standing paragraphs of a template. Identify it empirically rather than by hand — a line that appears in a large fraction of documents from the same source family is boilerplate regardless of what it says — then re-embed what remains and compare the two numbers.

from collections import Counter

def boilerplate_lines(docs: list[str], min_df: float = 0.05) -> set[str]:
    """Lines appearing in >= min_df of documents in this source family."""
    df = Counter()
    for d in docs:
        for line in {ln.strip() for ln in d.splitlines() if ln.strip()}:
            df[line] += 1
    cutoff = min_df * len(docs)
    return {ln for ln, n in df.items() if n >= cutoff}

def residual(doc: str, boiler: set[str]) -> str:
    return "\n".join(ln for ln in doc.splitlines() if ln.strip() not in boiler)

boiler = boilerplate_lines(release_notes_family)        # 412 docs
r_a, r_b = residual(doc_a, boiler), residual(doc_b, boiler)
raw  = cosine(embed(doc_a), embed(doc_b))               # 0.962
res  = cosine(embed(r_a),   embed(r_b))                 # 0.583

Run that over samples from each family and the two behave differently in a way that is stable enough to act on:

Read the Raw column alone and the table says nothing: every row is between 0.958 and 0.994, and the templated pairs are not even the highest. Read the Drop column, which is Raw minus Residual, and the five rows split into two groups separated by more than an order of magnitude — the two templated pairs lose most of a third of their score and the three genuine copies lose a few thousandths to a few hundredths. A genuine copy keeps its similarity when the boilerplate goes, because the boilerplate was never carrying it. A templated sibling loses 0.3 to 0.4, because the template was most of what the two documents had in common.

The last column is what turns that from an observation into a rule you can rely on. Shared-line fraction is counted, not scored: it comes from comparing lines of text and has no embedding in it anywhere. It orders the five rows exactly as the Drop column does. Two instruments built on different principles agreeing on the same split is much stronger evidence than either one alone, because a coincidence would have to fool both. Using the drop as a second condition — merge only when the raw score is above the threshold and the residual score is within about 0.05 of it — removed 94% of the templated pairs from the payments candidate set while costing 1.2% of the true mirrors.

Those true mirrors are the informative failure, and the reason they broke the pattern is stated rather than guessed: they were pages so short that stripping their shared header left almost nothing behind. The test assumes there is content underneath the furniture, so a passage that is almost entirely furniture makes the residual score meaningless in both directions. Knowing which population the test fails on is worth as much as knowing its overall rate, because it can be excluded by length rather than tolerated as noise.

Boilerplate stripping is worth doing at index time for its own sake and not only as a diagnostic, since a chunk that is 70% navigation furniture is a chunk whose embedding is 70% about navigation furniture. Doing it at index time also removes this family of false positives before the self-join ever runs, which is the cheaper place to solve it. The diagnostic version above exists for corpora where re-chunking the whole index is not on the table this quarter.

Chunks too short to be about anything#

A third family shows up in every corpus and produces the highest scores of all. Chunks of five to twenty tokens — ## Overview, See also, Back to top, a table header row that survived extraction, a single line reading Updated: 2024-03-14 — score 0.99 and above against each other in quantity, because in a space where every direction means something, two near-contentless strings land in nearly the same place.

On the payments corpus, chunks under 25 tokens were 3.1% of the index and generated 19% of all candidate pairs above 0.97. Those two figures are the whole finding, and only their ratio says anything: a population occupying about a thirtieth of the index is producing about a fifth of the top band, which is roughly a sixfold over-representation. A family that is over-represented by that much in the top band and contains nothing anyone would act on is a family worth excluding by rule rather than by score. None of them were duplicates worth acting on in any useful sense; they were fragments of extraction. Dropping candidate pairs where either member is under a minimum length is crude and correct, and the threshold is not delicate — anywhere between 20 and 40 tokens behaves the same on prose. The right long-term fix is upstream, in a chunker that merges undersized fragments into their neighbours rather than emitting them, but the candidate filter costs one predicate and can go in today.

One admitted pair can merge two clusters#

The union-find step that turns pairs into clusters has a property that makes every false positive above more expensive than it looks. Connected components are transitive: if A and B are linked and B and C are linked, A and C are in the same cluster whether or not anything ever compared them. A single wrong link therefore does not add one bad merge, it fuses two entire groups.

On the payments corpus this showed up as a cluster with 8,400 members. Its minimum pairwise similarity, computed after the fact, was 0.62 — the cluster contained pairs of documents that were not remotely alike. The chain ran through a family of weekly operations reports, each week's report scoring 0.968 against the previous week's because they shared a template and a rhythm, so ninety consecutive weeks formed a rope in which every adjacent link was above the cutoff and the ends had nothing in common. One end of that rope also touched a quarterly summary that shared a header with an unrelated finance template, and from there the component swallowed a second family.

Every link in that rope satisfied the threshold and the finished cluster satisfies nothing, which is the property to hold onto: a connected component makes a claim about adjacency and is read as a claim about membership. Nothing checked the ends against each other because nothing ever had to.

Chaining is diagnosable without labels. For each cluster, sample pairs of members and compute the minimum and mean pairwise similarity; a cluster of genuine copies has a minimum close to its formation threshold, while a chained cluster's minimum falls far below it. On this corpus, 96% of clusters had a minimum pairwise similarity above 0.94 and the remaining 4% held 71% of all clustered chunks.

Those two percentages count different things, and noticing that is the point of quoting them together. The first counts clusters and the second counts chunks, so a tiny share of clusters holding most of the chunks means the bad components are enormous rather than numerous. That is the signature of a few ropes rather than of widespread noise, and it changes what to do: a handful of components can be inspected by hand, where a broad low-grade error rate could not be.

Two fixes are available and they cost different things. Complete-linkage clustering requires every pair inside a cluster to exceed the threshold, which eliminates chaining by construction and costs a pairwise check inside each candidate component — affordable, since components are small once the ropes are gone, and awkward on the ropes themselves, which is where the cost lands. The cheaper alternative is to keep connected components and reject any component whose minimum pairwise similarity falls below a floor, splitting it by removing the weakest links until the constraint holds. Applying the complete-linkage constraint broke the 8,400-member cluster into 1,900 clusters with a maximum size of 46, and left the count of clusters over the whole corpus at 4,100 rather than the 2,600 that connected components alone had reported.

Those last two figures do not add up in the direction one expects, and the discrepancy is informative. Splitting one component into 1,900 pieces should raise the corpus-wide count by nearly that many on its own, yet the corpus-wide count rose by rather less. Complete linkage cannot merge anything, so the only way the total can fall short is if the constraint elsewhere reduced components to fragments small enough that they stopped counting as clusters at all. In other words the constraint did two jobs: it broke the ropes apart, and it dissolved a number of weakly-held components into singletons.

Raising the threshold makes the trade worse#

The tempting response to a candidate set full of things you would not merge is to demand more similarity. It is the wrong lever, and it is wrong in a way the numbers make plain. Moving the payments corpus cutoff from 0.97 to 0.985 removed 61% of the templated pairs and 58% of the genuine cross-source mirrors, because a PDF export with a page-number footer and one corrected sentence scores 0.976, comfortably below the new line, while two release-notes pages from a verbose template score 0.987, comfortably above it. Pushing to 0.99 left 340 real duplicate clusters out of the 4,100 the corpus contained and still admitted templated siblings from the longest templates.

Those two percentages are nearly the same number, and their being nearly the same is the finding. A lever that removes both populations at the same rate does not discriminate between them at all; it only makes the candidate set smaller, and it reports the same reduction whether the pairs it dropped were copies or siblings. That is the defence which stays constant while the thing it is supposed to separate varies, and it is tempting precisely because the number moving in the right direction feels like evidence. What is actually happening is that the two score distributions overlap, so a line drawn anywhere cuts through both.

That is exactly why the exclusions in this chapter are applied before a threshold rather than instead of one. Same-document pairs, templated siblings and undersized fragments are removed by rules about provenance — which system and which document each chunk came from — and about structure, which they obey reliably. Those properties do not overlap the way scores do: a pair either shares a document identifier or it does not. What remains is a candidate set where similarity means what you want it to mean, and only then is a number worth arguing about.

After the three exclusions, the payments candidate set held 2.4M pairs, of which 4,900 sat above 0.97 and 180,000 above 0.90. Somewhere in that range is a line that says merge above here and leave alone below here, and the value of that line is not 0.95 because a blog post said so. Cosine 0.95 on one embedding model and cosine 0.95 on another describe different degrees of relatedness, and the only thing that turns a number into a defensible number is a set of labelled pairs. Building that set, and reading the false-merge and false-miss rates off it, is the next chapter.