Four hundred shared tokens collapse a document's vectors together

A header is prepended text, and prepended text is input. An embedding model handed a chunk with a paragraph glued to its front has no way to treat that paragraph as annotation, because nothing in the interface marks it as one — the model receives a string. Whatever that paragraph contributes to the resulting vector, it contributes identically to every chunk that carries it. This chapter is about what happens when the prepended paragraph is a good one, chosen by people who were right about the previous change, and long enough to dominate the input.

The setting is a corpus of technical documentation for a hardware product line, averaging around 900 chunks per manual, ingested by a pipeline that splits each manual into 800-token chunks and embeds them one batch at a time. Every manual carries a hand-written 400-token abstract at the top describing what that manual covers. Two terms are needed to read any of the measurements below. Recall@10 is the share of a fixed set of test queries whose known-correct chunk appears somewhere in the first ten results. That fixed set is an evaluation set: the list of queries with the chunk that answers each one recorded beside it, decided by a person in advance.

What a shared prefix does to a vector#

An embedding model turns a sequence of tokens into a single vector, and however elaborate the architecture, the output is a pooled function of every token in the input. That word pooled is where the mechanism lives. The model reads the whole input, produces something per token, and then combines those per-token results into one vector of fixed width. A token that is present in two different inputs contributes to both of those combinations.

Start with a pair smaller than the manuals, because the arithmetic is visible at that size. Two inputs that share 400 of their 850 tokens are, at the input layer, nearly half the same document. The model can and does weight positions unevenly, but it cannot manufacture separation that the input does not contain. Sharing a prefix of that size means each input is approximately that prefix, perturbed by its own 450 tokens.

The manuals are the same shape at a different ratio, and the ratio is the only thing that changes. Each chunk is 800 tokens of its own text. Put a 400-token abstract in front of it and the model reads twelve hundred tokens, of which four hundred are the same four hundred on every chunk of that manual. Divide the shared count by the total and a third of what the model reads is a string the chunk did not choose and does not distinguish it from anything. Two chunks of the same manual therefore agree, by construction, on a third of their input before either of them has said a word about torque or reservoirs. Sharing the abstract means every chunk in a manual is now approximately the abstract, perturbed by whatever its own 800 tokens happen to say.

Why agreement at the input turns into agreement at the output is the step worth walking rather than asserting. The vector is pooled from contributions made by every token. Two inputs that share a third of their tokens share the contributions those tokens make. The pooled results therefore share a large common component and differ only in what the unshared tokens contributed. Two vectors built from a large identical component plus a small differing one point in nearly the same direction, since the identical part dominates the sum. The angle between them — which is the quantity a vector store ranks on — is small whatever the chunks say.

The measurement is easy to take and it is the one worth taking before shipping a change like this. Sample pairs of chunks from within the same document and score them against each other:

import itertools, random
import numpy as np


def mean_pairwise_cosine(vectors: np.ndarray, pairs: int = 2000) -> float:
    idx = range(len(vectors))
    sample = random.sample(list(itertools.combinations(idx, 2)), pairs)
    v = vectors / np.linalg.norm(vectors, axis=1, keepdims=True)
    return float(np.mean([v[i] @ v[j] for i, j in sample]))


# same document, 900 chunks
print(mean_pairwise_cosine(vecs_doc_A_heading_path))   # 0.41
print(mean_pairwise_cosine(vecs_doc_A_full_abstract))  # 0.88

# across documents, one chunk from each of 900 different manuals
print(mean_pairwise_cosine(vecs_cross_doc_heading_path))   # 0.19
print(mean_pairwise_cosine(vecs_cross_doc_full_abstract))  # 0.34

Read what that function computes before reading its output. It normalises every vector to unit length, so the dot product of any two of them is the cosine of the angle between them and nothing else. It then draws a sample of distinct pairs and averages those cosines. The number it returns is the typical angular agreement between two chunks drawn from the same population, and the population is chosen by which set of vectors is passed in — one manual's chunks for the first two calls, one chunk from each of many manuals for the second two.

Mean cosine between two arbitrary chunks of the same manual rose from 0.41 to 0.88. A set of 900 vectors with mean pairwise cosine 0.88 is not 900 points, it is a tight ball with 900 labels on it. Whatever information distinguished chunk 412 from chunk 87 is still present in the vector, but it has been reduced to a small residual on top of a large shared component, and a nearest-neighbour search over that residual is a search with most of its dynamic range spent on a constant.

The four printed figures invite a comparison that would be wrong, and it is worth stopping to disarm it. The cross-document figure under the full abstract, 0.34, sits almost exactly where the no-header within-document figure sits in the table further down. Those two numbers are not measuring the same thing and neither is a check on the other: one is the agreement between chunks of different manuals carrying different abstracts, the other the agreement between chunks of the same manual carrying no header at all. Two figures that nearly coincide can be answers to different questions, and the only way to tell is to read which population was sampled.

Information retrieval has known this since long before embeddings existed. A term that appears in every document in the collection carries no power to discriminate between them, which is the entire content of the inverse document frequency factor in TF-IDF, and the reason "the" is not an index term. What the abstract did was insert 400 stopword-equivalents into every chunk of every manual — not stopwords in the vocabulary sense, since the words are technical and specific, but stopwords in the sense that matters, in that they are constant across the population you are trying to separate. The difference is that TF-IDF has an explicit mechanism to discount them and a dense embedding does not.

That is the stumble this chapter exists to name, and it is tempting for good reasons. The abstract passes every check a careful person would think to run: it is accurate, it was written by a human, it is about the right subject, and it is plainly more informative than a heading path. Each of those is a property of the abstract considered on its own. Retrieval never considers a chunk on its own — it ranks a chunk against every other chunk — and the quantity that decides a ranking is a difference between chunks. A string that is the same on all of them contributes nothing to any such difference, however good it is. The general form is a defence that stays constant while the thing it has to tell apart varies, and a constant cannot explain a difference in anything.

What the search actually returned#

Collapse of that kind changes the shape of a result list in a specific and recognisable way. When a query matches a document's abstract even loosely, it matches all 900 of that document's chunks nearly equally, so the top-k fills up with siblings.

Here is a query from the evaluation set, with the same k against both indexes:

query: "torque spec for the M6 mounting bolts on the pump housing"

--- heading-path headers, top 5 ---
0.681  PX-400 Service Manual > Pump assembly > Mounting     [GOLD]
0.644  PX-400 Service Manual > Pump assembly > Torque table
0.612  RX-200 Service Manual > Pump assembly > Mounting
0.588  PX-400 Service Manual > Fasteners > Thread sizes
0.571  General Fastener Guide > Metric > M6

--- full-abstract headers, top 5 ---
0.792  RX-200 Service Manual > Hydraulics > Reservoir bleed procedure
0.791  RX-200 Service Manual > Hydraulics > Filter replacement interval
0.790  RX-200 Service Manual > Electrical > Sensor harness routing
0.789  RX-200 Service Manual > Hydraulics > Pressure test
0.788  RX-200 Service Manual > Appendix B > Revision history

Read the two lists as gaps rather than as scores. In the first list the five results come from four different documents and the scores step down visibly from one rank to the next, which is what it looks like when the chunk bodies are deciding the order. In the second list every result comes from one manual, and the step from each rank to the next has shrunk to a thousandth. The vertical spread of a result list is the room the ranking has to work in, and the second list has almost none.

The gold chunk — the chunk a human labelled as the one that answers this query, which is what the evaluation scores the results against — was at rank 34 in the second index, behind thirty-three other chunks of the RX-200 manual. The RX-200 abstract mentions pump housings and metric fasteners, so it matched the query slightly better than the PX-400 abstract did, and that small advantage was applied uniformly to all 900 of its chunks. The scores in the second list are separated by thousandths, which is the signature to look for: when the top ten results are all from one document and their scores agree to three decimal places, the ranking is being decided by the shared prefix and the chunk bodies are casting a vote too small to matter.

The mechanism that puts the wrong manual on top is worth stating as a rule, because it predicts the next case. Whichever document's abstract matches a query best wins that query's whole result list, and it wins by whatever margin the abstracts differ by, which can be tiny. A single comparison between two documents is deciding the order of nine hundred chunks apiece. Ranking has been moved up a level without anybody choosing to move it: the system is now retrieving documents and returning their chunks in an order it can barely distinguish, and the gold chunk's own text is not what put it at rank 34 or what could have saved it.

Sibling crowding is also why the failure survives a reranker for longer than it should. A reranker is a second-stage model that rescores the candidates the vector search returned before they are handed on, and the usual choice is a cross-encoder, which reads the query and the chunk together in one pass rather than embedding each separately and comparing points. Rescoring the top 50 can only reorder what the first stage handed it, and if all 50 came from the RX-200 manual then the gold chunk was never a candidate. Diversity-aware selection such as maximal marginal relevance will thin the sibling run and pull in a second document, which makes the symptom less dramatic without touching the cause, because the vectors are still collapsed and the second document surfaced is still chosen by abstract similarity rather than by chunk content.

Both of those are worth reading as the same kind of mistake, made one stage apart. A reranker and a diversity policy each operate on a candidate list, and a candidate list is the output of the collapsed stage. Neither can restore information the first stage never used, so neither is a fix, and both are convincing enough to spend a sprint on.

The property a header needs#

The abstract satisfied the wrong criterion. It was relevant, accurate, and about the right subject, and none of those are what a prepended header is for. The header's job is to carry the information that separates this chunk from the chunks it will be competing against, which is a comparative property rather than a property of the chunk in isolation.

Two populations matter, and a good header does something different in each. Against chunks from other documents, the header should identify the document, so that a query naming the PX-400 does not land on the RX-200. Against chunks from the same document, the header should identify the section, so that the ranking within a document is decided by where the chunk sits and what it says rather than by a constant. The abstract does the first job passably and the second not at all — worse than not at all, since it actively drowns out the chunk text that was doing that job unaided.

That gives a diagnostic that is cheap enough to run at ingestion time. Count how many distinct header strings a document produces:

from collections import Counter


def header_diversity(chunks) -> dict[str, float]:
    per_doc = Counter()
    distinct = {}
    for c in chunks:
        per_doc[c.doc_id] += 1
        distinct.setdefault(c.doc_id, set()).add(c.header)
    ratios = [len(distinct[d]) / per_doc[d] for d in per_doc]
    return {
        "mean_distinct_ratio": sum(ratios) / len(ratios),
        "docs_with_single_header": sum(1 for r in ratios if r < 0.01),
    }

The ratio that function builds is distinct headers divided by chunks, computed per document and then averaged, and the reason to divide rather than to count is that a raw count of distinct headers says nothing without knowing how many chunks were asking for one. Sixty distinct headers is excellent across nine hundred chunks and useless across nine hundred thousand. A document that produces one distinct header for 900 chunks has a distinct_ratio of 0.0011 and is contributing 900 identical prefixes to the index. A document whose headings give it 60 distinct headers across 900 chunks sits at 0.067, which is enough to separate sections from each other even though chunks within a section still share a prefix. The number does not have a threshold you can copy; what it has is a floor, in that a ratio near zero means the header is pure ballast, and a ratio near 1.0 means every chunk has a unique header, which is what per-chunk generation produces and what chapter 3 of this topic weighs against the templated alternative.

The crossover moves with chunk length#

Two corpora can run the same header and get opposite results, because what determines the damage is the ratio of shared tokens to distinctive ones rather than the header's length in isolation. Four hundred shared tokens against an 800-token chunk means a third of the input is constant. The same four hundred against a 200-token FAQ entry or a support-ticket message means two thirds of it is, and the residual carrying the chunk's actual identity is a quarter the size.

Both halves of that are the same division done twice, and doing it explicitly is what makes the rest predictable. The shared count sits in the numerator and does not move: four hundred tokens, the same four hundred, whatever the corpus. What moves is the denominator, which is the shared count plus the chunk's own length. Against eight hundred tokens of chunk the denominator is twelve hundred and the quotient is a third. Against two hundred it is six hundred and the quotient is two thirds. Halving a chunk's length does not halve the harm; it raises the shared fraction, and it does so faster the shorter the chunk gets, because the term that is shrinking is the only term that varies. The other reading is the residual: the part of the input that carries identity falls from eight hundred tokens to two hundred, a quarter of what it was, while the part that carries nothing stays exactly where it is.

Rerunning the same measurement on the shorter-chunk half of the corpus made that concrete. On 200-token chunks, the full abstract pushed mean sibling cosine to 0.94 and recall@10 to 0.41, well below the no-header baseline of 0.62; on the 800-token chunks it was the 0.88 and 0.58 already reported. Same abstract, same model, same query set — the difference is entirely the denominator. Corpora built from short units are the ones where a header that seems modest does the most harm, and they are also the ones where an informative header helps most, which is an uncomfortable combination that gets resolved by making the header short and specific rather than long and general.

Vector normalisation does not rescue any of this. Most retrieval setups normalise embeddings to unit length and use cosine similarity, so the collapse shows up as vectors crowded onto a small patch of the sphere rather than as vectors of similar magnitude. The angular separation between siblings is what shrank, and normalising does not restore an angle.

A cheap guard is worth adding to the ingestion pipeline for this reason: assert at build time that the median header is no more than about 15% of the median chunk length for that source, and fail the build rather than warn. Fifteen per cent is a working figure rather than a derived one — the point of the check is that a change which quietly doubles header length gets caught by continuous integration instead of by a recall regression three weeks later.

Laid out against the same dimensions, the three candidate designs separate cleanly:

Run the guard across that table's own rows and it partitions them exactly. Against an 800-token chunk, no header at all costs nothing, the title alone is a little over one per cent of the chunk, and the title with its heading path is under three per cent; all three clear the threshold comfortably. The abstract is more than half the chunk length and fails. Reading the last column beside that verdict, the three designs the guard admits are the three whose recall is at or above the no-header baseline, and the one design it rejects is the only one that falls below it. The guard's threshold was picked by hand, and this is the evidence that it is picked somewhere sensible: applied blind to four designs it would have passed exactly the three that helped and stopped the one that hurt.

What that licenses is narrower than it looks, and the residual is worth stating rather than hiding. The guard measures length, and length is not the mechanism. Nothing in it would object to a header of moderate length that happened to be identical on every chunk, which is a design that collapses vectors while passing the check. The guard and the distinct-ratio diagnostic sit on two different axes, and each is blind to the axis the other measures; a build that runs only one of them is protected against half of this failure.

The table also settles the question of whether length is the variable, and it settles it against length. Compare the two rows in the middle rather than reading each as a level. Going from the title alone to the title with a heading path adds more than twice as many tokens, and mean sibling cosine goes down — from 0.55 to 0.41 — while recall goes up. If tokens were what collapsed the vectors, more tokens could not undo the collapse. The distinct-headers column is the one that tracks the outcome: the two designs with a single distinct header per document produce the two highest sibling cosines in the table, and the design with sixty-one distinct headers produces the lowest of the three headers tested, despite sitting between them on length. Length and redundancy have been separated by the chapter's own measurements, and redundancy is the one doing the damage.

Document title alone already shows the effect in miniature: it helps, because it separates manuals from each other, and it costs sibling separation, because nine tokens are now shared by every chunk in the document. Nine tokens against 800 is a small enough tax that the net is positive. Four hundred tokens against 800 is not, and the crossover is not at a fixed ratio you can look up, because it depends on how much the header discriminates as well as how long it is.

That row carries the honest residual of the whole design. Adding the title raises mean sibling cosine by more than a fifth of the scale, and it is adopted anyway, because the recall it buys between documents exceeds the recall it costs within them. The decision being made there is a net across two populations rather than an improvement in both, and a change that helps one population and harms the other can only be judged on a measurement that reports both.

Truncation, near-duplicates and the sparse arm#

The failure is not truncation. The whole reason it is confusing is that every diagnostic aimed at input limits comes back clean, and a team that has been burned by silent truncation before will spend a day proving that it is not happening again. Raising the embedding model's context window makes this worse rather than better, because the constraint that would have stopped the abstract at 500 tokens is gone.

It is not the near-duplicate problem, where a corpus genuinely contains the same passage in five places and the index dutifully returns all five. That is a deduplication concern with its own fixes at ingestion. Here the chunk bodies are entirely distinct and the vectors are near-duplicates anyway, which is why deduplicating by content hash finds nothing.

Those two together define the shape of the trap. One family of checks asks whether the pipeline did what it was told, and it did. Another asks whether the stored content is distinct, and it is. The thing that is duplicated is neither the instruction nor the content but the input, which is assembled at embed time, lives for one call, and is not written down anywhere a later check can inspect it. A team without a habit of logging the exact embed input has no artefact to look at, which is why the pairwise-cosine sample ends these investigations and reading the code does not.

It is also not a lexical problem, which is why a hybrid index masks it partially and confusingly. BM25 discounts terms that appear in every document of the collection, so the abstract's vocabulary is downweighted automatically on the sparse side and the lexical arm keeps working. A team running hybrid retrieval therefore sees a smaller regression than a dense-only team and can spend a week concluding the embedding model has degraded, when what has happened is that the dense arm stopped contributing and the sparse arm is carrying the whole result. Score the two arms separately when a hybrid system regresses; a dense arm whose top-10 is all one document is diagnostic.

That contrast is the cleanest statement of the whole chapter, because the two arms were given the same input and only one of them was harmed. BM25 carries an explicit term that discounts anything appearing everywhere, so a constant prefix is priced at nothing on the sparse side by construction. A dense encoder has no such term, and a constant prefix arrives at full weight. The lesson generalises past headers: any component that pools its input without a mechanism for discounting what is common to the population will be damaged by adding something common to the population.

And it is not an argument that longer headers are always worse. A 60-token header that says something different about every chunk costs more tokens than a 22-token heading path and collapses nothing, because none of those 60 tokens are shared. Length and redundancy are separate axes, and the abstract was catastrophic on the second one while being merely expensive on the first.

Prepended context is read as content, then, and it is priced as content in every comparison the store makes. That is why the question to ask of a candidate header is never whether it is accurate or useful in isolation. The question is what it adds to the difference between two chunks that will be ranked against each other, and a string identical on both adds nothing to their difference while adding its full weight to what they have in common. Four hundred good tokens shared across nine hundred chunks bought a tight ball of vectors with nine hundred labels on it, a result list separated by thousandths, and a gold chunk queued behind thirty-three of its own siblings.

Which leaves the question of how to produce a header that varies per chunk without paying to think about every chunk individually. The heading path varies for free wherever a document has headings, and there is a definable population of chunks where it does not vary, does not exist, or does not mean anything — and that population is what decides whether a model has to be involved at all.