One pooled vector gets blurrier as the passage it summarises gets longer

Capping the length of the pieces you index removes one cause of poor long-document retrieval and leaves a second one standing, and the second one is worth being able to name because it changes what a disappointing evaluation result means. A team that has fixed the first cause and measured the second without naming it will keep tuning the thing it already fixed. The second cause lives in the step that turns a passage of text into a single vector, and that step is arithmetic you can watch happen.

Four terms carry the rest of this. A retrieval system stores each passage as a vector — an ordered list of numbers of fixed length — produced by an embedding model, and answers a query by embedding the query with the same model and returning the stored vectors nearest to it. Nearness here is cosine similarity, a number that is largest when two vectors point the same way and falls as the angle between them opens. Chunking is the step that decides what counts as a passage, cutting a long document into pieces that are embedded and stored separately. Truncation is the defect where a piece longer than the model's sequence limit has its tail silently discarded before the model sees it; capping chunk lengths under that limit removes it, and removing it is what exposes the defect this chapter is about.

What pooling does to a passage#

A sentence encoder produces one vector per token and then collapses them into a single vector of fixed size, usually by averaging over the non-padding positions. A token is the unit the model actually reads — roughly a word or a fragment of one — and the padding positions are the empty slots a short input is topped up with so that every input in a batch has the same shape, which is why they are excluded from the average rather than counted as zeros. The averaging is done coordinate by coordinate: the first component of the result is the mean of every token's first component, the second component is the mean of every token's second, and so on to the end of the width.

That average is a centroid, and the geometry of a centroid is the whole mechanism. The mean of a set of points always lies inside the region those points occupy — it cannot sit outside them, because it is built from them. So when the token vectors of a passage cluster in one place, their mean lands in that cluster and the passage's stored vector sits on its subject. When they cluster in four places, the mean lands between the four clusters, and a centroid of a set of points related to four different subjects sits in the middle of all four rather than on any of them.

The intuition transfers from something less abstract, and it is worth taking as an analogy with its limits stated rather than as a description of what the model does. Take the average colour of a photograph and you get a single value that summarises the whole image reasonably well when the image is one thing — a close-up of a lemon averages to yellow, and if you were sorting images by "is this yellow" you would rank it first. Take the average colour of a photograph of a street scene, with sky and brick and asphalt and a red awning, and you get a muddy brown that is not the colour of anything in the picture. Two completely different street scenes average to nearly the same brown, so the average stops distinguishing them, and a query for "red awning" does not find the one that has one.

A pooled embedding of a 500-token passage covering a service's authentication, its rate limits, its pagination and its error codes behaves the same way. The vector is a real point in the space and it is genuinely near all four subjects, which is exactly the problem: near is not the same as on top of, and there are thousands of other vectors in the index that are also near.

The analogy breaks in two places, and both places are where a reader would otherwise form a wrong belief. The first is what is being averaged. The pixels of a photograph carry no meaning individually and know nothing of each other, so a mean colour is genuinely ignorant of everything structural in the image. A pooled embedding averages token vectors that were each computed with the rest of the passage visible, so a pronoun's vector already carries something of its referent and a heading's vector already carries something of the section under it. The pooled vector is therefore less structurally ignorant than a mean colour is, which is exactly the property the late-chunking work later in this chapter exploits. The second break is width. A colour has three channels and the pooled vector has hundreds, so it retains far more of its input than a mean colour retains of an image. That is why dilution degrades a match rather than destroying it: the diluted passage still scores well above nothing, still has a well-formed vector, and still appears somewhere in the ranking. It loses a race rather than disappearing, and the two mixed passages that the analogy says average to the same brown do not in fact get the same vector.

The capacity argument#

There is a second way to state the same thing that some people find more convincing, and it is worth walking one link at a time because each link is checkable on its own. A 768-dimensional float vector holds the same amount of information whether it was produced from 20 tokens or 500. The fixed width is a design decision and not a law — nothing in the mathematics forces a model to return the same number of components for every input — and it is made because an index compares any two vectors with one arithmetic operation of fixed cost, which requires every stored vector to have the same shape. Given that decision, nothing that follows involves another choice.

The next link is a division. The components available per token of input are the fixed width divided by the number of tokens, and that quotient falls as the token count rises: 768 components spread over 20 tokens is a generous budget each, and the same 768 spread over 500 is a thin one. Something has to give, and what gives is detail. Averaging preserves what the tokens have in common, because a property shared by most of the token vectors survives being summed and divided, and it erodes what only a few tokens carried, because a handful of values is diluted by the many they are averaged with. The specific phrase that answers a query is carried by a few tokens. The passage's broad subject is carried by most of them.

The last link is what makes this awkward. Every step above followed from the width alone, and none of them mentioned what the model was trained on or how well it was trained, so this is a compression bound and not a training deficiency, and it is not fixed by a better checkpoint or by more fine-tuning data. Higher dimensionality raises the ceiling somewhat — which is one of the things the storage-cost trade in topic 8.6 was buying, where a wider vector's index and storage bill is weighed against what the extra components buy — and it raises the ceiling without changing the shape of the argument, since the quotient still falls as the input grows. A 3,072-dimensional vector of a 4,000-token document is still a summary of a document, and a query about one sentence inside it is asking the summary a question it does not contain.

Three encodings of one paragraph#

The mechanism makes a prediction that can be checked directly, by holding the text that contains the answer fixed and changing only what surrounds it. Concretely, taking one paragraph about token refresh and embedding it three ways:

Read the first three rows as gaps rather than as levels, because the levels depend on the model and the gaps do not. Going from the paragraph alone to the paragraph inside a focused section takes 0.791 down to 0.712, a drop of a little under eight hundredths, and it was bought by growing the input from 96 tokens to 500 — a bit over five times as much text. Going from the focused section to the mixed section takes 0.712 down to 0.583, a drop of a little under thirteen hundredths, which is over half again as large as the first drop. That second drop was bought with no extra length at all: both passages are 500 tokens.

What that comparison licenses is the point of running it. If length alone were the driver, rows two and three would have fallen by the same amount, since they contain the same number of tokens averaged into the same fixed width. They did not, so the extra damage in row three has to come from the one property that separates the two passages, which is the number of subjects the section covers. Length costs something and multi-topic content costs more, and it costs more per unit of the change that produced it: a fivefold increase in length bought the smaller drop, and going from one subject to four at constant length bought the larger one.

Producing that table for your own corpus is a short script, and it is the measurement that tells dilution apart from every other explanation, because the passage holding the answer is identical in all three arms and only its surroundings change:

para = open("token-refresh-paragraph.txt").read()          # 96 tokens
focused = open("auth-section.txt").read()                  # 500 tokens, one subject
mixed = open("api-overview-section.txt").read()            # 500 tokens, four subjects
competitor = open("key-rotation-paragraph.txt").read()     # 88 tokens, wrong answer

q = "Represent this sentence for searching relevant passages: " \
    "how do I refresh an expired access token"

vq, *vs = model.encode([q, para, focused, mixed, competitor],
                       normalize_embeddings=True)
for name, v in zip(["alone", "in focused", "in mixed", "competitor"], vs):
    print(f"{name:<12} {vq @ v:.3f}")

Each of the three containing passages is well under the 512-token limit, so the boundary sweep from chapter 2 — where an input is grown a little at a time until the vector stops changing, and the length at which it stops is the model's effective limit — reports nothing at all for any of them.

The last row is the one that decides the outcome. At 0.583 the correct passage now ranks below a passage about a different subject entirely, because the competitor is short and focused and therefore sits closer to its own topic than the diluted chunk sits to its. What makes the row decisive is where 0.604 falls relative to the others: it sits above the mixed figure of 0.583 and below the focused figure of 0.712, so the wrong-answer passage beats the right one when the right one is diluted and loses to the same right one when it is not. The competitor is 88 tokens against the paragraph's 96, so length is not what separates them either. Nothing was truncated in any of these; every token reached the encoder.

The gap that survived the re-index#

Two defects, one symptom#

Truncation and dilution both present as "long documents retrieve badly", and they are separated by what happens to the tokens.

Splitting is the standard response to dilution and it trades one problem for another rather than removing it. A 4,000-token document split into ten 400-token chunks gives each chunk a tighter topic, so the chunk containing the answer scores higher than the whole document did. It also spreads the document's evidence across ten vectors, so a query needing two facts from opposite ends of the document now needs two chunks to be retrieved together, and the surrounding material inside each 400-token chunk still pools with the answer sentence. Push the chunks smaller and the pooling improves while the context each chunk carries gets thinner, until chunks become sentences that cannot be interpreted without their neighbours — the pronoun with no referent, the configuration value with no setting name. Two quantities moving in opposite directions as one knob turns is what produces an interior optimum, it is corpus-specific, and it is usually somewhere between 200 and 400 tokens for prose.

The response that gets proposed first in most rooms is to raise k — the number of top-scoring chunks the retriever hands on to whatever consumes them — on the reasoning that long documents are being edged out and more slots will let them in. It is worth working through why that buys so little. Against truncation it buys nothing at all, because the evidence is absent from the index and a longer candidate list cannot contain text that was never encoded; moving from k=10 to k=50 on the clipped incident postmortem from chapter 3, whose only statement of root cause sat in a final section the encoder never saw, still misses a chunk at rank 214. Against dilution it buys something, and the something is small and expensive.

Putting the two sides of that exchange next to each other is what settles it. A diluted chunk sits at a mediocre similarity in a region of the ranking that is dense with other mediocre matches, so going from k=10 to k=50 might raise the chance of catching it from 0.61 to 0.68 while quintupling the context handed to the generator, which costs tokens, latency, and precision in the answer. What was bought is seven hundredths of recall. What was paid is five times the retrieved context, and it is paid on every query, while the seven hundredths accrue only on the minority of queries whose diluted chunk happened to be sitting between rank ten and rank fifty. That asymmetry is what makes it a bad knob rather than merely a weak one: a weak knob moves the number a little for a proportionate price and can be turned further, whereas here the price rises with k on every query while the return flattens, because each extra slot admits many more mediocre competitors than wanted chunks. Turning it further makes the exchange worse, and it leaves the defect exactly where it was, so the next chunk-size change starts from the same place.

The instruments that report clean either way#

The tempting mistake at this point is to believe the problem is already instrumented. This topic builds two real instruments — the boundary sweep of chapter 2, which finds the length at which a model stops reading, and the per-chunk fields of chapter 6, which record a token count, the effective limit in force, a truncation flag and a retained-token ratio for every chunk at index time. Both work. A reader who has just been taught to cap chunk lengths and count tokens will reasonably conclude that the whole family of long-document failures is now visible on a dashboard.

Run those instruments over the passages in the table above and the belief comes apart. Every containing passage there is well under the 512-token limit, so the sweep finds nothing to report, the truncation flag is false, and the retained ratio is one. That reading is identical for the focused 500-token section, which retrieves the right answer, and for the mixed 500-token section, which loses to a passage about the wrong subject. The instrument returns the same value whether or not the fault is present, which is precisely what a defence must not do: its output is constant while the thing it is being asked to discriminate varies. Nothing about the reading is wrong — the chunks really are under the limit — and that is what makes it comfortable to trust.

What can discriminate dilution is the last row of the table above, and it is worth being explicit about why it cannot be run on a chunk sitting in an index. Comparing the same passage embedded alone against the same passage embedded in context requires two encodings of one piece of text — one where the paragraph is the entire input, one where it is a part — and a query to score both against. A chunk in isolation supplies exactly one of those three: it is a single encoding, there is no second one to compare it with, and nothing stored alongside it says which queries will be asked of it.

So the closing observation follows from the measurement rather than being asserted: dilution has no per-chunk observable. Truncation announces itself the moment you count tokens: a chunk is either over the limit or it is not, and the fields chapter 6 emits make that a number on a dashboard. Dilution is a property of the relationship between a chunk's content and the queries people will ask of it, so there is no measurement you can take of a chunk in isolation that tells you it is too blurry. Proxies exist — the variance of a chunk's token vectors around their own mean, the number of distinct headings a chunk spans — and they are weak enough that nobody alerts on them. Detection is empirical and query-driven, which means it costs an evaluation run rather than a log line.

Where the field has got to#

Three approaches attack dilution rather than working around it, and they are worth knowing about even where they are out of scope for the system in front of you.

Long-context embedding models push the sequence limit to 8,192 tokens or beyond, and it is important to be precise about what that buys. It removes truncation for almost any realistic chunk, and it does nothing whatsoever for dilution — pooling 8,000 tokens into one vector produces a blurrier average than pooling 500, so a naive migration to a long-context encoder with chunk sizes raised to match can make long-document retrieval worse while eliminating every truncation event. That combination is a good way to convince a team that truncation was never the problem.

The trap there is a design decision rather than a property of the model, and separating the two is what stops it. Raising the sequence limit is something the model does and it is unambiguously good, since it removes a defect and adds none. Raising chunk sizes to match the new limit is something a person chooses afterwards, usually because the old cap looks like an obsolete workaround once the limit moves. That second step pools more tokens into each vector, so it trades a defect that was already solved for one that is worse and that no instrument reports. The truncation counter of chapter 6 goes to zero at the same moment retrieval quality falls, which is the most misleading pair of signals in this topic.

Late chunking keeps the long-context encoder and moves where the pooling happens. The whole document goes through the encoder once, producing contextualised token vectors, and chunk boundaries are applied afterwards, pooling only over each chunk's own token positions. Each chunk vector is then focused on its own text while having been computed with the whole document visible, so the pronouns and elliptical references have their referents.

The third line is late interaction, which declines to pool at all: ColBERT-style retrievers store one vector per token and score a query against a passage by maximum similarity per query token, summed. Dilution largely disappears because nothing is averaged, and the cost is an index one to two orders of magnitude larger and a scoring step that is no longer a single dot product. Stop Losing Context! How Late Chunking Can Enhance Your Retrieval Systems is a reasonable entry point to the long-context and late-chunking material if you want to go further than this.

Dilution is what a fixed-width average does to a passage that is about more than one thing. The vector is the centre of its input, the centre of a focused passage sits on its subject, and the centre of a mixed passage sits between subjects where thousands of other vectors also sit. That is enough to put a correct passage below a wrong one in a ranking, as the third and fourth rows of the cosine table do, without a single token being lost and without any check that counts tokens noticing. The defect is invisible to every instrument that looks at one chunk on its own, because it is a fact about a chunk's relationship to a query rather than a fact about the chunk.

None of the three approaches above is a prerequisite for the work at hand. The immediate value of naming dilution is diagnostic: it tells you which part of a disappointing result is still explainable and which part is not, so that a chunk-size change that improves long-document recall by 22 points and leaves 4 points on the table is a finished piece of work rather than an open question. Distinguishing those two numbers requires an experiment designed to attribute the delta, which is what the next chapter builds.