Truncation removes the tail, where the conclusion lives

An embedding model — the model that turns a passage of text into the list of numbers a vector search compares — encodes a fixed maximum number of tokens in one call, a token being one of the pieces the model's tokenizer cuts text into, mostly whole common words and fragments of anything rarer. When a chunk, meaning a passage of a document sized to be embedded and stored as a single vector, arrives longer than that ceiling, almost every client library keeps the first tokens that fit, drops the rest, and returns an ordinary-looking vector with no exception and no warning. The ceiling itself is architectural. Keeping the prefix rather than the suffix is a design decision on top of it: a library could as easily raise an error, split the input, or keep the last tokens instead, and the three behaviours a real client might have are chapter 2's subject (the tail test, which embeds a text twice with a distinctive ending appended and compares the two vectors). What the common decision guarantees is that the survivor is always the opening.

That guarantee is what makes the cost of truncation hard to estimate, because the obvious estimate is the fraction of text removed while the number that matters is the fraction of queries that stop working. Those two are different numbers, and they differ by a large factor in the direction nobody guesses. This chapter is about why they come apart and by how much. Recall, throughout, is the share of queries whose supporting text actually comes back from the search.

Percentage of tokens lost is not percentage of value lost#

Clipping is positional because a maximum length is enforced by keeping the first n positions and no others. One document makes the shape concrete before the general rule arrives. An incident postmortem in a runbook repository — a directory of operational documents kept by the team that runs the service — is 780 tokens long. Its first two thirds are the timeline: alert fired at 04:12, on-call paged, dashboards checked, service restarted, alert cleared at 04:51. That section contains the words every postmortem contains — latency, alert, restart, degraded, customers, mitigation. The last 190 tokens are the section headed "Root cause", and they are the only place in the document where the phrase connection pool exhaustion in the reconciliation worker appears, along with the specific setting that caused it and the ticket number of the fix.

Indexed as a single chunk against a 512-token encoder, the document is clipped at 510 and the root cause section is gone. The arithmetic of that clip is worth doing slowly, because it closes exactly. Of the 780 tokens, 510 survive, and 510 out of 780 is a bit under two thirds — that ratio is the 65 percent of its text the chunk retains. The tokens that do not survive are 780 minus 510, which is 270, and 270 out of 780 is the 34 percent by which the stored text is shorter than the document. Those two percentages are one subtraction read from its two ends. The token counts behind them sum back to 780 with nothing left over, and each percentage is that count against the same denominator, written to whole percent and rounded down.

What a closing identity like that licenses is a negative claim, and the negative claim is the useful one here. Every token the encoder failed to see is accounted for by its position in the document, and none of it is accounted for by anything the model did. There is no residual: no third bucket of tokens that went missing somewhere in the tokenizer, the pooling step or the distance calculation. The vector that comes back is a correct encoding of a correct prefix. Any search behaviour that changes when the tail is removed therefore changes because of the missing text and for no other reason, which is what makes the experiment later in this chapter a clean one — it has exactly one variable in it.

What sits at the end of a document is a property of how humans write, and the pattern is consistent across the document types that end up in retrieval corpora.

The left column and the right column differ in a way that matters for retrieval beyond their content. Openings are generic, because they establish shared context using the vocabulary of the domain, and every document in the corpus establishes roughly the same context. Endings are specific, because that is where a document says the thing that distinguishes it from its neighbours. Losing the end therefore removes the part of the vector's input that made it different from a hundred other vectors, and leaves the part that made it look like all of them.

That asymmetry is why the arithmetic misleads, and the misleading form of it deserves writing down in its strongest version before it is taken apart. We only lost a third of the chunk, so we lost about a third of its usefulness. It is a reasonable-sounding sentence and it is tempting for a real reason: the token arithmetic genuinely is linear. Two thirds retained plus a third discarded really does account for the whole document, as the paragraphs above just showed. The error is applying that linearity across a boundary it does not cross, from a continuous quantity (how much text) to an outcome that is not continuous at all (whether a given query finds this chunk).

What is actually true starts from what recall measures for one query against one chunk. Either the chunk appears among the results the retriever hands on, or it does not. There is no half-retrieved chunk and no partial credit for a chunk that arrived with most of its evidence. Now consider where a query's evidence lives. The words that make this chunk the right answer to a specific question occupy some span of token positions, usually a short one, because a document says the thing that answers a specific question in one place rather than spread evenly throughout. Truncation removes one contiguous span, the one at the end. So for any given query the evidence lies wholly inside the surviving prefix or wholly inside the removed span. In the first case the stored vector is built from essentially all the text that query cares about and the score barely moves. In the second the vector is built from none of it and the score collapses.

The aggregate effect is therefore the share of queries whose evidence sat in the removed span, and that share equals the share of tokens removed only in a corpus where queries are aimed at token positions uniformly. The table above is a list of reasons why they are not. Concretely, for this postmortem, a chunk clipped from 780 to 510 tokens retains 65 percent of its text and retains close to 100 percent of its retrievability for queries about the timeline — the alert, the duration, the customer impact — and close to zero for queries about the cause, which is the class of query anyone actually types. Truncation does not shave the odds evenly across queries. It moves one identifiable group of queries from reliably-succeeding to reliably-failing and leaves the rest untouched.

Three queries against one document, whole and clipped#

The demonstration fits in a script. Build a document whose distinguishing vocabulary lives only in its final paragraph, embed it twice — once whole and once clipped at the limit — and score the same three queries against both.

import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("BAAI/bge-base-en-v1.5")
tok = model.tokenizer

doc = open("postmortem-2026-03-14.md").read()          # 780 tokens

ids = tok(doc, add_special_tokens=False)["input_ids"]
clipped = tok.decode(ids[:510])                        # what the encoder actually sees

v_full, v_clip = model.encode([doc, clipped], normalize_embeddings=True)

queries = [
    "how long was the reconciliation outage on 14 March",
    "who was paged for the settlement incident",
    "what caused the reconciliation worker outage",
]
qv = model.encode([f"Represent this sentence for searching relevant passages: {q}"
                   for q in queries], normalize_embeddings=True)

for q, v in zip(queries, qv):
    print(f"{v @ v_full:.3f}  {v @ v_clip:.3f}  {q}")

The two scored columns are cosine similarities, meaning the dot product of two vectors that have each been scaled to unit length, so the number reports the angle between them and nothing about their sizes. Run against a corpus of 11,000 chunks from the same repository, with the clipped and full versions of the postmortem each inserted in turn, the numbers come out like this.

Read the two cosine columns as gaps rather than as levels, because the absolute similarities are set by how the model happens to score this kind of text and carry no meaning on their own. The first row moves by about a hundredth, upward. The second moves by about a hundredth, upward. The third moves by more than two tenths, downward. The finding is the ratio between those gaps: the third is upwards of thirty times the size of the other two and points the opposite way. A uniform degradation — a chunk that got a bit worse at everything because a bit of it went missing — would have moved all three rows in the same direction by comparable amounts. This is a different shape of damage, and the gap ratio is what proves it.

The rank column carries the same story more starkly, and rank is the column that decides whether a document is retrieved at all. Two of the three ranks are effectively unchanged: 2 stays 2, and 4 improves to 3. The third moves from 1 to 214 in an index of 11,000 chunks. One document, one clip, three queries, and the outcomes are nothing happened, nothing happened, and gone.

The two rows that improved are the step that makes the whole table make sense, so they are worth a paragraph rather than a shrug. A sentence encoder produces one vector per token position and then collapses those into the single vector it stores, and the usual way of collapsing them is mean pooling: take the average of the token vectors over the positions that are actually present. The average is over what is present, which is the property that matters here — it is not a fixed summary that decays as pieces are chipped off it. Removing 270 tokens of root cause material, which has nothing to do with how long the outage lasted or who was paged, removes 270 contributions that were pulling the average away from the timeline. Every contribution that remains is timeline, so the centroid lands nearer the timeline than it did before. Truncation is not a uniform degradation of the chunk; it is a re-aiming of the chunk's vector at whatever survived, and the surviving part can end up better represented than before. The same averaging seen from the other side is what makes a long multi-subject passage match many things mediocrely and nothing well, which is chapter 4's subject (pooling as a centroid, and why a chunk covering four topics sits between all four rather than on any of them).

The third row is the failure. At rank 214 with a typical k — the number of top-scoring chunks the retriever actually hands on to the generator — of 5 or 10, the document is not in the candidate set, and no amount of prompt work downstream recovers it, because the generator is never shown the text. The chunk is present, its vector is well-formed, its distance calculations are correct, and the evidence needed to answer is not in the index in any retrievable form.

Overlap rescues the middle and never the ending#

Chunk overlap — repeating the last stretch of each chunk at the beginning of the next, so that a sentence cut in half by a boundary still appears whole somewhere — looks like it should absorb this, and for interior chunks it partly does. If chunk n is clipped and chunk n+1 begins with the last 200 characters of chunk n, then some of the discarded region is encoded after all, at the head of the following chunk. The arithmetic decides how much: a 2,000-character chunker with 200 characters of overlap re-encodes roughly the last 100 tokens, while an over-long chunk of 780 tokens against a 510-token limit discards 270. Two thirds of the lost region is recovered nowhere, and the recovered third arrives without the context that preceded it.

Those three figures are one piece of arithmetic and they agree, which is worth checking rather than taking. Of the 270 discarded tokens, roughly 100 come back at the head of the next chunk. The remainder is 270 minus 100, which is a shade under two thirds of 270 — hence two thirds recovered nowhere and a recovered third. Nothing is unaccounted for between the recovered part and the lost part, so the two fractions are a split of the discarded region rather than two independent estimates that happen to sit near each other.

The part that looks wrong and is not is the 200 characters standing for roughly 100 tokens. That pairing implies about two characters per token, which is dense — ordinary English prose written for humans compresses to somewhere between four and five characters per token. It is right for this corpus, and stating why turns it from a general rule into a condition on the claim. Runbooks, YAML fragments, command invocations and pasted stack traces are dense in punctuation, identifiers, hyphens and digits that a word-oriented vocabulary holds no whole-word entries for, so they decompose into two- and three-character fragments. Chapter 1 measures exactly this (a table of the same 2,000-character sample pushed through a real tokenizer for seven kinds of text), and it puts configuration and log data down at roughly two characters per token while marketing prose sits at more than twice that. On a corpus of ordinary prose the same 200 characters of overlap would come back as fewer than half as many tokens, and the recovered fraction of a truncated tail would be smaller still.

The ending is the case overlap cannot touch at all, because the final chunk of a document has no successor to carry its tail. Every document in the corpus therefore loses its last section outright whenever its final chunk is over the limit, and the final chunk is where the conclusion is by definition. Overlap is a mitigation for boundaries falling in awkward places, which is a different problem with a similar-sounding description, and it is worth being clear about the distinction before someone proposes raising overlap from 10 percent to 30 percent as a fix for truncation. That change costs a third more vectors, a third more storage and a third more embedding spend, and it leaves the document endings exactly where they were.

Structure-aware splitters interact with this badly in a way that is not obvious. A markdown splitter that keeps each ## section intact produces chunks whose length is the length of the section, so the distribution has a long right tail, and the sections in that tail are the ones with the most content — the detailed explanation, the long troubleshooting table, the thorough root cause. Splitting on structure rather than on length is usually the better choice for retrieval quality, and it concentrates truncation on precisely the sections that were worth writing at length. The combination that works is structure-first with a hard token cap as a second pass, so that a section over budget is split further instead of clipped.

Why the aggregate metric stays calm#

A regression like this rarely shows up as a dramatic movement in a headline number, which is what allows it to survive a launch review. Suppose 8 percent of chunks are over the limit, and for those chunks perhaps a third of queries target the truncated region. The number those two shares produce is their product, and the reason multiplication is the right operation rather than addition follows from the all-or-nothing property of a single query's outcome established earlier. Both conditions have to hold together for a query to fail this way. The chunk holding its answer must be over the limit, and the query must be aimed at the part that was removed. A query against an under-limit chunk fails for neither reason. A query against an over-limit chunk whose evidence sits in the surviving prefix still succeeds, which is precisely what the first two rows of the results table are. Adding the two shares would count both of those groups as casualties.

Taking a third of 8 percent gives a figure between two and three percent, and that is the drop of two or three points in recall@10 — the share of evaluation queries whose answer appears somewhere in the ten chunks retrieved for them — well inside the range a team will attribute to query-set noise or to a model version change. The multiplication treats the two shares as independent, and where they are not independent the product understates the damage: a corpus whose over-limit chunks are concentrated in one source system that users query disproportionately during incidents fails more often than the product predicts. It understates it into a range that is still small enough to hide, which is why the conclusion survives the loosening of its assumption.

The queries that fail are concentrated, so the users who hit them fail nearly every time, and the users who do not hit them see nothing wrong. That distribution is the whole trouble with reading the average. An evaluation set reports the mean over its queries, and a mean is exactly the wrong statistic for a defect that ruins a small group of queries completely and leaves the majority untouched, since it reports the same two or three points whether the damage is spread thinly over everybody or concentrated on the questions that brought people to the search box. Complaints arrive as "search is useless for postmortems" rather than as a metric alert, which routes the problem to a discussion about query understanding instead of to the preprocessing step where it lives.

Where the tail does not matter#

Truncation is not automatically expensive, and treating it as always-catastrophic will send you optimising corpora that do not need it. Several common cases are genuinely harmless. Documents that end in boilerplate — legal footers, "see also" link lists, contributor credits, navigation chrome that the extractor failed to strip — lose nothing of value when clipped, and clipping arguably improves them by removing text that dilutes the vector. Corpora written in the inverted pyramid, which is to say news and press releases, put the distinguishing content in the first two sentences by convention. Anything with an abstract or an executive summary at the top has its own defence.

What all three exceptions have in common is that they break the same link in the chain. The damage in this chapter runs: clipping removes the end, the end carries the distinguishing vocabulary, queries are aimed at distinguishing vocabulary, so the clipped queries fail. A corpus whose endings carry nothing distinguishing breaks the second link, and the argument stops before it reaches a conclusion about retrieval. That is a fact about a corpus rather than about truncation, which is why no rule of thumb about document length can settle it.

The way to tell which corpus you have is to look at what the truncated regions contain, not at how many tokens they hold. Sampling twenty over-long chunks, reading only the part that gets discarded, and asking whether any evaluation query would need it is half an hour of work and produces a more reliable answer than any heuristic about document length.

There is also one recovery path that is real and frequently forgotten. A cross-encoder reranker — a second model that reads the query and a candidate passage together in one sequence and outputs a relevance score, rather than embedding each separately — scores the query against the stored text of a candidate, not against its stored vector, so a reranker whose own input limit is larger than the encoder's can see the tail that the embedding never encoded. That helps only for documents that make the candidate set in the first place — it reorders, it does not retrieve — so it rescues the case where the truncated chunk lands at rank 40 out of a retrieved 100 and does nothing at all for the case above, where the chunk landed at 214 out of 11,000. Reranking is a mitigation for near-misses and not a fix for missing evidence.

The failure this chapter describes is fully explained by text that never reached the encoder. The token arithmetic closes with nothing left over, the vector is well-formed, the scoring is correct, and the only thing missing is the 270 tokens at the end — which is why the fix is unglamorous and reliable: size the chunks under the limit and re-index, and the sentence that names the cause gets a vector of its own. The recovery will not be proportional to the tokens restored, for the same reason the damage was not proportional to the tokens removed. The queries that were failing were failing completely, so they come back completely, and a group of queries that went from never working to always working moves the metric by more than the token accounting ever suggested it should.

What makes long documents difficult is that doing exactly that improves them without making them as good as short documents. The residual gap has a different cause entirely, and the two are told apart by what happened to the tokens: here some of them never arrived, whereas in the gap that remains every token reaches the model and the match is weak anyway, because one averaged vector standing for a long passage is a blurrier object than one standing for a short one. That is the next chapter's subject, and it is the reason a re-index that removes every truncation event can still leave a visible deficit on the long-document slice of an evaluation set.