Truncation costs you deep ranks long before it touches the top ones

Rank depth is a property of a result list, not a detail of it. A ranked retrieval result is usually summarised by one number, recall@k — the fraction of queries for which a document known to be relevant appears somewhere in the top k results, where k is a cutoff chosen when the measurement is set up. Choosing k chooses a question. Recall@10 asks whether the answer is on the first page; recall@100 asks whether it is anywhere inside a pool a hundred documents deep. Those are different questions about the same ranking, and a change to an index can hold one of them flat while moving the other a long way, because the two depths are not built alike: the top of a ranking is a few documents separated by wide score gaps, and the deep band is many documents separated by thousandths.

That distinction acquires teeth when something other than a person reads the list. A retrieval pipeline that fetches a wide pool of candidates and then narrows it — with a cross-encoder reranker, a second and slower model that reads the query and one candidate together, scores that pair, and reorders the list by those scores, or with a filter on a metadata field — is consuming the deep band as raw material. Shrinking vectors by truncation, which means keeping a prefix of the coordinates and discarding the tail, is normally justified with recall measured at the cutoff a user sees, and that cutoff is the one place the damage does not show. The cost of getting this wrong is invisible on every retrieval dashboard and perfectly visible in whatever comes out of the last stage.

The gaps that truncation has to survive get smaller with depth#

Whether a perturbation changes a ranking is a comparison between two lengths measured along the same axis. Two documents adjacent in a ranking are separated by a gap, which is the difference between their similarity scores. Truncation moves each score by some amount. The pair keeps its order if the movement is small against the gap and swaps if the movement is larger than the gap, so neither figure decides anything on its own. Both are quantities in units of cosine similarity, which makes their ratio a pure number with no units left in it — the only dimensionless comparison the two of them admit — and that ratio is what says whether the ordering at a given depth is stable.

Take one query and score every document in a corpus against it at full width. The resulting similarities are not spread evenly down the ranking. Near the top, the documents that genuinely answer the query sit well clear of everything else: across Halvorsen's 240-query set, the median cosine gap between the tenth-ranked and eleventh-ranked document was 0.031. Two hundred ranks further down, the same measurement gives 0.0027 between rank 100 and rank 101. The tail of the ranking is a dense band of documents that are all vaguely about the right subject and differ from each other by thousandths.

Truncation perturbs every similarity by some amount, and that amount can be measured directly rather than reasoned about. Comparing 256-dimension similarities against 3072-dimension ones over a sample of query–document pairs from that corpus gave a difference with a standard deviation of 0.012 and a mean shift of +0.04, the shift being the systematic upward bias noted at the end of the last chapter. The standard deviation is the figure to carry into the comparison: it is how far an individual score typically moves once the part of the shift shared by every score is set aside.

Now form the ratio at each depth. At rank 10 the perturbation of 0.012 is set against a gap of 0.031, and it covers about a third of the distance it would have to travel to put the eleventh document above the tenth, so a typical perturbation leaves that pair alone and swaps happen only out in the tail of the perturbation's own distribution. At rank 100 the same 0.012 is set against a gap of 0.0027, and it is more than four times the distance between adjacent documents, which means a typical perturbation carries a document past several of its neighbours rather than threatening one boundary. The ordering in the deep band is not degraded so much as re-drawn, with the true rank-100 document landing anywhere between 60 and 180.

There is a step in that comparison worth being honest about, because it is where a careful reader will stop. Both scores in a pair move, so the quantity that actually decides a swap is the difference between two perturbations rather than one perturbation on its own, and for two independent draws that difference is wider than either by a factor of the square root of two. Carrying that correction through changes the arithmetic and neither verdict: at rank 10 the perturbation covers about half the gap instead of a third, which still leaves the ordering intact most of the time, and at rank 100 it goes from more than four times the gap to several times it, which was already far past the point where the ordering means anything.

The measurement returned two numbers, and only one of them can reorder anything. A shift of +0.04 is common to every score in the list, and rank order depends on differences between scores, so adding the same amount to two scores leaves their difference exactly as it was and leaves their order exactly as it was. This is the same invariance that made a constant harmless in chapter 2, where slicing a unit-norm vector left every prefix with a length of its own: a single factor applied to every stored vector could not reorder anything, and the per-document factor could. What reorders is the spread, and the spread here is 0.012.

The shift is not harmless, though, and what it breaks is the thing chapter 2 ended on. Any absolute threshold — a rule that discards candidates scoring below some fixed similarity, or that decides a query has no good answer at all — was calibrated against full-width scores. Every truncated score now arrives inflated by roughly 0.04, so the same threshold admits a band of documents it was built to exclude, and the "no good answer" branch stops firing on queries that have no good answer. Ordering survives a common shift and calibration does not.

import numpy as np

def gap_profile(sims: np.ndarray, ranks=(1, 10, 100, 500)) -> dict[int, float]:
    """sims: (n_queries, n_docs) exact full-width similarities."""
    ordered = -np.sort(-sims, axis=1)
    return {k: float(np.median(ordered[:, k - 1] - ordered[:, k])) for k in ranks}


def truncation_error(full: np.ndarray, short: np.ndarray) -> tuple[float, float]:
    """Both (n_pairs,) similarities for the same query-document pairs."""
    d = short - full
    return float(d.mean()), float(d.std())

# gap_profile -> {1: 0.089, 10: 0.031, 100: 0.0027, 500: 0.0009}
# truncation_error(full=3072, short=256) -> (0.041, 0.012)

The profile that function returns is worth reading across all four of its depths rather than at the two the argument needed. The gap at rank 1 is 0.089, roughly three times the gap at rank 10, and the gap at rank 500 is 0.0009, roughly a hundredth of the gap at the top. The contraction is smooth and monotone, which is what makes the two-depth comparison above a description of a curve rather than an anecdote about two ranks that happened to be convenient.

What the prefix was trained to separate#

The reason the top survives at all is a chain of four steps, each of which can be checked separately. Nested training — the objective from chapter 1, under which every prefix of the vector is trained to work as a representation on its own — puts a contrastive loss on the first 256 coordinates, and a contrastive loss asks those coordinates to score the query against its true match higher than the query against each negative in the same training batch. Those in-batch negatives are simply the other documents that happened to be in the batch. Those documents are drawn at random from a large corpus. A random draw from a large corpus is almost never a near-duplicate of the positive, because near-duplicates of any given document are a vanishing fraction of a corpus.

Follow that to its end. The prefix was therefore never asked to rank two near-identical flood-exclusion clauses against each other, only to tell a flood-exclusion document from an auto-liability one, which is a coarse task and one the prefix learns to do well. The distinctions that separate rank 40 from rank 90 were learned by the full-width term of the objective instead, and they live in the coordinates that only that term ever touched. Discarding the tail discards them.

One link in that chain is a property of the training data rather than of the method, and it is the third one. Nothing in the nested objective requires negatives to be random; a recipe that mined hard negatives, deliberately placing near-duplicates of the positive in each batch, would be forcing the prefix to make exactly the fine distinctions it currently cannot. The shape of the loss in the table below is therefore a joint property of truncation and of how this particular checkpoint was trained, and a future checkpoint trained on harder negatives could move it. The other three links are structural and will hold for any model of this family.

The plain-words version, once the mechanism is in place: the prefix knows what a document is about, and the tail knows which of the many documents about that thing is the one that answers the question.

Recall and overlap are not measuring the same event#

The full profile across dimensions, on the same 240 queries, comes out like this.

Reading the table by column rather than by row is what makes the choice tractable. The recall@10 column is nearly flat down to 256 and only breaks at 128, so a team that looks only there will conclude the model tolerates a factor of twelve. The recall@100 column starts moving at 512 and has lost twelve points by 256. The overlap column moves first and moves hardest, losing seven points at the very first halving. Each column has a legitimate claim to being the number that matters, and which one it is depends on the architecture rather than on the embedding.

The 1536 row is where the two kinds of column come apart most cleanly, and the finding in it is in no cell of the table. Against full width that row is identical in recall@1, identical in recall@10 and identical in recall@500, and it differs by a single point in recall@100 — while its top-100 overlap with full width is 0.93. Halving the dimension changed essentially nothing any recall column is able to see, and swapped seven documents in every hundred out of the candidate pool.

Both things are true because the recall columns and the overlap column are not measuring the same event. A recall column asks whether the documents already known to be relevant are still somewhere inside the cutoff, which is a question about a handful of documents named in advance by whoever built the evaluation. The overlap column asks whether the pool is made of the same documents, which is a question about all hundred of them, most of which no evaluation ever labelled. The labelled few can sit perfectly still while the unlabelled majority is reshuffled around them, and that is precisely what the 1536 row records. A pipeline whose next stage reads the whole pool is governed by the second question and is told nothing by the first.

The overlap column is also the one almost nobody computes, and at 256 dimensions it is the one that matters most. The candidate list handed to a reranker there shared 55 documents with the list the full-width index would have produced. Recall@100 of 0.84 says most known-relevant documents are still somewhere in there; the overlap of 0.55 says nearly half the pool is different material, and the reranker's output is a function of the pool it was given.

The consumer decides whether deep recall is real#

Retrieval quality has no meaning independent of what reads the result list, and the same 256-dimension index is a good decision or a bad one depending on which of these sits downstream.

Serving ten results to a person is the case where truncation is close to free. The user reads the list, and recall@10 falling from 0.83 to 0.79 costs four queries in a hundred where the right document is on page two instead of page one. Recall is a fraction of queries, so four points of it converts directly into that count without any further modelling. That is a real cost and a small one, and it is the configuration the storage saving was designed for.

A reranker over a wide candidate pool inverts the analysis, because the reranker exists precisely to fix deep ranks. Halvorsen instrumented which initial ranks the finally-cited chunks came from: 41% arrived from initial ranks 20 through 200. A cross-encoder that promotes a document from rank 63 to rank 3 is the entire value of that stage, and it can only promote documents it was given. Compressing the first stage in the region where the second stage does its work removes the second stage's raw material while leaving its own metrics — nDCG, the rank-weighted ordering score rerankers are usually judged by, computed over the pool it received — looking fine.

Post-filtering has the sharpest version of the problem, and it is sharp because of an arithmetic that runs in two directions. Halvorsen's adjusters scope queries by line of business, and marine claims are 6% of the corpus. Read forwards, the retrieval width multiplied by the pass rate gives what survives: retrieving 200 candidates and then filtering leaves around 12 survivors on average, because each retrieved candidate carries roughly the corpus-wide chance of being a marine document. Read backwards, the survivor count you want divided by the pass rate gives the width you would have to retrieve to get it: 200 survivors at that rate needs something like 3,300 candidates pulled from the index.

Those are one relation read in two directions — survivors equal width times rate, so width equals survivors divided by rate — and the backwards reading is the dangerous one. The effective cutoff for a marine query is not 200 but something like 3,300 in unfiltered terms, which is a depth at which no column of the table above was measured and a depth where the gaps between adjacent documents are smaller than any figure in this chapter. Every point of recall lost in the deep band is multiplied by the selectivity of the filter, and a filter that keeps one document in twenty turns a 12-point recall@100 loss into an empty result set for the queries that were already marginal.

There is a partial remedy for the reranker case that costs nothing in storage, which is to widen the candidate pool to compensate. If the 256-dimension index reaches recall@500 of 0.92 where the full-width index reached recall@200 of about 0.97 — a figure read between that table's recall@100 and recall@500 rows and therefore approximate — retrieving 500 instead of 200 recovers most of the gap. It is not free in latency: an HNSW traversal to depth 500 costs roughly 1.6 times the traversal to depth 200, because a graph search reaches deeper by visiting more nodes rather than by restarting.

The reranking side of that bill is the one to price, and it is priced differently. A cross-encoder scores one pair per candidate, so its cost is linear in the number of candidates, and 500 pairs instead of 200 is two and a half times the GPU time. That stage is usually the dominant cost of the whole query, which means the multiplier that governs the total is the multiplier on the dominant term rather than the one on the graph. Halvorsen measured that swap at 340 ms of added p95 latency, and the reason it landed there rather than somewhere proportional to the 1.6 is exactly that ordering of costs. The traversal ratio is the number people quote and the reranking ratio is the number they pay.

Two other consumers belong on the same list. Diversity reranking with MMR, or maximal marginal relevance, which picks each next result for relevance minus similarity to what it has already picked, selects a spread across the candidate pool, so a pool that has lost its deep half loses the alternatives MMR was there to surface. Multi-step agents that issue several queries and union the results accumulate the deep-band loss once per query, and the union is exactly where near-duplicate candidates from different sub-queries have to be ordered against each other.

Why "recall@10 held" is the wrong sentence#

The claim that a compression is safe because the metric at the serving cutoff barely moved is a defence that stays constant while the thing it is supposed to discriminate varies. Between full width and 256 dimensions recall@10 falls four points while the composition of the 200-document candidate pool changes by nearly half, and a metric that returns very nearly "pass" whether the pool the reranker consumes is intact or replaced is not distinguishing the two cases at all. It is tempting for a good reason: recall@10 is the metric that matches what a user actually sees, and in a system that serves ten results directly it is the correct number and the four points are the whole cost. It becomes wrong at one identifiable moment, which is when the serving cutoff stops being the retrieval cutoff. In a pipeline with a reranker the number that governs is recall at the first-stage cutoff, and the correct question is what the candidate pool contains rather than what the final list shows. Measuring at the stage boundary, not at the user boundary, is the fix, and it costs nothing beyond passing a different k to a harness that already exists.

The second defect is that recall@10 is an average and averages are generous. Halvorsen's 240 queries split into subsets, and at 256 dimensions the aggregate loss of four points decomposed into one point on the 150 queries whose answer was topically distinctive and eleven points on the 90 that had to distinguish between near-identical wordings from different policy years. The mechanism is the relative size of the two groups: the immune slice is close to two thirds of the set, and a near-zero delta over two thirds of the queries dominates the mean whatever the remaining third does. The second group is small enough not to move the mean and important enough that adjusters notice. That decomposition is what chapter 7 turns into an acceptance criterion; here it is enough to see that a single averaged number was never going to catch it.

A third habit worth adopting is to stop treating the dimension as a global constant. Nothing requires the truncation point to be the same everywhere, and the two-stage structure that the next chapters develop for quantisation applies here as well: a 256-dimension index can retrieve a wide pool while the full-width vectors, held elsewhere, rescore it. The research has gone further in the same direction — the "2D Matryoshka" line of work applies the nested objective across transformer layers as well as across dimensions, so a model can be truncated in depth and width together, and several 2024 and 2025 releases publish per-dimension recall curves alongside their headline scores precisely so that this table does not have to be rebuilt by every team. That is useful context and no substitute for measuring on your own corpus, since the shape of the curve depends on how near-duplicate your documents are, and insurance wordings are far more self-similar than the web text those curves were computed on.

None of this argues against truncation. The 1024-dimension row of that table costs two points of recall@100 and a third of the vector bytes, and it would have been an uncontroversial change. The error was choosing a dimension by looking at the metric that the pipeline architecture had made least relevant, and the correction is a rank-depth profile rather than a scalar, evaluated at the cutoff the next stage consumes.

It is also worth knowing that this shape of damage is specific to truncation. Losing dimensions removes resolution from the fine distinctions and leaves the coarse structure intact, which is why the loss concentrates in the deep band. The other way to shrink a vector keeps all 3072 coordinates and stores each one in fewer bits, and that damages the ranking in a different pattern — closer to uniform noise across the whole list, top ranks included. It also changes something truncation left alone, which is the arithmetic your index performs to compare two vectors at all.