A truncated prefix has to be re-normalised before anything compares it

Normalising a vector means dividing every one of its components by that vector's own Euclidean length, so what comes out has length one and carries only its direction. A prefix is what dimension truncation leaves behind: the first components of an embedding, kept in their original order, with the remainder discarded. Chapter 1, where a nested Matryoshka training objective is what makes a prefix a trained representation rather than an arbitrary slice, settled which models may be cut at all. What follows is about what your own code has to do to a prefix once you have one, and the answer is a single division that is almost always missing.

It matters because a vector store scores with an operation that reads a vector's length as well as its direction. An inner product between a query and a document is the length of one, times the length of the other, times the cosine of the angle between them, and a Euclidean distance is assembled from those same three quantities. When every stored vector has length one, those lengths are all the same number and the score is the cosine alone, which is why a dot-product index and a cosine index return the same ordering over full-width embeddings. Truncation breaks that condition silently. Over a corpus of two million insurance chunks, an index built from raw slices ranked partly by a quantity that had nothing to do with whether a chunk answered the query, returned ten results for every query, and raised nothing.

Slicing changes the length, and the length varies per document#

Euclidean length is the square root of the sum of the squared components, so a unit vector is one whose squared components sum to one. Take a vector of four components, each equal to one half. Each component squared is one quarter, four quarters sum to one, and the square root of one is one, so that vector has unit length. Now keep the first two components and discard the rest. The two surviving squares are a quarter each, they sum to one half, and the length of the prefix is the square root of one half — a little over seven tenths, and no longer one.

Nothing in that example depended on the components being equal. Every squared component is non-negative, because squaring removes the sign, so dropping components can only ever take non-negative amounts out of the sum. A proper prefix of a unit vector — a prefix that leaves at least one component behind — therefore has a sum of squares strictly below one and a length strictly below one. How far below is not a constant of the model. It is the share of the squared length that the leading coordinates of that particular document happened to receive, and that share is a fact about the document rather than about the encoder.

The full vectors coming back from a nested model such as text-embedding-3-large are unit length, which is why a dot product over them is a cosine similarity and why every store in the stack can treat the two as interchangeable. The sum of all 3072 squared components is 1.0, so the sum of the first 512 squared components is some number below 1.0, and how far below depends on how much of that particular document's signal the nested training pushed into the leading coordinates. Across two million Halvorsen chunks the 512-dimension prefix norm averaged 0.81 with a standard deviation of 0.06, and the extremes ran from 0.62 to 0.94. That measurement has a standard deviation at all for the reason just derived: the prefix norm is a per-document property, so it is a distribution and not a number.

That spread is the whole problem, and the mean is not. An inner product is linear in each of its two arguments, which means multiplying one argument by a number multiplies the result by that same number: the query dotted with a stored vector scaled by some factor equals the factor times the query dotted with the original vector. Scale every stored vector by one shared factor and every score in the list is multiplied by it. Multiplying a whole list of scores by one positive number cannot change their order, because the comparison between any two of them survives dividing both sides by that number. A uniform prefix norm — every document at 0.81, say — would therefore be invisible, since every score would shrink by the same amount and the ranking would come out untouched.

Scaling each document by its own number is a different operation, and it does change the order. When two documents are compared the multiplier is no longer a common factor that can be divided out, because the two sides carry different multipliers. What decides the ranking is each document's cosine against the query multiplied by that document's own prefix norm, and a document with the smaller cosine can win by carrying the larger norm. What you have instead of a constant is a per-document multiplier that correlates with properties of the document — length, vocabulary specificity, how close it sits to the dense middle of the corpus — and has nothing to do with whether it answers the query. An inner-product index will happily fold that multiplier into the score.

import numpy as np

q = normalise(full_query[:512])          # query prefix, unit length
a = full_a[:512]                         # ||a|| = 0.94
b = full_b[:512]                         # ||b|| = 0.71

# what you meant to rank by
cos_a, cos_b = 0.62, 0.68                # b is the better answer

# what an inner-product index actually ranks by
float(q @ a)   # 0.62 * 0.94 = 0.583  -> returned first
float(q @ b)   # 0.68 * 0.71 = 0.483  -> returned second

Read the two products in that listing against each other. Document b is the better answer by direction, since its cosine of 0.68 beats a's 0.62, and the cosine is the quantity the ranking was supposed to be about. The index scores a at 0.583 and b at 0.483, so a comes back first. The reason sits entirely in the norms: a's prefix carries 0.94 of its original length against 0.71 for b, so a's smaller cosine is multiplied by the larger number and the product crosses over. What the reversal cost is a document that is better by direction losing to one that is worse.

Six points of cosine separated those two documents, and twenty-three points of difference in prefix norm reversed them. The reversal is not random noise, which would at least average out; it is a systematic preference for documents whose leading coordinates carry more mass, applied identically to every query in the system.

It also biases in a direction that is hard to notice from the results, because prefix mass is not distributed at random across a corpus. Chunks that sit near the centre of the embedding cloud — generic administrative prose, boilerplate headers, the standard indemnity paragraph that appears in four hundred policy documents — tend to have more of their length in the leading coordinates, since those coordinates encode the coarse topical position that such chunks share with everything around them. Specific chunks with unusual vocabulary carry more of their length in the tail. An unnormalised inner-product index therefore promotes boilerplate and demotes exactly the specific passages a search is usually looking for, and the resulting top-ten lists read as slightly bland rather than as obviously broken.

Euclidean distance tilts toward norms near the cosine#

Euclidean distance misbehaves differently and no more usefully. With a unit-length query and a document of length n, the squared distance is 1 + n² − 2·n·cos, and differentiating with respect to n shows the distance is minimised when n equals the cosine similarity rather than when the cosine is largest. Both halves of that claim are worth walking rather than accepting.

The expression comes out of expanding the squared distance between the two vectors, which is the query's squared length, plus the document's squared length, minus twice their inner product. The query's squared length is one, because the query is unit length. The document's squared length is n². Their inner product is the product of the two lengths and the cosine of the angle between them, which comes to n times the cosine, since the query contributes a length of one. Putting the three pieces back together gives the expression above.

Which document length makes that expression smallest, holding the cosine fixed, is answered by differentiating with respect to n. The derivative of the constant one is zero, since a constant does not move when n moves. The derivative of n² is two n. The derivative of the last term is minus twice the cosine, because the cosine is a property of the angle and does not depend on n, so it rides along as a constant multiplier on a term that is otherwise linear in n. Setting the sum of those three derivatives to zero gives two n minus twice the cosine equal to zero, and dividing both sides by two leaves n equal to the cosine similarity.

An L2 index over unnormalised prefixes therefore rewards documents whose norm happens to sit near the similarity value it is trying to measure, which is not a bias anyone can reason about or correct for downstream. The last clause is worth being precise about, because it is the part that makes this worse than the inner-product case. A bias that always favours a larger norm is monotone — more norm, more score, all the way up — and a monotone distortion can in principle be undone by dividing the score by the norm afterwards. This one is monotone in nothing. A document is favoured for having a length close to its own cosine against this particular query, so the same document is helped on one query and hurt on the next, and no per-document correction fixes both.

Where the normalisation has to sit#

The rule is that normalisation is the last thing that touches a vector before it is stored or compared, and that anything applied after it invalidates it. Truncation is applied after it, so truncation requires doing it again.

Ordering the two operations the other way makes no difference to the result — dividing the whole vector by its full-width length and then dividing the prefix by its own length gives the same unit prefix as slicing first — and the reason is that normalisation discards scale. Dividing a vector by its own length sends every positive rescaling of that vector to the same output, because a factor applied to the components is also applied to the length and cancels in the division. Any positive rescaling applied before normalisation is therefore erased by it, and a full-width normalisation is one such rescaling. What cannot be erased is an operation applied after the division, and truncation is one of those, since removing components changes the sum of squares instead of scaling it. The rule is not about order at all. It is about which operation is last, and the failure mode is doing it only once and doing it too early.

import numpy as np

def prefix(v: np.ndarray, dims: int) -> np.ndarray:
    """Truncate to `dims` and restore unit length. Use this on both sides."""
    p = v[..., :dims]
    return p / np.linalg.norm(p, axis=-1, keepdims=True)


# invariant worth asserting at index build time
norms = np.linalg.norm(vectors, axis=1)
assert np.allclose(norms, 1.0, atol=1e-5), f"min={norms.min():.3f} max={norms.max():.3f}"

The second requirement is symmetry, and it is easier to break than the first because both sides can look correct in isolation. OpenAI's dimensions parameter truncates and re-normalises server-side, so a query embedded with dimensions=512 arrives unit length. If the documents were embedded at full width and sliced locally without re-normalising, both sides are 512-dimensional float arrays, the index accepts them, the dimensions match, and every search returns ten results ordered by a score that is half cosine and half document magnitude. Nothing in the stack can detect this, because there is no representation-level signature that distinguishes a correct 512-dimension vector from an incorrect one.

The belief that fails here is that a shape check is a correctness check, and it is worth stating in its strongest form: the index's validation returns exactly the same verdict for a correctly re-normalised prefix and for a raw slice. Both are 512-dimensional float arrays, both are accepted, both produce ten results per query. A defence that is constant when the thing it must discriminate varies is not a defence at all, whatever it happens to be checking. It is tempting for a nameable reason: a dimension mismatch is the one error in this family that does fail loudly, immediately and with a legible message, so an engineer who has met that failure reasons outward from it and concludes that the store validates its vectors. The store validates their shape. What discriminates is the assertion on the norms in the listing above, which passes on correct input and fails on incorrect input, and a comparison of search results against an exact full-width baseline, which is the only check that can see a ranking that is plausible and wrong.

Symmetry has a second edge that shows up during migrations rather than at build time. An index rarely changes over in one atomic step; a fifty-million-chunk reindex runs for hours or days, and during that window the collection contains vectors written by the old path and vectors written by the new one. If the change is a truncation point or a normalisation convention, every query in that window searches a collection whose vectors are not mutually comparable, and the results skew toward whichever cohort the current convention favours. Teams that reindex in place discover this as a period of degraded search they cannot reproduce afterwards, because by the time they investigate, the migration has finished and the collection is consistent again. Building into a new collection and switching an alias avoids the whole class.

Mismatched truncation points are the benign version of the same family, because a 512-dimension query against a 256-dimension index raises a shape error immediately. The dangerous mismatches are the ones that preserve shape: a query path that re-normalises against a document path that does not, a document path that slices from a model revision whose coordinate ordering differs, a re-embedding job that used dimensions=512 for new chunks while the backfill sliced locally. Each of those produces a working system with degraded ranking, and each is invisible to any check short of comparing search results against an exact full-width baseline.

What your store already does, and what it leaves to you#

Vector databases disagree about who owns normalisation, and the disagreement is why this bug travels between teams. Some normalise on write, some normalise at query time as part of the metric, and some do neither and expect unit vectors as a precondition they never verify.

The rows that say "None" are not an argument for using those configurations. Cosine metrics that normalise per comparison pay for two square roots on every distance evaluation, which is measurable inside an HNSW traversal — Hierarchical Navigable Small World, the layered proximity graph most vector stores walk to find approximate nearest neighbours — that performs several thousand of them per query, and it is the usual reason a team moves to an inner-product metric in the first place. Both readings of those rows are true at once: the configuration is immune to this bug, and the immunity is bought with arithmetic on every hop of every search. Stores that normalise on write solve the document side and leave the query side to whatever your application does. The practical position is to own normalisation in your own code, keep the store on inner product, and treat the assertion above as an index invariant rather than trusting a configuration flag you did not set.

Two points that belong to the model#

Re-normalising recovered seven of Halvorsen's nine points and left two, and those two are real. A correctly truncated, correctly normalised prefix of a nested model still loses information, and the next chapter is about where that loss lands. The temptation at this stage is to treat any remaining gap as more of the same measurement problem and go looking for another bug, which is why it is worth fixing the arithmetic before forming any opinion about the model — an incorrect harness will absorb an arbitrary amount of genuine quality loss into what looks like a plausible compression penalty, and a correct one will not.

Normalisation is also not a repair for the wrong model. Slicing bge-large to 256 dimensions and dividing by the resulting length gives a unit vector pointing in an arbitrary direction, and unit length does not make an arbitrary direction meaningful. The pilot from the previous chapter was already computing cosine similarity through FAISS's normalised path, and it still lost fifteen points, because the problem there was the subspace rather than the scale. These are two independent preconditions: the model has to have been trained for prefixes, and your code has to handle the prefix correctly. Satisfying one and not the other fails, and the two failures look similar enough from the outside that teams regularly diagnose the wrong one.

There is one more thing normalisation cannot do, which is rescue a comparison across different truncation points. Similarity scores from a 512-dimension index and a 3072-dimension index are not on the same scale even when both sides are unit length; the same query-document pair typically scores higher at 512 because the prefix has thrown away the components most likely to disagree. A threshold of 0.75 tuned at full width will admit substantially more documents at 512, so any code that filters on an absolute similarity value has to be retuned alongside the dimension change. Halvorsen had such a filter in their citation step, set at 0.72, and it went from rejecting 31% of retrieved chunks to rejecting 12%.

That movement reads at first like a contradiction of everything above, since truncation is supposed to lose information and here it is apparently letting more chunks past a quality gate. The chain that resolves it is short. The discarded tail coordinates are the ones carrying fine distinctions, and fine distinctions are what push two vectors apart, so removing them raises similarity scores systematically across every pair in the collection. A fixed absolute threshold then sits lower in a distribution that has moved up, and rejects less of it. The filter did not get better at its job; its job got easier to pass, which is worse than an unchanged rejection rate would have been, because the chunks it now admits are the marginal ones it was installed to catch.

Everything between the slice and a trustworthy measurement is one division. A prefix of a unit vector is not a unit vector, its length is a property of the document rather than of the model, and any metric that reads length will read that property as though it were relevance. Divide the prefix by its own length, do it on both sides of every comparison, and assert the result at index build time rather than trusting a store's configuration to have done it for you.

With the harness correct, the numbers finally described the model rather than the code: at 512 dimensions recall@10 fell by 2 points while recall@100 fell by 7. That asymmetry holds across corpora and models, it has a specific cause, and whether it matters depends entirely on what reads the result list.