The dot product

Representing an artifact as a list of numbers and comparing those lists is older than any of the machinery that currently produces them. Search systems have described a document as a vector of term weights, recommendation systems have described a user as a vector of preferences, and clustering has described anything at all as a point in a space where nearness is the whole of the question. What arrives from an embedding model is a dense vector, and topic 8.4 owns that object; what follows here is what happens to it once it exists.

The subject has a boundary, and the boundary is worth fixing before any of it is worked. How an embedding model comes to place one text near another — what it was trained on, what objective shaped it — is a property of the model rather than of the arithmetic run over its output. What semantic similarity means, as a claim about language, is settled somewhere other than in a sum of products. How a vector index stores millions of vectors and finds the nearest ones without comparing every pair is a data-structure problem with its own literature. The operations below run on vectors that have already been produced, by whatever process, and they behave identically whether the coordinates came from a neural network or from counting words.

Two facts follow from that definition and carry everything after it. These operations are the step that turns vector coordinates into a retrieval score, so they sit directly on the path between a stored document and the ranked list a user sees. And changing which operation computes that score can change the order the list comes back in, on identical vectors, with no document edited and no model swapped.

What that second fact costs is easiest to see on a system small enough to hold in view. A payments platform runs a documentation search over its own developer docs. The docs are split into chunks, each chunk is embedded once at write time, the query is embedded at request time, and the index returns the highest-scoring chunks. A developer is trying to stop a webhook delivery from giving up too early.

How do I increase the timeout for a failed payment webhook?

Two chunks in the index are candidates. One of them is the answer.

webhook-timeouts        Configuring delivery timeouts for webhook endpoints.
                        Sets the per-attempt timeout in seconds, the maximum
                        permitted value, and how to change it per endpoint.

retry-and-safeguards    Retry behaviour after a failed delivery, backoff
                        intervals, dead-letter handling, alerting thresholds,
                        rate limits, and the billing implications of repeated
                        delivery attempts.

1.  retry-and-safeguards    26.0
2.  webhook-timeouts        15.0

The chunk named webhook-timeouts is about setting the timeout for a webhook, which is the sentence the developer typed, and it comes second. The chunk that comes first is long, covers retries and failures and several operational topics beside them, and mentions timeouts in passing. Nothing is broken. The index is returning exactly what its scoring operation ranks highest, and that operation rewards a vector's length as well as its alignment with the query, so a chunk that covers more ground earns a bigger number for covering more ground. The problem this topic solves is that one similarity score can quietly mix points in the same direction with is larger, and retrieval then returns the less useful passage even when the query and the direct answer are the closest pair in meaning.

The measure that produced that ranking, and the ones that would have reversed it — dot product, cosine similarity, Euclidean distance — are documented in several places worth naming for what each actually supplies. Google's Measuring similarity from embeddings gives the formal treatment of similarity measures computed over embeddings. Pinecone's Vector Similarity Explained compares the three and sets out where each fits. Elastic's Vector similarity measures and scoring in Elasticsearch shows how those measures are configured and scored inside a working vector index, which is where the choice is actually made. Zilliz's Vector Similarity Metrics: Cosine Similarity shows that cosine similarity is the dot product of normalized vectors. SH AI Academy's Cosine Similarity vs. Euclidean Distance vs. Dot Product: Choosing the Right Metric for AI Search works the same comparison as a stack decision. What none of them supplies is an originator: no person is named as having introduced vector arithmetic, and no date is given for its introduction. Those sources are later formal treatments and practical explainers of operations that were already in use, and reading any of them as the origin of the idea would be inventing a history none of them claims.

The eight destinations#

Those explanations converge on the same eight ideas. Vector magnitude, the L2 norm, the dot product, vector normalization, the unit vector, cosine similarity, magnitude bias and numerical tolerance are the destinations, and they are arranged in that order because each one is built out of the ones before it.

The first pair is the pair most easily misread as two separate stages, so it is worth being exact about the relationship before either is worked. What vector magnitude and the L2 norm share is a single measurement: for any vector in this chapter, they name the same number. Vector magnitude is the geometric quantity — the length of the vector measured from the origin — and the L2 norm is the particular formula, the square root of the sum of the squared components, used to compute that length from coordinates. Magnitude names what is being measured; the L2 norm names how it is calculated here. They are two aspects of one measurement rather than two independent effects, and a worked example that produced different values under the two names would be a worked example with an arithmetic error in it.

From there the dependencies run in a line:

Magnitude bias is worth naming only here, at the end of the line, because only now is there enough machinery to say precisely what went wrong in the ranked result above and precisely which repair addresses it.

Watching that line run takes numbers, so here are the ones the rest of the chapter uses. Every vector below has three coordinates, small enough to work by hand. In this example the first coordinate carries timeout-configuration content, the second carries retry-and-failure content, and the third carries billing and rate-limit content. A real embedding has hundreds or thousands of coordinates and none of them is labelled, and every operation performed here is performed identically on all of them.

Vector magnitude#

The developer's query embeds to a vector, and each stored chunk embedded to one at write time. Start with the query alone, before anything is compared to anything.

q = [4, 3, 0]

Four units along the first coordinate, three along the second, none along the third: the query is mostly about timeout configuration, partly about failure, and says nothing about billing. Read as a position, that is a point reached by going four steps out on one axis and three on the next, and the straight-line distance from the origin to that point is five. The arrow has a direction — the particular slant produced by going four one way and three the other — and it has a length, and the length is five.

Now stretch it without turning it.

2q = [8, 6, 0]

Eight and six stand in the same ratio as four and three, so the arrow points exactly where it pointed before: same slant, same axis with nothing on it, same everything a direction consists of. The length is now ten. Only the length moved.

Vector magnitude is the length of a vector measured from the origin.

That is a complete description of one vector and it settles nothing about a pair of them. The doubled query is twice as long as the original and points at precisely the same content; one of the stored chunks, whichever length its own vector happens to carry, points somewhere else. Length is a property each vector has on its own, computed without reference to any other vector, so no comparison of lengths can report whether two vectors point toward similar content. The quantity has a name now. What it does not yet have is a rule for getting it out of coordinates, since reading five off a picture works for three numbers and not for a thousand.

L2 norm#

The rule is arithmetic on the components, and the arithmetic is worth performing in full on the vector whose answer is already known.

q = [4, 3, 0]

square each component        16, 9, 0
sum them                     25
take the square root         5

That five is the query vector's magnitude, obtained here by applying the L2 norm.

Run it on the chunk that ought to be answering the question.

L2 norm of webhook-timeouts

a = [3, 1, 0]

square each component        9, 1, 0
sum them                     10
take the square root         3.162

Three units of timeout-configuration content, one of failure content, nothing on billing, and a length of 3.162 — shorter than the query's five, which says only that the chunk's coordinates are smaller and nothing yet about what it is about.

Move a single component and the length moves with it. Holding the first coordinate at 3 and the third at 0, raise the second from 1 to 3.

[3, 3, 0]

square each component        9, 9, 0
sum them                     18
take the square root         4.243

The same formula, one changed input, and the magnitude the formula reports has changed from 3.162 to 4.243 to match. The stored vector for webhook-timeouts remains [3, 1, 0]; the variation exists to show that the number tracks the coordinates.

An L2 norm is the square root of the sum of a vector's squared components.

Squaring and square-rooting is one particular geometric choice among the possible ways of turning a list of coordinates into a length, and it is the choice that corresponds to straight-line distance. Other norms exist and report different numbers for the same vector — but each of them, this one included, reports a fact about coordinates rather than a judgement about meaning: 3.162 does not say that this chunk is a good answer, a bad answer, or an answer at all.

Dot product#

Comparing lengths cannot compare vectors, so the operation that compares them has to touch both sets of coordinates at once. It does so by pairing them up: the dot product is the sum of the products of corresponding components.

Work it first on coordinates carrying signs, because the signs are where hand computation goes wrong and a sign error is enough to reorder a result set.

a = [1, -2, 3]
b = [4,  0, -1]

first components     1 × 4     =  4
second components   -2 × 0     =  0
third components     3 × -1    = -3

sum                             =  1

Three things in that arithmetic are worth reading off it directly. The second component of a is paired with a zero, and a pair containing a zero contributes nothing to the total no matter how large the other member is. The third pair has opposite signs, so its product is negative and it subtracts from the running total rather than adding to it — dropping that minus gives 7, which is the ordinary slip. And what comes out at the end is a single signed number, not a vector: the operation consumes two lists of three and returns one scalar.

The operation needs the vectors to carry the same number of components, since a component with no partner has nothing to be multiplied by, and it needs the components paired by position, since pairing the first of one with the second of the other computes something else entirely. When the sum comes out negative, the vectors point in broadly opposing directions.

The retrieval vectors satisfy both requirements — query and chunk come out of the same embedding model, so they carry the same components in the same order, and every coordinate is a count of relevant material and so never negative. Back to that task, then, with the query and the chunk that answers it.

Query against webhook-timeouts

q = [4, 3, 0]
a = [3, 1, 0]

4 × 3 = 12
3 × 1 =  3
0 × 0 =  0

q · a = 15

Now the retry material. Consider it first as it would have been embedded on its own, before the operational safeguards were appended to the same chunk — the same query, a shorter passage covering retries and backoff and little else.

b₀ = [1, 3, 1]

4 × 1 =  4
3 × 3 =  9
0 × 1 =  0

q · b₀ = 13

Thirteen against fifteen. On this scoring, the chunk that answers the question comes first, which is the outcome the developer wanted.

The chunk actually in the index is not that passage. It is that passage with dead-letter handling, alerting thresholds, rate limits and billing implications written into the same chunk, which raises every coordinate and pulls the direction toward the billing axis that the query has nothing on.

Query against retry-and-safeguards

b = [2, 6, 3]        the same query, the retry passage extended

4 × 2 =  8
3 × 6 = 18
0 × 3 =  0

q · b = 26

Twenty-six. The extended chunk is less aligned with the query than the short retry passage was — it now carries content on an axis the query is silent about — and its score went up rather than down, past the fifteen that the direct answer earned. That is the ranked result from the opening, derived.

What the dot product computes, then, is alignment and scale together, in one number, with no way to read off which contributed what. Where magnitude carries meaning that ranking should respect — a confidence weight baked into a vector, a deliberately larger representation for an item that deserves to win ties — that combination is the desired behaviour. Where magnitude is an artifact of how long a passage happened to be, it is a ranking factor nobody chose, and it passes straight through this operation into the score.

Vector normalization#

Removing scale from the comparison means removing it from the vectors, and the L2 norm is already the number that measures it. Divide each vector by its own length.

q = [4, 3, 0]     length 5

[4/5, 3/5, 0/5]  =  [0.8, 0.6, 0]

The components are still in the ratio four to three, because every one of them was divided by the same number, and a ratio survives division by a common factor. The arrow points where it pointed. Its length is no longer five.

a = [3, 1, 0]     length 3.162   ->  [0.949, 0.316, 0]
b = [2, 6, 3]     length 7       ->  [0.286, 0.857, 0.429]

Each chunk vector divided by its own length, which is a different number for each of them — 3.162 for one and 7 for the other — so this is not a common rescaling of the set. It is each vector individually stripped of whatever length it happened to have.

Vector normalization is dividing a vector by its L2 norm to give it unit length.

The operation has a hole in it and a cost. The hole is the zero vector — the vector whose every component is zero, [0, 0, 0] in these three dimensions: its L2 norm is zero, dividing by zero is undefined, and a write path that can produce a zero vector — from an empty chunk, a failed extraction, a document that reduced to whitespace — needs to reject or flag that vector rather than hand it to a division. The cost is that the length is gone, permanently, from everything computed downstream. An application where length carried something real — a corpus where longer vectors genuinely correspond to richer, better documentation, established by measurement rather than assumed — has just thrown that signal away along with the artifact.

Unit vector#

Dividing by the length produced something with a property worth checking rather than assuming.

[0.8, 0.6, 0]

square each component        0.64, 0.36, 0
sum them                     1.0
take the square root         1.0

[3/√10, 1/√10, 0]

square each component        9/10, 1/10, 0
sum them                     1
take the square root         1

The second of those is exact rather than rounded, because squaring divides out the root: each squared component is the original squared component over 10, and the original squared components summed to 10.

A unit vector is a vector whose L2 norm equals one.

Vector normalization and the unit vector are the operation and its output, which is why they share a definition between them — the operation is named for the length it produces, and the artifact is named for the length it has. Feed the operation a vector of any length and the artifact comes out the same.

2q = [8, 6, 0]    length 10   ->  [0.8, 0.6, 0]

The same unit vector the original query produced. Scale is exactly what the operation discards, so every vector along a given direction collapses onto one representative of that direction, and comparisons between unit vectors are comparisons of direction with nothing else in them.

That simplification is about geometry and stops there. A unit vector is a direction in a space built by a model, and putting it at length one does not make the text it represents true, does not make it follow from anything, and does not make it consistent with any other text. Entailment is a relation with an owner of its own, and factual validation is a separate discipline again; neither is computed by a division, and a score of any size between unit vectors is not evidence about either.

There is an operational hazard specific to this artifact, and it appears in running systems rather than in worked examples, because the arithmetic never goes wrong — the precondition does.

To see how that happens, take an ordinary ingestion path, one that calls a third-party embedding provider for its vectors. The index is configured for inner product, chunks are embedded and stored exactly as they come back from the provider, and there is no normalization step anywhere in the write path.

def index_chunk(chunk_id, text):
    vector = embed(text)
    index.upsert(id=chunk_id, values=vector)

The provider's model happens to return vectors that already have unit norm. So the inner product the index computes and the cosine it would compute are the same number, the rankings are correct, and every retrieval evaluation the team runs passes. Nothing here is a bug. The system is relying on an unstated precondition that currently holds, and nothing in the index, the schema or the code above asserts it or would notice its absence.

Two ordinary changes break it without raising anything. The first is a different embedding model, including a later version from the same vendor, whose outputs are not unit length — the call site is unchanged, the vectors are stored the same way, and the scores silently start carrying length again. The second is constructing a stored vector by averaging or summing several chunk embeddings, for a document represented by more than one passage. Averaging unit vectors does not yield a unit vector: pointing in different directions, they partly cancel, and the average of vectors that disagree comes out shorter than the average of vectors that agree. A document whose passages are topically tight then gets a longer stored vector than one whose passages scatter, and that length goes straight into every score.

In both cases the ranking becomes magnitude-sensitive again, no exception is raised, no configuration was changed, and the only symptom is retrieval that got worse. The repair is to make the property something the write path enforces rather than something it inherits from whichever model is currently in use.

def index_chunk(chunk_id, text):
    vector = embed(text)
    norm = math.sqrt(sum(c * c for c in vector))
    index.upsert(id=chunk_id, values=[c / norm for c in vector])

Configuring the index for cosine instead achieves the same guarantee by having the index divide the lengths out at query time. Either way the property the scoring depends on is established by something in the system rather than assumed of something outside it.

Cosine similarity#

Configuring the index that way works because of what the division does to the arithmetic underneath it. The dot product's problem is that scale rides along in the score, and the length of each vector is a number already in hand. Divide it out.

Cosine of the query and retry-and-safeguards

q · b  = 26
|q|    = 5
|b|    = 7

26 / (5 × 7)  =  26 / 35  =  0.743

Apply that division to a pair already in hand and the effect is visible in one line: the same two vectors, unedited, scored twenty-six in the previous section and score 0.743 here. Dividing by the product of the lengths puts every score for every pair on the interval from −1 to 1, since neither vector can contribute more than its own length to the sum of products.

Cosine similarity is the dot product of two vectors divided by the product of their magnitudes.

Dividing each vector by its length before multiplying, rather than dividing the product afterwards, reaches the same number by the same arithmetic — which is exactly Zilliz's demonstration that cosine similarity is the dot product of normalized vectors. Google's treatment gives the formal version, and Elastic's shows the operational one, where the division is a metric setting on an index rather than a line of code. What the number measures is directional proximity: how nearly two vectors point the same way, with length removed by construction — still no measure of entailment and no factual check, and still not a quantity whose value carries the same meaning from one embedding model to the next.

A vector index will normally offer a third metric beside cosine and inner product, and it is worth naming here because its relationship to what has already been derived settles several questions at once. Euclidean distance, also called L2 distance, is the straight-line distance between two vectors: how far apart their tips are. It shares its formula with the L2 norm, which is why the names collide — the L2 norm measures one vector's length from the origin, and Euclidean distance is that same L2 norm applied to the difference a − b. One argument versus two, same square-root-of-squared-components underneath.

Take the query and the chunk that answers it, both already normalized, and expand the squared distance between them rather than taking the relationship on assertion.

‖a − b‖²  =  ‖a‖²  +  ‖b‖²  −  2(a · b)

both vectors are unit length, so ‖a‖² = 1 and ‖b‖² = 1

‖a − b‖²  =  1 + 1 − 2(a · b)  =  2 − 2(a · b)

Every term on the right is fixed except the dot product, and the dot product enters with a minus sign in front of it. So the squared distance falls exactly as the dot product rises, and on unit vectors the dot product is the cosine. Check it on the pair.

q̂ · â  =  0.8(0.949) + 0.6(0.316)  =  0.949
‖q̂ − â‖²  =  2 − 2(0.949)  =  0.103      ‖q̂ − â‖  =  0.320

q̂ · b̂  =  0.8(0.286) + 0.6(0.857) + 0(0.429)  =  0.743
‖q̂ − b̂‖²  =  2 − 2(0.743)  =  0.514      ‖q̂ − b̂‖  =  0.717

The consequence is the one to be able to state cold. When every vector involved has unit length, ranking by descending cosine, ranking by descending dot product and ranking by ascending Euclidean distance return documents in the identical order. The scores differ — 0.949 against 0.320 for the same pair — and the order does not, so choosing among those metrics on unit vectors causes no re-ranking whatsoever.

That equivalence rests entirely on the normalization. The step that collapsed the distance onto the dot product was setting each squared norm to 1, and on vectors that are not unit length those terms stay in the expression: ‖a‖² and ‖b‖² vary per document, Euclidean distance responds to where a vector sits in the space rather than only to which way it points, and cosine, inner product and Euclidean distance can then order the same corpus three different ways.

One more pair through the same formula, because it decides what a high cosine is evidence of. Take a claim from the timeout documentation and its exact negation, embedded by a general-purpose sentence embedding model.

the payment webhook retries after a timeout
the payment webhook does not retry after a timeout

The cosine comes out high — in the same band as two paraphrases of a single sentence, rather than anywhere near zero or below it. The mechanism is visible in the sentences themselves. A general-purpose embedding model is trained to place text according to what it is about, and those sentences are about the same thing: the same webhook, the same timeout, the same retry behaviour, the same entities, nearly every token shared and in the same order. Whatever direction payment webhook retry timeout occupies in the model's space, each sentence is pointing almost entirely along it. The one word carrying the logical polarity is a small perturbation to a vector that was already committed by everything around it.

The explanation a reader reaches for first is contrastive training, and it is worth ruling out on its own terms rather than waving past. Contrastive objectives pull related text together and push unrelated text apart, and the pairs they push apart are pairs about different things: a webhook timeout passage against a passage on invoice formatting. That is a statement about topical relatedness. Nothing in it teaches a model that a sentence and its negation are opposites, because a sentence and its negation are maximally related by the only criterion the objective ever applies. Contrastive training makes contradictions score high for the same reason it makes paraphrases score high. The distinction between a claim and its negation is drawn by a model trained on labelled entailment and contradiction pairs, which is a different model performing a different task from the one being run here.

So a high cosine between a claim and its exact negation is the metric doing what it is specified to do. It is not noise, not a defect in the embedding, and not something a higher threshold removes.

Magnitude bias#

Everything so far has had the geometry coming out right — the claim and its negation scored where their directions said they should. The failure real systems run into does not come from the geometry being wrong; it comes from which scoring operation the index was configured to use.

Magnitude bias is a ranking distortion in which vector length affects similarity scores independently of direction, so that a longer stored vector outranks a shorter one that points at least as well toward the query.

Replay the developer's request against both scoring operations, with everything else held identical.

                          q · d      cosine
webhook-timeouts           15.0       0.949
retry-and-safeguards       26.0       0.743

Now change one thing and only one thing: the scale of the distractor. Suppose retry-and-safeguards had been split, so the stored chunk is half the size and its vector half the length, pointing in exactly the same direction as before.

b = [1, 3, 1.5]          the same direction, half the length

q · b   =  4 + 9 + 0  =  13.0
cosine  =  13 / (5 × 3.5)  =  0.743

The direction did not move, so the cosine did not move: 0.743 before and after. The dot product fell from 26 to 13 and dropped below the 15 that webhook-timeouts earns, so the ranked list under the raw score flipped. A number that reorders results when nothing but the length of a stored vector changed is measuring the length.

Which side of the comparison the normalization has to happen on is the part that gets guessed wrong, and the cleanest version of the question uses two documents that are directionally indistinguishable. Store document A as v = [3, 1, 0] and document B as 3v = [9, 3, 0]. They point in precisely the same direction; B is three times as long.

Raw dot product against v and 3v

q = [4, 3, 0]

q · v   =  12 + 3 + 0  =  15
q · 3v  =  36 + 9 + 0  =  45

B wins by exactly three, and it wins for every possible query, because q · 3v is 3(q · v) for any q at all — the factor of three multiplies out of the sum and lands on the total. Two documents that are indistinguishable as directions are ranked by storage length alone.

The repair most readers propose first is normalizing the query. Test it.

q̂ = [0.8, 0.6, 0]

q̂ · v   =  2.4 + 0.6  =  3.0
q̂ · 3v  =  7.2 + 1.8  =  9.0

Every score is one fifth of what it was, and the order is untouched: B still beats A, and still by exactly three. Dividing the query by its own length multiplies every document's score by the same positive constant, and a common positive factor applied to every entry in a list cannot change which entry is largest. Preparing the query differently changes the scale of the numbers printed next to the results and cannot change the results.

v  = [3, 1, 0]  length 3.162  ->  [0.949, 0.316, 0]
3v = [9, 3, 0]  length 9.487  ->  [0.949, 0.316, 0]

q · v̂    =  3.795 + 0.949  =  4.743
q · 3v̂   =  3.795 + 0.949  =  4.743

Dividing each stored vector by its own length divides one of them by 3.162 and the other by 9.487, which is what removes the factor of three, and the documents tie. A tie is the correct outcome for two vectors pointing the same way.

The rule to be able to give back is therefore asymmetric. Whether the query is normalized affects the scale of the scores and never the ranking; normalizing the stored vectors is what removes magnitude from the ordering. An inner-product index over unnormalized documents misranks no matter how carefully the query is prepared, and a team that normalizes the query and stops has done the half of the job that does not matter.

In production the symptom arrives before any of this, and the cause has to be inferred from it. Chunk embeddings are stored in an index configured for inner product, and across unrelated queries the longest chunk in the corpus comes back first almost every time — including for the webhook timeout query, where the chunk it beats is the one that answers the question.

That is magnitude bias, arriving through a specific correlation: chunk length correlates with vector norm, so the inner product is rewarding the norm on top of the direction. The fix is the one line already established — normalize the stored vectors, by whichever of the two routes the index offers — so that scoring compares direction only.

What that fix leaves untouched matters more than what it removes, because a team that ships it and stops will believe retrieval is repaired.

It does not fix the chunking. A long chunk covering several topics has a direction averaged across all of them, which is exactly what pulled retry-and-safeguards toward a coordinate the query was silent on. After normalization such a chunk stops dominating by size, and it still scores as a mediocre match for every query it partly touches, and the sentence that answers the developer's question is still buried inside a passage of operational material they now have to read past.

It does not fix any threshold or cut-off tuned before the change. Every score in the index has just moved onto a different scale — 26 became 0.743 on one pair alone — so a minimum-score filter, a confidence display or an escalation rule calibrated against the old numbers is now calibrated against nothing. And it forecloses the length signal for this corpus, irreversibly, in the way already described.

The remedy for what remains is re-chunking into smaller, topically coherent passages, so that a chunk's direction is the direction of one subject. It is not a further change of metric.

Numerical tolerance#

Fixing magnitude bias through re-chunking addresses what the vectors represent; the comparison operation itself carries a separate risk, in the arithmetic that computes it. Cosine computed by dividing a dot product by the product of two magnitudes, and cosine computed as the dot product of two already-normalized vectors, are the same quantity reached by different routes. In floating-point arithmetic they are usually not the same number.

direct    = dot(q, a) / (norm(q) * norm(a))
via_units = dot(unit(q), unit(a))
assert abs(direct - via_units) < 1e-9

direct      0.9486832980505138
via_units   0.9486832980505137
difference  1.1102230246251565e-16

The routes divide by the lengths at different points, each division and each square root rounds its result to the nearest representable double, and the roundings do not land in the same place. The difference here is one unit in the last place of a double near 0.95, which is the smallest possible disagreement between distinct doubles at that magnitude. An equality test would fail on it.

Numerical tolerance is an allowed difference used when comparing approximate floating-point results.

Its value is a decision, and the decision is visible when it moves.

tolerance 1e-9    |difference| < tolerance   ->  passes
tolerance 1e-17   |difference| < tolerance   ->  fails

At 1e-9 the assertion passes and keeps passing across platforms, compilers and vector lengths, since accumulated rounding over a few thousand components stays far below it. At 1e-17, below the spacing of doubles in this range, the assertion fails on arithmetic that is entirely correct, and the test reports a defect that does not exist.

The tolerance a numerical comparison and a retrieval cut-off each apply is a small number that decides whether a difference counts, which is what makes them easy to conflate, and they answer different questions. A numerical tolerance absorbs the gap between two ways of computing one quantity. It is not a licence to treat scores of 0.7431 and 0.7429 as interchangeable when the ordering between them is what a user sees, because that difference is not a rounding artifact — it is what the metric reports about two different documents.

Metric choice and boundary#

With tolerance settled, what remains open is the metric itself, and the conditions for choosing among the metrics an index offers follow from what each one was shown to do.

Read across the rows and the choice is operational — decided by which metric the index implements most cheaply, which one its approximate nearest-neighbour structure supports best, and whether the stored vectors are guaranteed unit length — rather than by which one is thought to understand meaning better.

Choosing well among them settles the ordering and nothing beyond it, since neither a similarity score nor a distance is a truth test. And a threshold on one does not transfer across embedding models or datasets, which is a rule that gets nodded at and then quietly disobeyed, so it is worth working rather than asserting.

A team measures a batch of documentation chunks they know to be equivalent, on one model and one corpus, finds the boundary between the equivalent pairs and the rest sitting around 0.80, and writes cosine > 0.80 means same meaning into the retrieval service as a global rule. What that constant records is not a fact about meaning. It is a cut-point on one model's score distribution, taken on one collection of documents.

Score the same labelled pairs under models whose distributions differ.

                        clearly unrelated     known equivalents
model with a wide
score distribution            ~0.10                 ~0.85

model whose space is
a narrow cone                 ~0.75                 ~0.95

On the first, 0.80 sits in the empty band between the groups and separates them cleanly. On the second, unrelated chunks already clear it, so the rule admits nearly everything and the service accepts unrelated pairs as the same meaning while raising no error and logging no anomaly. The identical number means barely related in one system and near-identical in the other, and a rule that is a real filter on one becomes a pass-through on the other.

Three diagnoses are available at that point and only one of them is the cause.

A different universal constant is not a repair. Raising the rule to 0.95 rejects genuine equivalents on the first model exactly as 0.80 accepted unrelated pairs on the second, because what differs between the models is the entire distribution — where unrelated pairs sit, where equivalents sit, how wide the gap between them is — and not the position of one number inside a fixed distribution. Any constant chosen without reference to a particular model's distribution is the same mistake with different digits.

Document length is not the cause either. Length distorts the ordering through magnitude bias, on an index whose stored vectors are unnormalized, and cosine has the lengths divided out by construction. On normalized vectors, two chunks of wildly different sizes can score anywhere at all with respect to each other, and nothing about the sizes determines where the calibrated cut-point falls.

And "cosine is not a probability", though true, is not the reason. Thresholds are applied to non-probabilistic scores constantly and legitimately; picking an operating point on a score that has no probabilistic interpretation is ordinary practice. A cut-point calibrated on one model and one corpus works on that model and that corpus without ever being a probability. What breaks it is the move to a different distribution, not the absence of a probabilistic reading.

That last sentence names the cause: the cut-point was fitted to one model's distribution on one corpus and then carried, unchanged, to a distribution it was never fitted to. The repair, then, is to calibrate the cut-point per model and per corpus against labelled pairs, and to re-calibrate whenever the model or the corpus changes. Which pairs, at what operating point, and how the resulting cut-off is stored belongs to the similarity threshold as its own subject, and to the vector index as the thing configured with it.

That prescription is easy to state and easy to half-apply, so one transfer case is worth deciding out loud, since these tools are meant to settle questions rather than to be recited.

A team builds a duplicate-clause detector across a contract corpus. Every clause is embedded, any pair scoring above cosine 0.85 is flagged as a duplicate, and on a sample it works — except for one report.

the Supplier shall indemnify the Customer against all third-party claims
the Customer shall indemnify the Supplier against all third-party claims

Those clauses assign the obligation in opposite directions, which is the difference between two commercial positions, and the detector calls them the same clause. It does not ship, and the reason is one the chapter has already established rather than a new mechanism. Reversing the roles changes almost no tokens and nothing about what the clause is about — same parties, same obligation, same third-party claims, same structure — so the direction of the embedding barely moves, exactly as it barely moved between a claim about webhook retries and its negation. The score is high because cosine reports aboutness, and there is no cut-off anywhere on the interval that separates same clause from reversed clause when both sit at the same angle. Raising the bar to 0.95 discards genuine duplicates and keeps this pair, because this pair is a near-perfect lexical match.

What ships instead keeps the score in the role it is good at. Use the embedding comparison as a cheap recall filter that narrows millions of clause pairs to a candidate list, and decide duplicate-or-not with a check that can see what cosine cannot: extract the parties, the obligation and the direction of the obligation from each clause and compare those fields, or hand the candidate pair to a model trained on labelled clause pairs for this decision. The operation sensitive to the direction of an obligation is then the operation making the call.

Set the filter's cut-off by the calibration just prescribed — labelled clause pairs drawn from this contract corpus, scored under the model actually deployed — and choose the operating point for high recall. A pair the filter drops never reaches the second stage and can never be recovered; a pair it passes wrongly costs one more check.

Every step of that checklist runs on a small vocabulary, and the terms are worth setting side by side. Magnitude is the length being measured and the L2 norm is the formula that computes it; normalization divides a vector by that length and a unit vector is what comes out; the dot product combines corresponding components and carries alignment and scale together; cosine similarity divides the scale back out; magnitude bias is what the undivided scale does to a ranking; and numerical tolerance is what makes two routes to the same score comparable in a test.

Together they make a scoring choice explicit and testable. They do not make retrieval relevant. Every one of them is arithmetic over coordinates that a model produced for reasons of its own, and what remains open after the metric is chosen is how that metric behaves on the actual model and the actual data — which is a measurement on a labelled query set, not a property of the operation.