Similarity metrics and matching the metric to the model

Read these first

• What an embedding is

What you will know

• Choose the metric the model was trained for, not the one that's default

  Verify the embedding model card or training configuration and record the corresponding index setting so deployment cannot silently change it.

• Run the same query-corpus set with and without vector normalisation and verify when cosine and dot product produce identical rankings

  The equivalence holds when both vectors are unit length; otherwise dot product also rewards vector magnitude and can reorder results.

• Show a wrong-metric configuration degrading results

  Compare a documented metric configuration with a deliberately mismatched one using recall@k or ranking agreement, while keeping embeddings and data fixed.

• Compute similarity between two embeddings without a library

  Computing it by hand once removes the mystery from every similarity threshold discussion.

• Measure score distributions for positive and negative pairs across two embedding models before setting a threshold

  Identical numeric cutoffs can represent very different relevance probabilities because score scale and concentration vary by model and preprocessing.

Chapters

1. 1

   Cosine is a dot product divided by two norms

   Take two four-dimensional vectors and work all three numbers out by hand: the dot product, the norms that turn it into a cosine, and the squared Euclidean distance that expands into those same terms. Doing this once strips the mystery out of every later argument about whether 0.75 counts as a high score, and shows precisely which quantity each metric is sensitive to.

   Read chapter
2. 2

   Unit length collapses three metrics into one ranking

   When every vector has norm 1 the denominator cosine divides by is 1, so dot product orders results identically and squared Euclidean distance simply falls as cosine rises — three index settings, one ranking. Drop that condition and dot product starts rewarding sheer magnitude, which can lift a long, vague document above the passage that answers the query. Run the same query set both ways and watch rank 1 change.

   Read chapter
3. 3

   The model card names the metric; the database default cannot

   A card stating that a model was trained with dot-product similarity on unnormalised vectors is telling you something was encoded in magnitude, and letting the vector database's cosine default stand throws that away — the result reads as a mediocre model rather than a misconfiguration. Pin the metric next to the model in configuration, and price the mistake by running the documented and mismatched settings over fixed embeddings.

   Read chapter
4. 4

   A half-normalised index ranks by magnitude, not relevance

   During a migration the bulk loader normalises and the incremental upsert path does not, so half the corpus carries norm 1 and half does not while the index scores with dot product; the unnormalised half then outranks everything regardless of relevance. Use that failure shape to triage the classic report of recall@10 falling from 0.81 to 0.63 with model, corpus and queries untouched.

   Read chapter
5. 5

   An approximate index returns the neighbours its search effort happened to find

   A vector index does not hand back the true nearest neighbours; it hands back what a tunable amount of search effort reached, and two systems sitting at their own defaults explore different amounts. That is the third hypothesis for a migration that loses recall with model, corpus and queries untouched, and it is the one to test last, because its fix buys quality back with latency permanently while leaving a real configuration defect in place. The one-line check is to run the same queries against an exact brute-force scan over the same vectors and measure the overlap with what the index returned: if exact and approximate agree, the problem is metric or normalisation. The size of the drop points the same way — approximation erodes the tail gradually rather than moving an aggregate eighteen points.

   Read chapter
6. 6

   A threshold is a property of the model that produced the scores

   Two models can both put positives near the top of their range and mean completely different things: 0.85 against 0.78 is compression, 0.7 against 0.25 is separation. Measure the score distribution over labelled positive and negative pairs before fixing any cutoff, and see why linearly rescaling an inherited cutoff preserves the overlap and therefore the error rate. Then replace the instrument rather than the number: cut on the relative gap between consecutive scores in the top k, calibrate raw scores into a relevance probability using the labelled pairs you just collected, or normalise scores within each query so that one fixed line stops competing with query difficulty.

   Read chapter