Equal dimensions do not make two embeddings compatible

The ticket was titled "switch image embeddings to Provider B — cheaper, same dims" and it was estimated at two points. Provider A charged $0.00012 per image and Provider B charged $0.00007, both returned 1,024 floats, both documented cosine similarity as the metric. The change was one environment variable in the query service. It shipped to staging on a Tuesday.

Staging search did not throw. It did not return zero results, and it did not return obviously random ones — the top ten for "stainless steel kettle" were all kitchenware, which is what made the next four days expensive. The person testing it reported the results as "a bit worse, maybe the model is weaker", and the team spent two days benchmarking Provider B's model quality before someone thought to embed a single image with both providers and compare the two vectors directly.

img = open("kettle-01.jpg", "rb").read()
va = provider_a.embed_image(img)     # 1024 floats
vb = provider_b.embed_image(img)     # 1024 floats

print(cosine(va, vb))            # 0.021
print(cosine(va, provider_a.embed_image(open("bicycle-77.jpg","rb").read())))  # 0.284

Two unit vectors drawn at random in 1,024 dimensions have a cosine distributed around zero with a standard deviation of 1/√1024 ≈ 0.031. A cross-provider cosine of 0.021 for the same photograph is well inside that band: as far as the geometry is concerned, Provider B's encoding of the kettle and Provider A's encoding of the kettle are unrelated vectors that happen to have the same length. The staging results were not a weaker model; they were queries in one coordinate system being matched against a corpus in another, and the reason they looked like kitchenware is that the query happened to land nearest to a cluster whose contents were correlated with nothing in particular.

The diagnostic that would have saved four days takes ten images and about thirty seconds: embed each with both models, take the cosine of each vector with its counterpart, and compare the mean against 1/√d. Anything in the neighbourhood of that value means the two spaces are unrelated; anything above about 0.5 means the models share a lineage and you should still not mix them without measuring. It is a cheap enough check to run as the first step of any embedding migration, before the corpus job is even written, and it converts a class of failure that presents as "quality regression, cause unknown" into a one-line assertion.

Same corpus, both models, and the rankings still disagree#

Fixing the mixed-space bug — re-embedding the whole corpus with Provider B so that queries and documents share a model again — produces a system that works. It does not produce the same system.

Run the 900-query blind evaluation from the previous chapter against two indexes over the identical 300 images, one built with each provider, and compare the result lists rather than the scores.

import numpy as np
from scipy.stats import kendalltau

def agreement(hits_a: list[list[str]], hits_b: list[list[str]], k: int = 10):
    jac, taus, top1 = [], [], []
    for a, b in zip(hits_a, hits_b):
        sa, sb = set(a[:k]), set(b[:k])
        jac.append(len(sa & sb) / len(sa | sb))
        shared = [x for x in a[:k] if x in sb]
        if len(shared) >= 3:
            taus.append(kendalltau([a.index(x) for x in shared],
                                   [b.index(x) for x in shared]).statistic)
        top1.append(a[0] == b[0])
    return np.mean(jac), np.mean(taus), np.mean(top1)

jac, tau, t1 = agreement(results_provider_a, results_provider_b)
print(f"Jaccard@10 {jac:.2f}   Kendall tau on overlap {tau:.2f}   top-1 identical {t1:.0%}")
# Jaccard@10 0.34   Kendall tau on overlap 0.21   top-1 identical 41%

A Jaccard of 0.34 at k=10 means the two lists share about five of their thirteen distinct members, and the Kendall tau of 0.21 — a rank correlation over the members they do share, where 1.0 is the identical order and 0 is no relation at all — says those five are not even arranged alike; the top result matches on fewer than half the queries. Aggregate quality, meanwhile, was nearly indistinguishable: recall@10 of 0.66 for A and 0.69 for B, a difference of 0.03 that a paired bootstrap — drawing 900 queries with replacement from the 900 you have, a few thousand times over, and recomputing both systems' scores on every draw to see how far their difference wanders — put at a 95% interval of [−0.01, +0.07]. The two models are equally good and substantially different, which is the combination that makes model choice feel arbitrary and makes model change expensive: nothing tells you the swap was wrong, and every downstream artefact tuned against the old rankings — the score cutoffs, the cached results, the golden-set assertions in CI — is now calibrated to a system that no longer exists.

The aggregate tie in that table is also concealing a real result, which the preprocessing rows predict. Broken out by query type, Provider B was ahead by 0.09 on queries naming something small or off-centre — a consequence of resizing to 384 pixels without cropping, so the edges of the frame survive — and behind by 0.06 on long descriptive queries, where A's shorter text encoder was not the limitation and B's was doing something else worse. Neither difference is visible in 0.66 against 0.69. Choosing between two models on a single aggregate figure is choosing on the average of populations you care about unequally, and the fix costs nothing beyond grouping the evaluation you already have.

The third column is the one that settles the mechanism. It is the same architecture, same code, same training data as Provider A, differing only in the random seed used for initialisation and data ordering. Its retrieval quality is identical within noise, and its vectors are as incompatible with A's as a different vendor's — Jaccard 0.29, cross-model cosine 0.014. Compatibility is not a function of architecture, dimension, training data or provider. It is a function of the specific weights, because cosine similarity is invariant under any global rotation of the space: rotate every embedding by the same orthogonal matrix and every within-model similarity is unchanged, so the training objective has no way to prefer one orientation over another and no reason to land on the same one twice. Each training run picks an arbitrary basis, and "arbitrary" is doing all the work in that sentence.

The differences listed above the fold — crop versus no crop, 77 tokens versus 512, one channel normalisation versus another — compound that. A 384-pixel input preserves detail a 224-pixel centre crop deletes, and a centre crop deletes the edges of the frame entirely, so for a photograph whose subject is off-centre the two models are not even looking at the same picture. A 77-token text encoder truncates any query longer than roughly fifty words, silently. These are the reasons the two models make different mistakes; the rotation is the reason their vectors cannot be mixed even where they agree.

Aligning two spaces instead of re-embedding#

There is a partial escape, and it is worth knowing so you can decline it deliberately. If you have N items embedded by both models, you can fit a single orthogonal matrix that maps one space into the other — the orthogonal Procrustes problem, solved in closed form by an SVD of Bᵀ A.

import numpy as np

def procrustes(source: np.ndarray, target: np.ndarray) -> np.ndarray:
    """Least-squares orthogonal map W with source @ W ≈ target."""
    u, _, vt = np.linalg.svd(target.T @ source)
    return (u @ vt).T

W = procrustes(anchors_a, anchors_b)      # (1024, 1024), fit on paired anchors
migrated = old_vectors_a @ W              # reuse A's corpus against B's queries

On the appliance corpus with 50,000 anchor pairs, mapping A's corpus vectors into B's space recovered recall@10 of 0.56 against B's native 0.69 — about 81% of the quality, for the cost of embedding 50,000 images twice rather than four million once. The gap is structural rather than a tuning failure: an orthogonal map can only rotate, and the two models differ in what they encode, not merely in how they orient it. Learning a full linear map instead of an orthogonal one recovers a little more and overfits the anchor distribution; a small MLP recovers more still and needs a training pipeline, a validation set and its own monitoring, at which point you have built and now own a model in order to avoid running a batch job.

The arithmetic usually kills it. Re-embedding four million images at Provider B's $0.07 per thousand is $280 and about seven hours of wall clock at a few hundred images per second. Procrustes alignment saves most of that money and costs 13 points of recall permanently, on every query, forever. Alignment earns its place when re-embedding is impossible rather than merely tedious — an archive whose source images have been deleted and only vectors survive, or a third-party embedding feed you do not control. For a corpus you still hold, re-embed.

What this is not#

It is not the modality gap. That gap sits inside one model, between its image cloud and its text cloud, and it is stable and measurable within a single space. This is a relationship between two models, and it is not a gap but an absence of any relationship at all: there is no fixed offset to subtract, because the two spaces do not share an orientation to be offset from.

It is not fixed by any post-processing you can apply to one vector in isolation. Re-normalising does nothing, since both sets are already unit length. Whitening or PCA within each space changes the axes but not the correspondence — two whitened spaces are still two spaces, and PCA's component ordering is only defined up to sign, so you introduce fresh arbitrary flips. Truncating both to a common 512 dimensions, which works within a single Matryoshka-trained model, does nothing across models beyond discarding half of each incompatible space.

And equal dimensionality is not even a requirement, only an appearance of one. A 768-dimension model and a 1,024-dimension model are incompatible for the obvious reason that the arrays are different lengths, and the equal-width case is dangerous precisely because it removes the obvious reason. The type checker passes, the vector database accepts the insert, the cosine returns a float in [−1, 1], and nothing fails until relevance quietly degrades.

Treating the model as part of the schema#

Since the model identity determines the meaning of every vector, it belongs in the schema next to the vector, not in a config file. The minimum viable version is a column and a constraint.

ALTER TABLE image_vectors ADD COLUMN model_id text NOT NULL;
ALTER TABLE image_vectors ADD CONSTRAINT one_space_per_index
  CHECK (model_id = current_setting('app.active_embedding_model'));
CREATE INDEX ON image_vectors USING hnsw (vector vector_cosine_ops)
  WHERE model_id = 'provider-b/embed-v4';

The identity has to be the exact served version, not the friendly alias, because providers ship model updates behind stable names. An endpoint documented as embed-v4.0 can be re-pointed to a retrained checkpoint on the provider's schedule, and when that happens your queries move to the new space overnight while your corpus stays in the old one — the mixed-space failure again, arriving without a deploy on your side, and presenting as a gradual relevance complaint rather than an incident. The defences are to pin a dated or hashed model identifier where the provider offers one, and to run the ten-image cosine check as a scheduled job against a stored reference vector so that a silent re-point sets off an alarm rather than a support ticket. Self-hosted weights are immune to this and acquire the equivalent problem in a different place, since a container image rebuild that picks up a new checkpoint does the same thing.

The query path then carries the model identity rather than inheriting it: the service reads which model an index was built with and embeds the query with that model, so a partial rollout cannot produce the staging failure that opened this chapter. A migration becomes a dual-write window — new assets embedded with both models, the old index still serving, the new index backfilling — followed by a shadow evaluation on the blind query set, a cutover, and a deletion once the new index has been serving long enough that rolling back is not the plan. The window costs double storage, which for four million 1,024-dimension float32 vectors is 16.4 GB per copy, so the migration's true peak footprint is 32.8 GB rather than 16.4 GB, and the plan should say so.

Accepting that a model change means a full re-embedding pass changes how the choice gets made. It becomes a decision you make a handful of times, on evidence, with a number attached — which puts the weight on knowing what the model actually buys you. That answer is not a single number, because it depends entirely on what the query asks for, and there is a whole class of queries no model in this family can serve at any dimension or price.