Two results when there should be twenty is truncation, not poor relevance

Truncation is a search returning fewer results than were asked for and fewer than exist, with no error. It is the user-visible end of both failures in this topic so far: post-filter starvation discards eligible candidates it never fetched, and a filtered graph walk exhausts its budget before it finds them. Poor relevance is a different failure entirely — a full page of results that are not the ones the user wanted. Both arrive as the same sentence from a customer, and the two teams that can fix them are not the same team.

The distinguishing feature lives on the response envelope rather than in the content: a result count below the requested k while the query itself succeeded. That is the whole signal, and it is enough, provided one other number exists to compare it against. This chapter is about producing that number cheaply on live traffic, alerting on it without being paged by every legitimately narrow search, and choosing between the fixes it makes available.

The comparison that turns a complaint into a metric#

Short lists are not automatically wrong. If a predicate matches four documents, four results is the complete and correct answer, and a system that alerted on every short list would alert constantly on small tenants and narrow folders. So the raw count cannot be the detector. What distinguishes a correct short list from a truncated one is the comparison between what came back and what could have come back, which means the detector needs a second number that the search path does not naturally produce.

That second number is an exact cardinality: how many documents match this predicate. Exact, rather than the estimate the router uses for plan selection, because the estimate is part of the system under test and a router misjudging a correlated predicate would otherwise hide the very truncation it caused. For equality predicates on indexed attributes it is a bitmap cardinality — a handful of microseconds — and for anything unindexed it is a count that may cost milliseconds, which is why the detector is worth running on all traffic for the cheap case and on a sampled slice for the expensive one.

def search_with_truncation_check(q_vec, predicate, k=10, sample_rate=0.01):
    hits = retrieval.search(q_vec, predicate, k=k)

    if cardinality.is_exact_cheap(predicate):
        n_eligible = cardinality.exact(predicate)
    elif random.random() < sample_rate:
        n_eligible = cardinality.exact(predicate)      # sampled shadow count
    else:
        n_eligible = None

    if n_eligible is not None:
        expected = min(k, n_eligible)
        span.set_attributes({
            "retrieval.k": k,
            "retrieval.returned": len(hits.ids),
            "retrieval.eligible": n_eligible,
            "retrieval.expected": expected,
            "retrieval.truncated": len(hits.ids) < expected,
            "retrieval.fill_ratio": len(hits.ids) / expected if expected else 1.0,
            "retrieval.plan": hits.plan,
            "retrieval.selectivity": n_eligible / CORPUS_SIZE,
        })
    return hits

fill_ratio is the metric worth graphing, and its three useful properties all come from the choice of denominator, which is worth deriving rather than accepting. The denominator is min(k, n_eligible) — the same expression the previous chapter used for recall, for the same reason. A query can never return more than k because that is what it asked for, and it can never return more than n_eligible because that is all that exists, so the smaller of the two is the largest honest answer.

Each property follows. Dividing by the largest achievable value bounds the ratio in [0, 1], because the numerator can never exceed the denominator. Taking the minimum rather than k alone makes the ratio defined and equal to one for legitimately short queries: three results out of three available divides to exactly one, so a narrow predicate never fires. And because the numerator is a count rather than a flag, the ratio degrades continuously — four results out of ten available is 0.4 — so a partial failure is visible as a partial value rather than having to cross a boundary before it registers.

Those three together are what make an aggregate alert possible. An alert on the p10 of fill ratio — the value that the worst tenth of queries in the window fall below — dropping under 0.9 over a fifteen-minute window catches a broken folder on the first few queries rather than on the first complaint, and grouping the metric by tenant and by filter shape tells you where before anyone opens a ticket.

The triage from a short result list splits along the two numbers, and the branches lead to genuinely different owners:

Each branch is a question whose answer eliminates one cause, which is why the order matters. The first asks whether anything was missing at all, and a no ends the investigation. The second asks whether the engine was told about the predicate, which separates a plan bug from an index limit. The third asks whether an exhaustive comparison over the eligible set can do better than the index did, and this is the branch that carries the most information for the least work, because an exact scan has no failure mode of its own.

The last branch is the one people forget. If an exhaustive scan of the eligible set also comes back short, the documents matching the predicate do not have vectors in the index — an ingest failure, a partially-applied backfill, or metadata written before embedding completed. That failure produces the identical user-visible symptom and none of the fixes in this chapter touch it, which is reason enough to make the exact scan part of the triage rather than assuming the cliff.

Alerting on this needs a little restraint, because a per-query alert on truncated = true will fire on every legitimately narrow search the moment n_eligible estimation is even slightly off. Aggregate first: alert on the fraction of queries with fill_ratio < 1.0 within a group, where the group is a tenant, a filter shape, or a selectivity band, and require both a rate threshold and a minimum query count so that a tenant issuing four queries an hour cannot page anyone. A rule along the lines of more than 20% of queries truncated, over at least 50 queries, in one group, for fifteen minutes is quiet in normal operation and fires within minutes of a change that pushes a group over the cliff.

Raising k is the fix everyone tries first#

The instinct on seeing two results where ten were wanted is to ask for more, and it is worth being precise about why that mostly does not work, because it sometimes appears to.

Under post-filtering, raising top_k genuinely does more work: the index returns more blind candidates and more of them survive the predicate. This is the first chapter's product read forwards. At 40% selectivity, moving from top_k = 10 to top_k = 30 takes the expected survivor count from 30 multiplied by two fifths, which is 12, up from 4 — and twelve survivors fills a page of ten. The over-fetch needed is k_fetch ≈ k / s, so the multiplier over k is one divided by s, and it grows without bound as selectivity falls: 25× at 4% selectivity, 1,000× at 0.1%, and at the legal-tech selectivity from chapter 1 a fetch of 11,400 candidates in order to keep ten.

A team that fixes a complaint by raising top_k from 10 to 100 has bought correctness down to about 10% selectivity — a hundred candidates at one document in ten is ten survivors, exactly a page — and has not changed anything about what happens at 0.5%. The fix appears to work because the tenant who complained was in the band it covers, and the failure returns the next time a narrower filter appears.

Under filtered ANN, raising top_k alone does close to nothing, and this is the stumble worth naming. top_k is the size of the answer the engine is asked to produce. ef_search is the size of the search it is allowed to perform. Asking a walk that explored 128 candidates for 100 results instead of 10 does not make it explore more candidates, so top_k is a quantity that does not move when the budget's adequacy moves — it cannot separate a healthy filtered query from a starved one, and raising it cannot repair one. The reason the confusion is tempting is that under post-filtering the two are the same knob, because there the candidate list is the answer list. Pushing the predicate into the engine splits them apart, and the knob everyone learned to reach for stays behind with the old plan. ef_search is the knob that moves, and chapter 2 measured what raising it buys: from 128 to 512 at 0.03% selectivity, result count went 4.6 to 8.2 for four times the work.

Read the two latency rows against each other rather than down the columns. At 40% selectivity all three plans are within a few milliseconds of one another, so nothing in the table can be decided from the loose-filter case. At 0.05% they separate by more than two orders of magnitude, and they separate in correctness as well: the first column is still short after 210 ms while the other two are complete in a fraction of the time. A comparison run only at the selectivities that dominate traffic would find all three columns equivalent, which is how a configuration change gets adopted as a fix.

Over-fetching is not worthless, and the table is not an argument for skipping it during an incident. It is the only column that ships the same day, it requires no new code paths, and for a product whose filters are mostly coarse — a status flag, a language, a workspace with thousands of members — raising top_k to 100 and ef_search to 256 may genuinely cover the traffic that exists. What it cannot do is hold, because selectivity drifts in one direction as a product matures: document sets grow, customers add taxonomy, and the filters users build get narrower every quarter. A configuration chosen to cover today's tightest filter is a configuration that will be quietly insufficient in six months, with no deploy to correlate the regression against.

The middle column is worth more attention than it usually gets, because it converts the problem into one the system can notice at runtime. When len(hits) < min(k, n_eligible) and n_eligible is small enough to scan, re-run the query as an exact scan and return that instead. It reuses the detector you already built, its cost falls entirely on the queries that were broken, and it degrades gracefully — as truncation becomes more common the fallback rate rises, which is itself a metric telling you the routing thresholds need moving. It is the change to make in the week after an incident, with the router as the thing you build once the traffic distribution is understood.

The queries that never generated a ticket#

A user noticing two results is the most visible point on a distribution, and it is not the most damaging one. The same mechanism that returned 2 of 10 for the Compliance folder returned 7 of 10, 8 of 10 and 9 of 10 for thousands of other queries that week, and nobody filed a ticket for any of them, because a page of eight plausible results looks like a page of results.

That is the second stumble in this chapter, and it is a belief about a defence rather than a slip: the belief that user reports are an adequate detector. They are not, and the fill-ratio definition says exactly why. A complaint requires the ratio to fall far enough that a human notices, which is somewhere near two results out of ten. Every value between there and one is invisible to the complaint channel and fully visible to the metric. Every one of those queries handed a downstream model a context window with a document missing from it, and the model answered from what it had, fluently and without hedging. The complaint you received is the part of the failure that happened to cross a threshold of human noticeability; the fill-ratio histogram is the part that has been affecting answer quality all along.

There is a second thing to be uneasy about, which is what the filter was doing. If the predicate in question is a permission or tenancy predicate, then the code path that can return too few documents is the same code path responsible for never returning the wrong ones. Truncation and leakage are opposite failures of one mechanism, and the asymmetry between them is total. Truncation produces a short list, which a metric can see and a user can report. Leakage produces a full list of plausible results, so it moves no count, no latency and no fill ratio, and the user who receives another organisation's document has no way to know it is not theirs.

Everything in this topic so far has been detectable because the failure removed something. Nothing here detects a failure that adds something, and no amount of sharpening the fill-ratio alert will change that — the metric is constant across exactly the axis a leak varies along. That is what makes tenant isolation a different kind of problem from everything measured so far, and it is why the answer to it is not a measurement of the results but a property of the store that produced them.