A reranker cannot fix an index that holds last year's policy

A document that has been replaced does not stop being about its subject. Last year's expenses policy still names the cities, still states figures, still uses the vocabulary somebody searching for it would type, and nothing in its text announces that a newer version exists. Every ranking signal a retrieval system computes is a function of that text and of the query, so every one of them keeps scoring the old document exactly as highly as it did on the day it was current.

That is the failure this chapter is about, and it is worth separating from the one it resembles. Reranking is a second scoring pass: a cheap retriever returns a candidate set, and a cross-encoder — a model that reads the query and one candidate together and predicts how well the candidate satisfies the query — reorders that set before the top few are handed to a generator. A reranker is the standard answer when the right document is present and losing, and it is the first thing an incident channel reaches for. Presence and currency are different problems, and only one of them is a ranking problem at all.

The per-diem answer and the four candidates#

A finance manager asks the assistant what the per-diem allowance is for travel to Zurich and gets back €60, with a citation. She raises it because the figure has been €85 since the first of January, and the citation points at spaces/team-ops/expenses-policy.md, a copy somebody made when they were drafting a team-specific addendum in 2023 and never deleted. The current policy lives at finance/policies/expenses.md, was indexed correctly, and lost.

The proposal in the incident channel is a reranker, on the reasoning that the right document exists in the index and is simply not winning. That reasoning is sound for the failures in chapter 2 and it does not apply here. Run the cross-encoder over the candidates for this query and it ranks the stale copy first.

query: "what is the per diem allowance for travel to Zurich?"

  cross-encoder      cosine   document
      0.941           0.812   spaces/team-ops/expenses-policy.md#3   (2023, superseded)
      0.913           0.799   finance/policies/expenses.md#5         (current)
      0.887           0.804   spaces/emea/travel-notes.md#1          (2022, superseded)
      0.402           0.771   minutes/finance-2024-11-08.md#2

The two score columns are worth reading against each other rather than separately, because the difference between them is the reranker's entire contribution. Sort by cosine and the order is the stale copy, then the 2022 travel notes, then the current policy, then the meeting minutes. Sort by the cross-encoder and the current policy has moved up past the travel notes, and the minutes have collapsed from within four hundredths of the current policy to well under half its score. Both of those movements are corrections. The reranker found that a 2022 note about EMEA travel is a worse answer than the policy itself, and that a set of meeting minutes is barely an answer at all, and it was right on both counts.

What it did not move is the top row. That is the shape of the whole chapter: the instrument worked, and the thing it measures is not the thing that decided the outcome. It closes off the most natural response, which is that a better reranker would have caught this. There is no evidence here of a reranker performing badly. It made two corrections and both of them were right, and the comparison that decided the answer is one its objective has nothing to say about.

The stale copy scores highest because it is the best answer to the question as asked. It names Zurich explicitly, states a figure, and spends two paragraphs on Swiss travel, whereas the current policy handles Switzerland in a table row inside a longer section about EMEA rates. A cross-encoder is trained to predict whether a document satisfies a query, and by that standard the 2023 copy satisfies it beautifully. Nothing in the text says the number is no longer in force.

Relevance is one axis and correctness is another#

The general statement is that a reranker reorders candidates along the single axis it was trained on, and currency, authority, canonicity and correctness are not that axis. Whatever ordering you impose on a candidate set, if every candidate in it is wrong, the top one is wrong. This is the same reason a reranker does not deduplicate near-identical documents, does not prefer the official copy over someone's fork, and does not know that a page marked "draft — do not implement" in a footer should never be quoted.

Set that out as an algebraic statement, because the algebra is short and it settles the question rather than illustrating it. A reranker produces a score for each candidate that depends on two things: the query, and the words of that candidate. Currency is a third thing — a fact about which document is in force today — and it appears nowhere in that pair of arguments. Reordering by the score therefore cannot be sensitive to currency, and the sense in which it cannot is stronger than "it weights currency too low": currency is not a term in the expression at all, so it cancels out of every comparison the reranker makes. Replacing the reranker with a better one changes the score function and leaves that cancellation untouched. Any amount of effort spent on the ranker moves the relevance ordering and leaves the currency ordering exactly where it was.

The belief that makes this hard to see is a reasonable one, and it is a belief about a defence. Retrieval quality is measured, the measurements are green, and green measurements feel like evidence that retrieval is not the problem. Ask what those measurements are computed against. Recall, mean reciprocal rank and their relatives are all scored against relevance judgements over the contents of the index, and a human asked whether last year's expenses policy is relevant to a question about per-diem allowances will correctly say yes. So the metric stays flat while the corpus ages, and it stays flat by construction — staleness is invisible from inside the index, because the index is the thing being aged and the metric only ever looks inward. The quantity you are watching is constant while the quantity that ruins the answer varies without bound.

The listing shows the same trap from the other side. Judged by rank, the current policy is doing well: it is second out of four, and it was third before the reranker touched it. A team looking at rank alone would read that as a system narrowly missing, which is the profile of a problem worth tuning. It is second because the document above it is better at the question, and no amount of tuning changes which document is better at the question.

Generation cannot arbitrate this either, and it is worth seeing why before someone proposes a prompt instruction. When both documents land in the context, the model has two passages asserting different figures and no basis for preferring one, since neither says when it took effect. Asking it in the system prompt to "prefer the most recent source" does nothing unless the recency is in the text it can read, and modification dates usually live in metadata that never reaches the prompt. Putting the effective date into each chunk's header before generation does help, and it helps because it is the same metadata fix arriving one stage later, with the model doing the filtering less reliably and at the cost of tokens.

A second proposal that surfaces at this point is recency boosting: multiply the similarity score by a decay term on document age and let fresh documents float up. It is blunt in both directions. A policy last revised in 2021 because it has not needed revision is penalised for being stable, and a superseded copy that somebody touched last month to fix a typo is rewarded for being edited. Decay also has a tuning constant nobody can justify, and the constant that fixes the per-diem query will demote a correct answer somewhere else in the corpus. Where a document's status is knowable, filtering on it is both cheaper and easier to explain to the person whose answer disappeared.

It is worth being precise about what a reranker does fix, because the two failures look identical from the outside. Both produce a bad answer with a confident citation. The difference is visible one layer down: in chapter 2 the correct chunk was in the candidate set at rank 14 and lost to near-misses, and reordering promoted it. Here the correct chunk was at rank 2 and lost to a document that is genuinely better at answering the question that was asked. The manifest tells you which case you are in within a minute, and the proposal that follows should depend on that rather than on which fix the team already knows how to build.

Those two numbers invert the intuition, and the inversion is the diagnostic. Rank 14 sounds worse than rank 2, so the instinct is that the rank-14 case is the harder one. It is the easy one, because rank 14 means the correct chunk lost to documents that are not really better and a sharper judge can see the difference. Rank 2 means the correct chunk lost to a document that is really better, and sharpening the judge widens the loss. What the rank tells you is not how far from winning the right document is. It tells you what beat it.

Ingest scope is a decision someone has to make#

"Embed everything" arrives as a default because a connector pointed at a shared drive has no opinion about which parts of the drive are documentation. The fixture corpus reflects a real one: 312 documents, of which about ninety are meeting invites, retrospectives, desk assignments and a folder of migration memes, and six are older generations of policies that also exist in current form. Running the forty-query fixture and counting the sources of every chunk that entered a context gives a number worth putting on the dashboard.

Both columns sum to a hundred with nothing left over, which is worth noticing before reading any single row, because it is what makes the table an argument rather than a set of observations. These are shares of one pool — the chunks that got into a context — and that pool has a fixed size set by k. A share taken by one class is a share taken from another. Nothing in the table can improve except at some other row's expense.

Read the two columns as a difference and the same point arrives as a number. Current documentation gains thirty points. Superseded versions give up nine, meeting notes give up twelve, and non-documentation gives up nine, and those three come to exactly the thirty that documentation gained. Not approximately: exactly, with no residual, because the filter created no new documentation and improved no ranking. It removed competitors, and every point of the gain is a slot that a competitor vacated. That identity is what licenses the claim in the paragraph below, which would otherwise be an assertion about how retrieval feels.

Nine per cent of retrieved chunks came from documents that had been formally replaced, and thirty per cent came from material nobody would have chosen to put in front of a model. Those two figures partition the same column: the thirty is the meeting notes and the non-documentation added together, the nine is the superseded row, and the three of them plus current documentation account for the whole pool. The cost of embedding those extra documents is negligible — 90 documents at roughly 3,500 tokens each is about 315,000 tokens, which is under five cents at commodity embedding prices — so cost is not the argument for excluding them. The argument is that each one is a competitor for a top-k slot, and on any query whose topic they graze, they take slots from documentation. Precision loss is the price of recall on junk.

The two unwanted classes behave differently, and the difference decides which lever works on which. About ninety documents in a corpus of 312 are junk, which is close to thirty in every hundred, and junk supplies thirty per cent of retrieved chunks. Those two figures come from instruments that share no step — one counts documents sitting on a disk, the other counts chunks that entered a prompt across forty queries — and they agree. Junk is being retrieved at roughly its base rate. The retriever is not being fooled by it; it is occupying slots in proportion to how much of it was indexed, which means the lever that moves it is the denominator, and the denominator is ingest scope.

The superseded documents do not behave that way at all. Six documents out of 312 is under two in every hundred, and they supply nine per cent of retrieved chunks — something like five times their share of the corpus. They are over-retrieved, and the listing at the top of this chapter says why: a superseded policy is not junk that happens to be nearby, it is an excellent answer to the question. Ranking is doing exactly what it should and the result is worse than chance would have given. That is the one class where a scope decision is not enough on its own, and it is the class the status metadata below exists for.

Both of those readings carry the same caveat, and it should be said rather than buried. Documents do not contribute chunks in proportion to their number, they contribute in proportion to their length, and a meeting invite is short while a policy is long. The agreement at thirty per cent is therefore closer than the underlying quantities justify, and the multiple for the superseded copies is a rough one. What survives the caveat is the direction and the order of magnitude: junk near its base rate, superseded well above it.

Three levers act on this, and they are not interchangeable.

Scope decides what enters the index at all, expressed as an allowlist of spaces, paths or owners rather than a denylist of file types, because a denylist has to be updated every time someone invents a new way to put a spreadsheet in a wiki. Status metadata decides what a document claims about itself: whether it is a draft, in force, superseded or archived, and between which dates. Query-time filtering decides what a particular request is allowed to see, which is where the difference between "what is the per-diem now" and "what was the per-diem in March 2023" is expressed.

One more distinction is worth drawing before the mechanism, because it decides what is negotiable. Finance having to answer as-of questions is a requirement imposed from outside the system. Keeping old versions retrievable rather than deleting them is the design decision that satisfies it. Hiding them by default, rather than showing them and hoping the ranker sorts it out, is a second design decision, and it is the one the whole rest of this chapter implements. The first of the three is imposed and the other two are chosen, and the reason to keep them apart is that a requirement can be pointed at while a decision has to be signed by somebody.

The lifecycle is a state machine, so model it as one#

Documents move between states, and the transitions are what the ingest job needs to detect.

The state that does the work is Superseded, and it exists because deleting old policies is not an option for anyone with an audit obligation. Finance genuinely needs to answer what the allowance was in March 2023 when reviewing an old expense claim, so the old document has to remain retrievable and has to be invisible to a question phrased in the present tense. That is a filter, not a deletion, and the distinction is the reason status metadata beats a cleanup script.

Deriving canonical_id is the part that takes judgement. The clean case is a wiki that records "supersedes" links, and it is rare. The workable substitute is a key derived from the document's stable identity — a URL path with version suffixes stripped, or a title normalised and scoped to its owning team — plus the rule that among documents sharing a canonical identity, the one with the latest effective_from is in force and the others are superseded. That rule misclassifies things, and it misclassifies them in a direction you can inspect: run it over the corpus, print every group with more than one member, and have the owning team read a list of forty groups once rather than audit three thousand documents.

def classify(doc: Document, siblings: list[Document]) -> dict:
    """Attach status metadata at ingest. Siblings share a canonical_id."""
    newest = max(siblings, key=lambda d: d.effective_from or d.modified_at)
    return {
        "canonical_id": doc.canonical_id,
        "status": ("draft" if not doc.approved_at
                   else "effective" if doc is newest
                   else "superseded"),
        "effective_from": (doc.effective_from or doc.modified_at).isoformat(),
        "effective_to": (None if doc is newest
                         else (newest.effective_from).isoformat()),
        "owner_team": doc.owner_team,
        "source_class": doc.source_class,     # documentation | minutes | other
    }

Follow the causal chain that function sits in, one link at a time. A document's standing is a fact about the world rather than about its text, so it cannot be read off the page by any model. It can be inferred from relations between documents — which of two versions is later — and those relations are computable at ingest, when every sibling is in hand. Computing them once at ingest and storing the answer turns a judgement into a field. A field can be filtered on, and a filter is exact where a ranking is a preference. Each link is checkable, and the last one is the payoff: the reason this works where reranking does not is that it converts the missing quantity into data before anything has to rank on it.

The filter that consumes it is unremarkable, which is the point — the work was all upstream.

{
  "query_vector": "…",
  "top_k": 50,
  "filter": {
    "must": [
      {"key": "source_class", "match": {"value": "documentation"}},
      {"key": "status",       "match": {"any": ["effective"]}}
    ]
  }
}

An as-of question swaps the status clause for a range condition on effective_from and effective_to, which is why storing both dates is worth the extra column even though almost every query uses neither.

That top_k of 50 also resolves something the two artefacts in this chapter would otherwise disagree about. The cross-encoder crushes the meeting minutes, and yet minutes and non-documentation still account for thirty per cent of the chunks that reach a context. Both can be true because demotion is not exclusion: fifty candidates are fetched, a context holds far fewer, and a chunk the reranker scored low still arrives when there are slots left over. Ranking changes the order in which junk arrives. Only a filter changes whether it arrives.

Where the metadata does not exist#

The metadata has to come from somewhere, and on most corpora it does not exist. Deriving status from filenames, paths and modification dates gets you most of the way and then stalls on the cases that matter: two documents with the same title owned by different teams, a policy whose current version was last modified in 2021 because it has not needed changing, a page that is authoritative for one department and advisory for another. Those cannot be inferred from the bytes, and the fix is that someone owns the corpus and answers the question when the ingest job raises it.

An analogy is useful here as long as its limit comes with it. An index that never retires a document is a filing cabinet in which nothing is thrown away and no folder is stamped, so both the old policy and the new one are in the drawer under the same heading. The place the picture breaks is the place that makes the software version worse: a drawer has a physical order, and someone filing a replacement puts it at the front. An index has no order at all until a query imposes one, and the order a query imposes is by resemblance, which is the axis on which the old copy wins.

That is why this anti-pattern survives longer than the others. Chunk size and k are settings an engineer can change on a Tuesday; ingest scope is a claim about which documents are authoritative, and making that claim requires a person with the standing to make it. Teams that skip the conversation end up with a review checklist that says "define ingest scope", a design document that says "we index the department's Confluence space", and a per-diem answer that is fourteen months out of date.

Two limits are worth stating plainly. Filtering hard on status will drop answers when the metadata is wrong, and metadata derived by heuristic is wrong somewhere, so the filter should be measured with the same fixture as everything else — the number to watch is how many queries lose their gold chunk to the filter. And scope narrowed too far reintroduces the failure it was meant to prevent, because a question whose answer genuinely lives in a meeting note now has nothing to retrieve. Neither of those is an argument against filtering; they are the reason the filter is a change you measure rather than a hygiene task you tick off.

The reason to reach for scope and status rather than for a better ranker is not that ranking is unimportant. It is that ranking operates on a set, and this failure is decided before the set exists. A cross-encoder given the same four candidates tomorrow will make the same three judgements, two of them correct improvements and one of them irrelevant to the outcome, because the quantity that decides the outcome never enters its input. Attaching that quantity at ingest is the only place in the system where it can be attached, since it is the only place where the sibling document is in hand.

None of this touches the case where the right document is in scope, marked effective, correctly chunked, and still does not come back — because the user typed a string that the embedding model has nothing to compare it to.