Post-filtering leaks into traces, logs and caches, not the response

Retrieval-augmented generation, almost always shortened to RAG, answers a question by first searching a body of documents for the passages most likely to be relevant, then pasting those passages into a language model's prompt, so that the model writes from text it has just been handed rather than from whatever it absorbed during training. To make that search fast, the documents are cut at ingest time into chunks — passages of a few hundred words — and each chunk is stored next to an embedding, a list of numbers placing its meaning in a space where similar text sits nearby. Asking for the top 50 means asking for the fifty chunks whose embeddings sit closest to the query's. The pattern exists because a model cannot be retrained every time a contract is signed or a policy is updated, and because an answer that cites a passage can be checked against it, so essentially every product that answers questions about a company's own documents is built this way.

The complication this topic exists for is that a company's documents do not have a single readership. Documents belong to several hundred customer organisations — tenants, in the vocabulary of software that serves many companies from one deployment — and inside each tenant, a case note is meant for a handful of named people rather than for the whole org. A vector index has no notion of who is asking; it returns whatever sits nearest the query. Permission-aware RAG is retrieval that carries the caller's identity down into the search itself, so that the passages reaching the model are only ones that particular caller was already entitled to read. You meet it wherever an assistant sits over a body of documents with mixed readership: internal document search across a company's drives, a support assistant reading one account's tickets, an HR assistant reading handbooks and case notes.

Where a discarded chunk actually goes#

The design that gets signed off looks like this. A query comes in, the service embeds it, asks a shared index for the 50 nearest chunks across all 1.8 million, and then filters the results in Python against the caller's tenant and group list before assembling the prompt. No unauthorized text ever reaches a user's screen, and a test can assert exactly that. What is easy not to trace is the path the other 48 chunks take through the process on their way to being discarded.

@tracer.start_as_current_span("retrieve")
def retrieve(query: str, principal: Principal) -> list[Chunk]:
    q_vec = embed(query)
    candidates = index.query(vector=q_vec, top_k=50)

    span = trace.get_current_span()
    span.set_attribute(
        "retrieval.candidates",
        json.dumps([
            {"id": c.id, "score": c.score, "preview": c.text[:240]}
            for c in candidates
        ]),
    )

    allowed = [c for c in candidates if authorized(principal, c)]
    log.info("retrieval", extra={"candidate_ids": [c.id for c in candidates],
                                "kept": len(allowed)})
    return allowed

Ten lines of instrumentation added during a latency investigation, kept because it was useful, and every one of them writes the full candidate set — not the filtered one — to a system with different retention and a much wider audience than the response. The preview field alone puts 240 characters of 48 unauthorized chunks into a searchable index, per request, for as long as that system retains spans.

Read the two statements in that paragraph together, because they are the whole mechanism. candidates is bound before the filter runs, and both the span and the log line close over candidates rather than over allowed. Nothing about that is a mistake in the instrumentation: it was written to answer "why is this query slow", a question about the candidate set, and it answers it correctly. The filter's position in the function is what makes the instrumentation a disclosure, and the instrumentation was not written by the person who chose that position.

The mental model that permits this design is that authorization is about what the user sees. Under that model, filtering before prompt assembly and filtering at the index are equivalent, because the bytes that leave the service are identical either way. The model is wrong about where the data goes, because a chunk that has been returned to your application process is not confined to the response path. It is a Python object that gets serialized, logged, cached, forwarded to a reranker — a second, slower model that re-scores the candidates for relevance — sampled into an evaluation set, and attached to an error report, and each of those destinations has its own retention, its own access control and its own presence or absence from your data processing agreement.

That is the stumble in its strongest form, and it is worth naming as a property rather than as a caution. The response body is a quantity that stays identical under both designs while the dangerous property — how many tenants' document text entered your process — moves from one to fifty. A review that inspects the response is measuring the constant. It is a tempting review to run, because the response is the thing the customer receives, it is easy to inspect, and a test over it is easy to write and passes honestly. It is the wrong measurement because it is insensitive to the variable in question by construction.

Read that table as gaps rather than as a list of surfaces. Between the first row and every other row, the item count moves from two to fifty, the retention moves from the length of a request to months, the readership moves from one caller to every engineer or an outside company, and the tenant-scoping answer flips from yes to no. Four properties move together in the same direction, and the row that looks safest on each of them is the only row the sign-off examined.

The cache row is the one that turns a retention problem into a live cross-tenant read. A semantic cache keyed on the normalized query, or on a hash of the query embedding, is a standard latency optimization, and if it caches the candidate set rather than the final response, then a tenant whose query hashes to an existing key receives another tenant's candidates directly. The post-filter still runs afterwards and still drops them, so the user sees nothing wrong, and the cache has now become a mechanism for moving one tenant's document text into another tenant's request context on a routine basis.

Every arrow that leaves the service carrying 50 items is a place the design's guarantee does not reach. The single arrow back to the caller is the only one the sign-off actually examined. Note where the drop sits in that sequence: it is the second-to-last step, after four arrows have already left the process, which is the diagram's way of saying that the filter's correctness is not in question and its position is.

Two names, and what each buys#

The two designs have names worth using precisely, because the interview version of this question turns on them. Pre-filtering pushes the caller's resolved grants — the set of records saying which resources this particular caller may read — into the ANN query itself as a predicate on the index. ANN stands for approximate nearest neighbour, which is the search that finds the closest embeddings without comparing the query against every chunk in the index, and constraining it means the engine's traversal is confined to eligible chunks, so the 48 are never scored, never returned and never present in the process. Post-filtering asks the index for an unconstrained top-k — the k nearest chunks with no predicate attached — and applies the predicate to the result set in application code. Topic 8.12 covered the mechanics of both and the recall cliff that appears when the predicate is selective; the point here is that when the predicate is an authorization decision rather than a facet, the difference between them stops being a relevance question.

Post-filtering loses two distinct things, and it is worth being specific about where each loss shows up rather than gesturing at "it's less safe." The first is the leak surface in the table above, which is an operational and contractual exposure that exists whether or not anyone ever exploits it — a customer asking who can read their case notes gets a materially different answer under each design. The second is the truncation, and this one shows up as a product defect rather than a security one.

The truncation follows from one multiplication and is worth doing rather than asserting, because it tells you how many results a post-filter yields for any entitlement share. An unconstrained top-k returns the k nearest chunks drawn from the whole index, and the caller is entitled to some fraction of that index. If the caller's entitled chunks are no more likely to be near the query than anybody else's, the expected number of survivors is k times that fraction. In one HR-tech corpus a typical caller was entitled to about 4% of the chunks, so a blind top-50 was expected to yield around two survivors, and the generator was routinely handed two passages and asked a question that needed six. Users reported it as "the assistant doesn't know about our leave policy," which was triaged as a retrieval quality problem and consumed two sprints of embedding experiments before anyone connected it to the filter.

The assumption inside that multiplication is worth stating, because it is the only thing standing between the arithmetic and the observation. It assumes the caller's entitled chunks are distributed through the ranking like everyone else's. In practice they are usually better than that, since a caller tends to ask about their own material, which is why the survivors are two rather than zero. It is also why the failure is intermittent and reads as a quality problem: the yield depends on the query, and a query squarely about the caller's own documents survives while a general one does not.

Pre-filtering is not free either, and pretending otherwise makes the argument easy to dismiss. Pushing a grant set into the ANN query costs latency in proportion to how rare the eligible documents are, it requires the grant set to be small enough to express as a predicate, and it forces the index to carry an authorization-relevant attribute that now has to stay correct. What it buys is that the failure mode changes shape: an over-restrictive predicate returns too few results, which is visible, whereas a post-filter that is bypassed by a logging call site returns the right answer to the user while quietly failing everywhere else.

The latency term is worth deriving rather than accepting, because it explains why the cost rises exactly where the entitlement share falls. An approximate search walks a graph of stored vectors, moving from a node to its neighbours in the direction of the query and stopping once it has collected enough results. A constrained walk still has to collect the same number of eligible results, so the rarer eligibility is, the more nodes it must visit before it has them. The work therefore scales roughly with the inverse of the eligible fraction, which is the same fraction that decided how many survivors a post-filter yielded. The two designs are paying for the same scarcity, and they differ in whether the payment is latency inside the engine or a truncated answer outside it.

That symmetry is why "which design is cheaper" is not the interesting question. Both are governed by the same quantity, and neither can conjure eligible documents that are not near the query. What differs is where the shortfall lands: pre-filtering spends more time and returns a full set of permitted results, post-filtering returns quickly with a set that has been thinned by an amount nobody in the calling code can see.

There is a third cost that shows up on the invoice, and it closes against the entitlement share by a completely separate route. The service reranked all 50 candidates with a hosted cross-encoder before filtering, at roughly 110 ms for the median request and $0.0011 per query. At 40,000 queries a day that is about $44 a day, or $16,000 a year, of which 96% was spent scoring text that was going to be thrown away, plus the latency of doing it. That 96% is the discarded share of a fifty-item candidate set, and it is also one minus the entitlement share that produced the two survivors above — two instruments, one derived from a count of chunks and one from a fraction of an index, landing on the same split. Reranking after the filter would have been cheaper and faster, though it would still have sent the unauthorized text to the vendor, so ordering the pipeline correctly mitigates the waste without touching the exposure.

Where post-filtering is still the right call#

None of this makes post-filtering a defect in general. When the predicate is not an authorization decision — deduplicating near-identical chunks, capping how many passages come from a single document, dropping anything older than the fiscal year — applying it in application code is normal and correct, because nothing in the discarded set is data the caller was forbidden to hold. The distinction is not where the filter runs; it is whether the discarded items were ever legitimately yours to receive.

There are also systems where post-filtering is the only option available, usually because the authorization data lives somewhere the index cannot see. A permission model with deep group nesting, resource-level shares and time-bounded delegations may resolve to a grant set too large or too dynamic to express as an index predicate, and some engines cap the size of an IN list at a few hundred values. In those cases the honest position is that you are running a compensating control, and the compensations have to be real: scrub the candidate set from spans and logs before it is written, key any cache on the principal as well as the query, and rerank after the filter rather than before. That is a maintained invariant across every logging call site in the retrieval path, which is a weaker guarantee than one the index enforces, and it should be recorded as such rather than described as equivalent.

The word carrying the weight there is maintained. An index predicate holds because the engine cannot return what it did not traverse, which is a property of one component. A scrubbing rule holds because every call site that touches the candidate set today obeys it and every call site added later also obeys it, which is a property of a codebase over time and of everyone who will work on it. Those are different kinds of guarantee, and the difference is exactly the ten lines of instrumentation above, which were added by someone chasing latency and were correct for their purpose.

It is also worth separating this failure from the one it gets confused with, which is the model reciting something it should not. Nothing here involves the generator seeing unauthorized text, because the filter runs before prompt assembly and does its job. The leak is entirely below the model, in the plumbing, which is why prompt-level defenses of the kind covered in topic 7.4 do not touch it and why a red-team exercise aimed at the assistant's outputs will come back clean.

The counterargument an engineer defending post-filtering will reach for is that pre-filtering is theatre, because their vector store evaluates metadata predicates after the graph traversal internally, so the unauthorized vectors were visited inside the engine regardless. That claim is sometimes true and it changes the analysis in a specific, bounded way rather than dissolving the distinction. Visiting inside an engine's own process is not the same event as returning to yours: the surfaces in the table above are all yours, and none of them exists inside the engine.

Which requires knowing what your engine actually does with a predicate, and, more urgently, knowing that the predicate was built from an identity the caller could not choose. A filter enforced perfectly against a tenant value the caller supplied is worth nothing. Everything in this chapter assumes the predicate is true; the next question is where its inputs came from.