Authorize on verified token claims, never on a gateway header

Two kinds of value arrive with a request, and telling them apart is the whole of this chapter. A verified claim is a statement about the caller that carries its own proof: a token signed by an identity provider, checked against that provider's published keys, so that altering the statement invalidates the signature. A transport hint is a value that merely travelled alongside the caller — a header a proxy added, a field in the request body, a query parameter. The two look identical by the time application code reads them, because both arrive as strings on a request object.

They differ in what has to remain true for them to mean anything. A verified claim depends on key management and on the checks actually being performed. A transport hint depends on the network topology that put the proxy in front of the service, and on that topology staying that way forever, including the paths added next year by someone who has never read the code that trusts it. Building an authorization filter is the act of turning one of these values into a database predicate, and the choice of which kind to build from decides whether the filter means anything at all.

A request that carries its own answer#

The shape of the failure is four lines of curl. A tester with a legitimate account on one tenant, and a valid session token to go with it, adds one header to an otherwise ordinary request:

POST /v1/assistant/query HTTP/1.1
Host: api.example.com
Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IjIwMjQtMDMifQ...
X-Tenant-Id: 4471
Content-Type: application/json

{"query": "what did we decide about the Q3 headcount freeze"}

{
  "answer": "The board deferred the headcount freeze to Q4 pending...",
  "citations": [
    {"doc_id": "4471/board-minutes-2024-07.pdf", "chunk": 12, "score": 0.83},
    {"doc_id": "4471/board-minutes-2024-07.pdf", "chunk": 13, "score": 0.79}
  ]
}

The retrieval path here was the good one. The tenant predicate was pushed into the ANN query, the index never scored a chunk outside the requested tenant, nothing unauthorized landed in a span, and the top-k came back full because the filter was selective in a way the engine could exploit. Everything chapter 1 recommends was in place and working. The service had simply asked the wrong question — it filtered correctly on a tenant id the caller was free to choose.

The two blocks above are worth reading against each other, because together they show that no part of the machinery misbehaved. The Authorization header carries a real token for a real account. The citations name real documents, scored and ranked normally, from a tenant the token has nothing to do with. Every component did its job: the gateway routed, the index filtered, the reranker ordered, the generator answered from what it was given. The defect is entirely in which value was handed to the filter.

The root cause is worth stating in a form that survives the specific bug, because the specific bug has a dozen variants. The service made an authorization decision from a transport-level hint rather than from a verified claim, which means the value it filtered on was under the caller's influence. Any fix aimed at the filter — tightening the predicate, adding a second predicate on document owner, moving the check into a middleware — leaves that property intact. A stricter filter applied to an attacker-chosen tenant is still an attacker-chosen tenant.

That is the stumble, and it is a belief that a defence works rather than a slip. The belief is that a network boundary authenticates: the service is unreachable except through the gateway, so anything the gateway sets is trustworthy. It is tempting because it was true when written, it is cheap, and it removes a lookup from every downstream service. It is false because the header's value is constant across every legitimate request — it always says the right thing — while the property that decides whether it means anything, namely whether some path exists by which a caller could set it themselves, varies with infrastructure changes that touch neither the header nor the service that trusts it. A quantity that never moves cannot report on a property that does.

What counts as an identity you can build a filter from#

The distinction is between values that carry cryptographic or lookup-backed proof of who the caller is and values that merely travel alongside them. A signed JWT — a JSON Web Token, which is a small bundle of statements about the caller, called claims, signed by the identity provider that issued it — verified against the issuer's JWKS, the set of public keys it publishes for exactly that check, with issuer, audience and expiry checked, is proof. A header set by a proxy is a hint whose integrity depends on every network path into the service being controlled by that proxy, forever, including the paths that will be added next year by someone who has not read this chapter.

The column that carries the teaching is the third, not the fourth. Every row depends on something, and the rows differ in what rather than in how much. The top three depend on facts about a network or on nothing at all, which are facts other teams change without telling you. The next two depend on facts about key material and about a store's own access control, which are facts you administer deliberately. The final column is a consequence of the third and not an independent judgement.

The last two rows also carry a distinction that matters more than it looks. A verified token tells you who the caller is; it does not necessarily tell you what they may read, even when it contains a groups claim. Identity providers mint group claims at login, so a token issued at 09:00 reflects group membership as of 09:00 and remains valid until it expires, which for a typical access token is 15 to 60 minutes.

That gives a floor on revocation that is worth deriving rather than asserting, because it holds regardless of how well the rest of the system is built. If the authorization data is a claim inside the token, then the only way a revocation takes effect is for the token carrying the stale claim to stop being accepted. Tokens stop being accepted when they expire. Therefore the interval between a revocation and its taking effect is at least the remaining lifetime of the already-issued token, and there is nothing the retrieval service can do about it, because the retrieval service is not the thing that would have to change. Resolving grants server-side against the verified subject replaces that term with the grant store's own propagation delay, which is a number you control.

def principal_from_request(req) -> Principal:
    token = bearer_token(req)                      # raises on absent header
    claims = jwt.decode(                           # verifies signature
        token,
        key=jwks.key_for(unverified_kid(token)),   # pinned issuer, no 'none'
        algorithms=["RS256"],
        audience="api.example.com",
        issuer="https://idp.example.com/",
        options={"require": ["exp", "sub", "org"]},
    )
    # Nothing from headers, query string or body participates below this line.
    org_id = claims["org"]
    grants = grant_store.effective_grants(subject=claims["sub"], org=org_id)
    return Principal(subject=claims["sub"], org=org_id, groups=grants.groups)

The comment is the load-bearing part. Once principal_from_request returns, no other request-derived value is allowed to influence the predicate, and a code review that spots req.headers anywhere below it has found a bug regardless of what the header is called. Two of the arguments above are doing less obvious work. Naming the algorithm list explicitly refuses a token that declares its own algorithm as none, which is a signature check that verifies nothing. Requiring exp, sub and org refuses a token that simply omits the claim the filter is about, which would otherwise arrive as a missing key rather than as a rejection.

The enforcement point, and what the engine does with it#

Pushing the predicate down is the enforcement point, and it has two common shapes. In a vector store with metadata filtering, it is a filter clause on the ANN query built from principal.groups. In Postgres with pgvector, it can be a row-level security policy — RLS, a rule attached to the table itself that decides which rows the current database session is allowed to see — which has the useful property that the check is enforced by the database on every query against the table, including queries written by someone who forgot the WHERE clause.

alter table document_chunks enable row level security;

create policy chunks_readable_by_grant on document_chunks
for select using (
  exists (
    select 1
    from grants g
    where g.resource_id = document_chunks.resource_id
      and g.subject_id  = current_setting('app.subject_id')::uuid
      and g.revoked_at is null
  )
);

That policy is worth reading clause by clause, because each one is a decision. for select using attaches the rule to reads, so it constrains every read path rather than one query. The exists subquery makes visibility depend on a row in grants rather than on a column of the chunk, which is what keeps a revocation a single write instead of a rewrite. revoked_at is null is what makes revocation expressible at all without deleting history. And current_setting('app.subject_id') is the one value that comes from outside the database, which is why the next paragraph is about where it comes from.

Supabase's RAG with Permissions walks through the pgvector variant of this in full, including how the policy interacts with the index scan. The behaviour worth internalizing is that the policy is evaluated as part of query planning, so the similarity search runs over the visible rows rather than over everything, and current_setting('app.subject_id') has to be set from the verified claim on the connection or transaction — setting it from a header reproduces the same bug one layer down.

The objection that comes up whenever a team compares this against post-filtering is that pushing the predicate into the index is cosmetic, because the engine evaluates metadata predicates after the graph traversal internally, so the unauthorized vectors were visited anyway. Whether that is true depends on the engine and on the predicate's selectivity, meaning how small a fraction of the index survives it, and it is often partly true: several engines implement filtered search by over-fetching candidates and applying the predicate to them, adaptively switching to an in-traversal filter only when selectivity crosses a threshold. Two things follow. The visiting is inside the engine's process, so the unauthorized text never reaches your application, your spans, your logs, your cache or your reranker vendor, and the leak surface from chapter 1 does not open. What does remain is the engine's own observability — slow query logs, query plans and any candidate-set debug output — which lives inside a component whose access control is usually much tighter than your tracing backend, but which is worth checking rather than assuming.

The claim also fails on a second axis, which is that in-engine post-filtering returns fewer results and the engine knows it, so it over-fetches to compensate and reports what it did. Application-side post-filtering silently returns two results and reports nothing. The two are not the same operation performed at different times. An engine that over-fetches has the information needed to correct for the shortfall, because it holds both the requested count and the surviving count in the same place; application code that filters afterwards has already lost the ability to ask for more without issuing a second query.

Two identifiers in one request, and only one of them verified#

Tenant confusion does not require forging anything. It happens whenever a request carries two identifiers — one proving who the caller is, one naming what they want — and only the first is verified. The citation-fetch endpoint that sits behind every RAG answer is the usual site: the assistant returns doc_id: "8802/policy.pdf", the frontend calls GET /v1/documents/8802%2Fpolicy.pdf to render the source panel, and the handler loads the object from storage using the path in the URL. A caller with a perfectly valid tenant-8802 token who substitutes 4471/board-minutes-2024-07.pdf gets the file, because the token was checked for authenticity and never checked against the resource.

The general shape is worth extracting, because it is what makes the variant survive a correct token. Authentication answers a question about the caller. Authorization answers a question about a caller and a resource together. A check that only consults the first identifier has answered the first question twice and the second question never, and it will pass every authentication test anyone writes.

The same shape appears in agent and background paths, where a service holds a broad credential and acts on a user's behalf. An ingestion worker with write access to every tenant's index, taking a tenant id from a queue message, is one forged message away from writing tenant A's documents into tenant B's namespace — and a poisoned chunk placed in the wrong namespace is a leak in the opposite direction, since it will be retrieved for every future query in that tenant. The rule that covers both is that the resource identifier is an input to the authorization check and never a substitute for it: resolve the resource, read its owning tenant from the record, and compare that to the verified claim rather than trusting the two to agree.

A service mesh — the layer that puts a proxy alongside every pod and authenticates traffic between them — does not fix this, and the belief that it does is common enough to be worth naming. Mutual TLS between pods establishes which workload is calling, which is genuinely useful for deciding whether the retrieval service may talk to the grant store, and it says nothing about which end user the call is on behalf of. Treating "the request came from an authenticated service inside the mesh" as grounds to trust the tenant field in its payload reproduces the gateway-header bug with better cryptography.

Testing the thing that actually broke#

The pen-test finding is reproducible as a unit test, and the class of finding is reproducible as a small suite. The cases worth having are the ones where a valid credential is combined with an invalid assertion, because those are the ones an authentication test will pass.

@pytest.mark.parametrize("mutation", [
    lambda r: r.with_header("X-Tenant-Id", OTHER_TENANT),
    lambda r: r.with_header("x-tenant-id", OTHER_TENANT),   # case variant
    lambda r: r.with_body_field("tenant_id", OTHER_TENANT),
    lambda r: r.with_body_field("filter", {"tenant": OTHER_TENANT}),
    lambda r: r.with_token(sign_with_wrong_key(claims_for(OTHER_TENANT))),
    lambda r: r.with_token(unsigned_jwt(claims_for(OTHER_TENANT))),
    lambda r: r.with_token(expired(claims_for(SAME_TENANT))),
    lambda r: r.with_token(drop_claim(claims_for(SAME_TENANT), "org")),
])
def test_no_other_tenant_content(mutation):
    resp = mutation(authenticated_request(TENANT_8802, "headcount freeze")).send()
    assert resp.status in (200, 401, 403)
    assert OTHER_TENANT not in resp.text
    assert not any(c["doc_id"].startswith(f"{OTHER_TENANT}/") for c in resp.citations)
    assert not any(OTHER_TENANT in s for s in captured_spans())

The mutations divide into two halves that fail for different reasons, and reading them that way is what makes the list extensible rather than a recipe. The first four supply an unverified assertion alongside a genuine token, so they test whether anything below principal_from_request still reads request data. The last four damage the token itself, so they test whether the verification is real: a wrong signing key, a missing signature, an expired token, and a token whose required claim is simply absent.

The last two assertions are the ones that would have caught the original finding, since the first would also pass against a service that returned tenant 4471's text without ever printing the string 4471 in the body. Asserting on citations and on captured spans rather than on the answer prose is a pattern this topic returns to. A generator that summarises a board minute without naming its owner produces prose containing the leaked content and none of the leaked identifier, so an assertion over prose is checking a string the leak has no reason to contain.

The case that is not covered by any of these mutations is the one where the token is genuine, the claims are verified, the grants are resolved correctly, and the grant store cannot be reached. Every line of principal_from_request above assumes grant_store.effective_grants returns. When it times out under load, the service has to decide what a caller may read without knowing, and a failed lookup is not a small permission set — it is no information at all.