Per-request identity

A multi-tenant server is a single deployment — one process, one database, one set of outbound credentials — serving several unrelated customer organisations at the same time. Each of those organisations is a tenant, which is the unit the system is obliged to keep apart, and a user belongs to exactly one of them and pays for the assurance that she can never see inside another. The arrangement exists for cost and operations rather than for elegance, because the alternative is a separate copy of the whole stack per customer, and forty copies mean forty deployments to roll out, forty schema migrations that can each fail halfway, forty sets of configuration that drift apart over a year, and forty machines idling overnight. Collapsing that into one deployment is worth a great deal, and what you buy it with is the risk that one wrong variable in one process hands one customer's records to another.

Where a backend engineer meets this is a remote MCP server — Model Context Protocol, the protocol behind the tool-calling servers whose bearer-token validation is the subject of topic 6.12, on authenticating callers to an MCP server — hosted once by the vendor and connected to by every customer's agent. Two customers' agents open sessions against the same URL and reach the same running process, and the tools they invoke read from the same tables through the same connection pool, over the same cache, using the same integration credentials. Nothing in the network topology separates them; the separation is entirely something the code does on each call. The property being defended has a name, tenant isolation: every operation the server performs is attributable to exactly one tenant and can reach nothing belonging to another. It decomposes into two questions the server answers on every request — which tenant is this call for, and with whose authority is the work done — and a leak needs only one of them answered wrongly. This chapter is about the first, and specifically about the difference between deciding who the caller is when the connection opens and deciding it when the call arrives.

The write and the read happen at different times#

Here is the shape of the defect, reduced to the two lines that matter:

@app.on_session_start
async def on_session_start(session, request) -> None:
    claims = await verify_bearer(request.headers["authorization"])
    session.state["user"] = await load_user(claims["sub"])   # written once

@tool("search_invoices")
async def search_invoices(ctx, query: str) -> list[Invoice]:
    user = ctx.session.state["user"]                          # read much later
    return await repo.search(tenant=user.tenant_id, q=query)

The read in the tool handler resolves at call time. The write happened when the HTTP session was established, which may have been eight minutes and four hundred tool calls earlier. Between those two moments the server is relying on an invariant it never checks and cannot see: that every request arriving on this session came from the caller who authenticated it. The handler has no way to notice when that stops being true, because the session object looks the same either way. session.state["user"] does not mean "the user who authorised this call". It means "the user most recently written onto this session", and under a transport that lets more than one caller reach the same session, those are different sentences.

The defence most people reach for at this point is the session id itself. It is unguessable, it is issued by the server, and it arrives on every single request, which makes it feel like a credential and makes binding identity to it feel safe. It is not a credential. A credential is a proof the caller produces; a session id is a label the server issued so that it can find state it stored earlier, and a label tells you which pigeonhole to open rather than who is standing at the counter. The test that separates the two is whether the value changes when the thing it must distinguish changes. Two callers holding two bearer tokens present two different proofs, and a handler reading the token can tell them apart; two callers routed onto one upstream present the same session id, because that id is a property of the pipe rather than of either of them. A defence that stays constant while the thing it has to discriminate varies cannot discriminate, and that sentence is the whole defect stated without reference to any particular gateway.

Three shapes that put a second caller on your session#

None of the three requires a change to your code, and none of them is exotic.

The first is a gateway that multiplexes. Enterprises increasingly put an MCP gateway in front of remote servers so that policy, audit and rate limiting live in one place, and a gateway of that kind normally maintains a small pool of upstream connections rather than one per end user — four or eight upstreams serving three hundred employees. If the transport is Streamable HTTP, the MCP transport in which a client posts every request to one endpoint and receives replies over a long-lived stream identified by a session id, the gateway holds one Mcp-Session-Id per upstream and puts requests from whoever is currently active onto whichever upstream is free. Your server sees one session, correctly authenticated once, carrying traffic from three hundred people.

The second is connection pooling below the gateway. HTTP/2 multiplexes many concurrent streams onto a single TCP connection, and a reverse proxy configured to keep upstream connections alive will reuse the same one across unrelated downstream clients. If any part of your identity resolution is keyed on the connection — a TLS session, a socket, a Server-Sent Events stream that you treat as belonging to one user — the pooling is invisible to you and the sharing is total.

The third is reconnection. A client that drops and reconnects will present the session id it remembers, because resumability is the point of having session ids at all. If that id is still live on your side, the reconnecting client resumes a session whose stored user was resolved from a token that may not be the token the reconnecting client is now presenting. Session ids that are sequential, short, or logged in plaintext turn this from an accident into an attack, but the accident alone is enough to produce a steady trickle of tickets.

The frequency follows directly from that picture. A collision needs two callers active on one upstream inside the same window, which is why the failure rate tracks concurrency and why staging, with one engineer clicking, never shows it.

That relationship is a mechanism rather than a coincidence, and it is worth walking once. A gateway keeps a fixed and deliberately small number of upstream connections, and it gives each arriving request to whichever one is free. Two requests therefore share an upstream in exactly one circumstance: both are in flight during the same interval. As the number of callers active at once rises against a fixed number of upstreams, the share of requests that land on a connection somebody else is already using rises with it, which is why the ticket rate follows load rather than following any particular tool. Run the same reasoning backwards and staging explains itself. One engineer clicking produces one request in flight at a time, no two requests overlap, every request finds its own identity where it left it, and the code is therefore correct on every run. The defect is not absent from staging; there is nothing in staging for it to collide with.

The pattern is tempting because it is inherited, and in the place it came from it was correct. A classic web application authenticates a browser once, writes the user id into a server-side session keyed by a cookie, and reads it on every subsequent page render; the cookie is held by one browser profile belonging to one person, so the connection really did stand for the user. Long-lived socket protocols carried the same habit forward, and a chat server that resolves the account when the WebSocket opens is doing something defensible, because the socket terminates in one client application on one machine. Remote MCP servers look close enough to that to invite the same design, and the resemblance stops at the point where the client is not a browser and not a person but an agent runtime, frequently a shared one, reached through infrastructure whose whole purpose is to consolidate connections.

Validating on the request and passing the result down#

The correction has two halves, and doing only the first leaves most of the defect in place.

@tool("search_invoices")
async def search_invoices(ctx, query: str) -> list[Invoice]:
    identity = await authenticate(ctx.request)   # per call, from this call's headers
    return await repo.search(identity=identity, q=query)


@dataclass(frozen=True)
class Identity:
    tenant_id: str
    user_id: str
    scopes: frozenset[str]
    token_exp: int

Validating on every request is the half people reach for, and it fixes the multiplexed case because the token attached to this HTTP request belongs to the caller who made it. Passing the resulting identity explicitly is the half that keeps it fixed. As long as the handler reaches into ambient storage for the caller — a session dictionary, a thread-local, an asyncio context variable set by middleware — the correctness of every downstream call depends on whether that storage was scoped the way you assumed. Context variables are a real improvement over session state, since they are set per task rather than per connection, and they still lose the identity across a thread pool executor, a background task spawned with create_task and awaited later, or any library that reuses a worker. When identity is a parameter, the function signature states where authority came from, and a call that lacks one fails to compile or fails to run rather than silently picking up somebody else's.

The frozen dataclass matters for the same reason. An Identity that can be mutated is an Identity that a retry handler can rewrite, and a value passed down four call frames is a value four call frames can edit.

A diagnostic benefit arrives with the same change. When the identity is resolved per call, the audit record for a tool invocation can carry the sub, tenant and token jti that were verified for that invocation rather than for the session, so a complaint about a stray record becomes a query — find calls where the tenant in the result set differs from the tenant on the verified token — instead of an inference from timing. Session-scoped identity makes that comparison impossible to write, because there is only ever one recorded identity per session and it agrees with itself by construction.

The claim above that a context variable is an improvement and not a fix deserves one level of mechanism, because a reader who stops at "it is scoped per task" will predict the next case wrongly. A context variable lives in a context object, and a task gets its own copy of that object from whoever created it at the moment it is created. A coroutine awaited inside the request therefore sees the value, and so does a task spawned inside the request — which is why the improvement is real. A thread pool executor does something different in kind: it hands the callable to a worker thread that existed before this request arrived and carries a context of its own, so the lookup inside the worker finds whatever that thread last held, or nothing at all. create_task copies at creation and then diverges, so a task created during one request and awaited after the response has a snapshot of a context whose owner is gone. Each of those is a case where the storage was scoped differently from the way the handler assumed, and the assumption is invisible at the call site — the same property that made the session dictionary dangerous, relocated one layer down.

What the extra validation costs#

The standing objection is that verifying on every call repeats work whose answer cannot have changed, so it is worth putting a number on it rather than arguing about it.

Those verification figures are for signature checking plus claim validation on a modern x86 core, with the signing keys already fetched; the JWKS request itself happens on key rotation, not per call. Against a tool whose median end-to-end latency is 180 ms — a database round trip, a search index, a serialisation pass — asymmetric JWT verification is between 0.05 and 0.2 percent of the request. The measurable cost belongs to opaque tokens verified by remote introspection, where an unconditional network hop can be 7 percent of the request; that is the case where a short-TTL cache keyed on a hash of the token earns its place, TTL being the time to live an entry is granted before it has to be verified again, and that interval is exactly the revocation lag you are choosing to accept.

Both of those percentages are one division, and doing the division in the open is worth more than quoting its answer. Take the slow end of the RS256 row and divide it by the tool's own median: three tenths of a millisecond against a hundred and eighty milliseconds is under two thousandths of the request, which is the two tenths of a percent quoted above. The fast end of the same row, nine hundredths of a millisecond, divided the same way gives the five hundredths of a percent at the other end of the range. The introspection figure is the identical division with a different numerator — twelve milliseconds is a fifteenth of a hundred and eighty, and a fifteenth is the seven percent. Division is the right operation here because latency added to a request and latency already in it are the same quantity in the same units, so the ratio is a share of one whole rather than a comparison of two unrelated things.

Writing the division out also shows what would have to change for the objection to become a real one, which quoting the answer does not. Only two quantities are in it. Either verification gets much slower, which asymmetric signature checking on a modern core does not, or the tool gets much faster. The second happens: a tool that is a pure in-memory lookup rather than a database round trip has a median in single-digit milliseconds, and the same three tenths of a millisecond measured against that is several percent of the request instead of a fifth of one. The argument in this chapter is not that verification is free. It is that verification is small against the specific denominator this chapter is using, and a reader who carries the percentage rather than the division will misapply it the first time the denominator changes.

A middle position exists and is worth naming, because a team under latency pressure will find it anyway: cache the verification result keyed on a hash of the presented token, then every request still reads the token off that request and looks up its own identity, while repeated presentations of the same token skip the signature check. That preserves the property this chapter is about, since two callers presenting two different tokens get two different cache entries and the session is not consulted at all. What it trades is revocation latency, bounded by the TTL you pick, and it only helps when the same token is presented many times, which is the normal case for a chat client and not the case for a gateway minting a fresh downstream token per call.

The cost is real and small, which means it is not what the argument is about. Two things are.

The answer to "is this token still good?" genuinely changes during a session. Access tokens are minted with lifetimes of five to sixty minutes, refresh flows issue new ones with different scope sets when an administrator narrows a role, users get deprovisioned, and a tenant whose subscription lapses gets suspended. A session that authenticated at 09:02 and is still serving calls at 11:40 is serving them on a decision that expired more than two hours ago. Validating once means the first bad request after a revocation succeeds, and so does the ten thousandth.

More importantly, one user per connection is not an assumption the server is entitled to make. It is a property of everything sitting between the caller and your process, none of which you deploy, most of which you will never see, and any of which can start pooling connections in a routine upgrade. A server whose isolation holds only when no intermediary multiplexes is a server whose isolation is a customer's network configuration.

Work that outlives the request, and paths that never ask#

Passing an identity down settles the handler and leaves two gaps that behave differently.

The first is the boundary of the request itself. An Identity parameter travels as far as the call stack does, and work that outlives the response — a task created and awaited later, a retry queued by a shared client, a job handed to a broker — starts again somewhere with no request in scope. Nothing in the parameter discipline reaches those, because there is no frame to pass a parameter into.

The second gap is the one that survives every correction here. Everything above is about a value that was read from the wrong place. It says nothing about the code paths that never read a caller identity at all, because they were built to use a credential the process already holds — the service account that resolves a document title, the connection that warms a cache, the client constructed once at startup. Those paths cannot return the wrong tenant's data as a result of session confusion, since they were never asked whose data to return. They return whatever they are asked for, to whoever asks, and no amount of per-request authentication changes that, because authentication was never consulted.

Identity resolved per request, from that request's own credentials, and passed down as an argument is the whole of the first correction, and its value is that it removes a dependency on infrastructure you do not control. A session is a pipe, and the number of people on a pipe is decided by gateways, proxies and reconnect logic that were configured by somebody else and can be reconfigured without telling you. A request carries its own proof of who sent it, so a handler that reads that proof is correct under every arrangement of the pipes, including the arrangements that do not exist yet.

That is a claim about which caller the handler believes it is serving, and it is narrower than it sounds. A process can serve exactly the right caller on every request and still hand that caller another tenant's data, provided the code path in question never asked who the caller was — which is a different defect, with a name of its own, and it is the subject of the next chapter. A team that reads this chapter as the fix for tenant isolation rather than as the fix for one of its two halves arrives at that chapter's incident with a green conscience.