Credentials as arguments

The previous chapter named the defect: a credential the code can use without anyone having passed it in, so that nothing at the call site records that authority was exercised. The correction is to make credentials arguments rather than state, and the reason to spend a whole chapter on it is that the naive version relocates the problem instead of removing it. A per-tenant client is only as narrow as the thing that produced it, and a process that will produce any tenant's client for any caller has kept every property that made the startup singleton dangerous while acquiring a function signature that looks careful.

The other half of the chapter is the part that has no request in it at all. Tool handlers enqueue work, and a worker that picks that work up later has no HTTP request, no bearer token, no session and no context variable set by any middleware. Whatever it knows about who caused the work, it knows because someone wrote it down, and the interval between the writing and the reading is where the assumptions rot.

A credential the process holds is a credential every path inherits#

A function that needs to call a downstream API receives a client that can only act for one tenant, and there is no way to obtain a broader one from inside a request:

@dataclass(frozen=True)
class TenantHelpdesk:
    _http: httpx.AsyncClient
    tenant_id: str

    async def search(self, q: str) -> list[Ticket]:
        # organisation is pinned by the credential, not by a parameter
        r = await self._http.get("/api/v2/search", params={"query": q})
        return [Ticket(**t) for t in r.json()["results"]]


async def search_tickets(identity: Identity, helpdesk: TenantHelpdesk, q: str):
    assert helpdesk.tenant_id == identity.tenant_id
    return await helpdesk.search(q)

Two things carry the weight here. The credential inside _http is a per-tenant one — the OAuth grant that tenant made when they connected their helpdesk, not a vendor install key — so the vendor enforces the boundary on its side and a query of organization:* returns that tenant's organisation. And the object is constructed outside the handler and passed in, so nothing in the call graph below search_tickets can widen it. A path that wants a different tenant's tickets has to be handed a different object by someone who had the authority to produce it.

Post-filtering — running the broad query and dropping the rows you were not entitled to — is a weaker construction for a reason worth being precise about. When the boundary is enforced by the vendor against the credential you presented, the rows you were not entitled to never enter your process, so there is nothing to leak through a count, an aggregate, a rate-limit header, a timing difference or a stack trace. When the boundary is enforced by your code after the fact, every one of those channels is open by default and each has to be closed individually, which is a list nobody finishes. The same argument applies internally: a query filtered in the database returns two rows, while a query filtered in Python fetched two hundred and then decided.

The step that usually goes unsaid is why enforcement at the vendor and enforcement in your code are not two implementations of one rule. A filter you apply is a decision about what to return. A credential the vendor enforces is a decision about what exists. When the vendor decides, the response arriving at your process was constructed as though the other tenants' rows were not there, so every property of that response — its length, its counts, its pagination cursor, the time it took to compute, the share of the rate-limit budget it consumed — is a property of your tenant's data alone. When you decide, the response was constructed over everybody's data and you have removed a list from it, so all of those properties still describe the whole. That is the reason the list of side channels is open-ended in the second case and empty in the first: you cannot close a channel in a number you did not compute.

This is the same discipline as dependency injection, arriving for a different reason. Injection is usually argued for on testability: you pass the collaborator so the test can pass a fake. Here the argument is that a collaborator holding authority should appear in the signature so that the authority appears in the signature, and the testability follows as a side effect. The mechanical consequence is the same in both cases, which is that you can enumerate the authority a function has by reading its parameters.

The store is where the ambient authority moves to#

Those per-tenant clients have to come from somewhere. Sixty tenants have sixty OAuth grants, and the server keeps them in a credential store — a table of encrypted refresh tokens, a secret manager path per tenant, a vault namespace. The read from that store at request time is the moment the whole design can quietly revert, because a store that any code path can ask for any tenant's credential is exactly the startup client of the previous chapter with a lookup in front of it.

The difference between the two signatures below is the entire design:

# Ambient, wearing a lookup as a disguise
async def get_helpdesk(tenant_id: str) -> TenantHelpdesk: ...

# Scoped: you cannot ask for a tenant, you can only present an identity
async def get_helpdesk(identity: Identity) -> TenantHelpdesk: ...

The first takes a string that any caller can construct. Anywhere in the process, a loop over tenant ids yields sixty clients, and the loop looks like ordinary code. The second takes a value that only the authentication path produces, and — assuming Identity is not constructible from user input, which is worth enforcing with a private constructor or a module boundary — the set of credentials reachable from a request is exactly one. The store then does the mapping from identity to credential internally, and the mapping is the only place where a tenant id is used as a lookup key.

The temptation to read those two signatures as equivalent is strong, and what makes it wrong is the same property that governed the module-level client in the previous chapter, arriving in a different costume. A parameter of type str places no constraint on what a caller may put in it, so the set of credentials the function will produce is the set of tenants that exist, and that set is the same whoever is calling. It does not vary with the caller, so it cannot discriminate between callers. The identity version narrows the set by construction, because the only way to obtain an Identity is to have come through authentication and authentication produces exactly one. Notice what the narrowing does not require: no check inside get_helpdesk counts the caller's tenants, no policy engine is consulted, and there is no rule anyone has to keep in sync with anything. The constraint is in what the type is able to hold, and a signature that cannot express the wrong request needs no discipline to keep enforcing it.

Enforcing that at the store rather than at every call site is what makes it hold. A convention that all callers pass the right tenant id is a convention that decays; a signature that cannot express the wrong thing does not. The audit question also becomes answerable: every credential issuance carries the identity it was issued against, so "which credentials did this request obtain" is a query rather than an archaeology exercise.

Reading the store on every request costs something, and the numbers vary enough that it is worth measuring rather than assuming. A decrypt through a KMS — a key management service, which keeps the key itself and performs the operation on your behalf so the key never enters your process — is typically 8–25 ms for a stored refresh token; a secret manager GetSecretValue is 20–60 ms; an OAuth refresh against the vendor to exchange that for a fresh access token is 120–400 ms. Doing all three per call would double the latency of a tool whose useful work is 180 ms. The usual resolution is a per-process cache keyed on (tenant_id, user_id) holding the access token until it expires, which is the shape almost everyone lands on, and it carries a cost that should be stated plainly: the process now holds decrypted credentials for every tenant that has been active in the last hour. Chapter 6 of this topic, on what a single process compromise yields, is about what that means; the cache is not wrong, it is a position on a spectrum, and the spectrum has an end.

Those three intervals add rather than take a maximum, because the stages run in sequence: you cannot exchange a refresh token you have not decrypted, and you cannot decrypt a secret you have not fetched. Add the fastest figure from each of the three and the total lands near the tool's own hundred and eighty milliseconds, which is where "would double" comes from — and that is the optimistic end of the estimate. Add the slowest of each instead and the credential path costs several times the tool's own work. The reason to write the addition out rather than quote the doubling is that the addition says which stage to attack, and the three terms are nothing like equal. The OAuth refresh is the largest by a wide margin at both ends of the range, and it is larger than the other two put together at either end, so caching the access token removes most of the cost. Caching the decrypted refresh token would remove very little of it, while holding the more dangerous of the two secrets in memory for longer — which is the whole argument for the shape almost everyone lands on, stated as arithmetic rather than as folklore.

Nothing in the queue has a request#

A publish_document tool writes a row and then needs the search index updated. Doing the reindex inline adds 600–900 ms to a call whose user is waiting, so the handler enqueues a job and returns; a worker picks it up a median of 200 ms later, with none of the request context the handler had.

{
  "job": "reindex_document",
  "document_id": "doc_9f3a2c",
  "tenant_id": "t_northwind",
  "acting_user_id": "usr_4471",
  "authorized_scopes": ["documents:write", "index:write"],
  "enqueued_at": "2026-07-14T09:12:44.118Z",
  "request_id": "req_2b81f0",
  "attempt": 1
}

The tenant and the acting user are recorded at enqueue time from the validated request identity — from the same Identity value the handler was given, never from a tool argument and never from the document row, since the document row is the thing whose ownership is in question. The worker then re-checks that identity's authority over the target before it acts, rather than trusting the payload's own claim that the work is allowed.

That re-check is the part teams skip, and the reason it matters is time. The median delay is 200 ms, which makes the re-check feel like a formality. The distribution is what bites: a queue backed up behind an outage delivers jobs 40 minutes late, a dead-letter replay runs four hours later after someone clicks a button — the dead-letter queue being where jobs go when they have exhausted their attempts, to wait for a human — and a job with an exponential backoff of five attempts, each waiting longer than the last, can land twenty minutes after the user who caused it was deprovisioned. In that interval a document can be moved between tenants, a user's role can be narrowed, a tenant's subscription can lapse. A worker that reindexes because the payload said so is a worker executing a decision made under conditions that no longer exist.

What the payload deliberately does not contain is a credential. Putting the user's access token into the job body puts it in the broker's storage, in its dead-letter queue, in whatever replay tooling an operator uses, and in the logs of anything that inspects message bodies for debugging — and it leaves a token sitting there for hours after the request that minted it. The payload carries identity, which is a claim about who, and the worker converts that into authority through the same store the request path uses. If the stored grant has been revoked in the meantime, the store fails to produce a client and the job fails, which is the correct outcome and one you get for free from routing both paths through the same place.

The rule underneath both halves of that design is that identity is durable and authority is not. A claim about who caused a piece of work stays true: the analyst who pressed the button pressed it, and nothing that happens afterwards makes that false. A claim about what she may do is true only at the instant it was evaluated, because roles narrow, subscriptions lapse and documents move between tenants. Writing the durable claim into a payload that will be read four hours later is therefore safe, and writing the perishable one into it is not, which is the same distinction that decides what a verification cache may hold and for how long. The worker's re-check is nothing more elaborate than evaluating the perishable half at the moment it is needed instead of trusting a copy of an answer given earlier.

Idempotency keys, the strings that let a queue or an API recognise a resubmission of work it has already done and execute it once rather than twice, deserve the same treatment. A key like reindex:doc_9f3a2c is shared across tenants if document ids are ever reused or guessable, and a deduplication check that hits on another tenant's in-flight job will silently drop work or, worse, return that job's result. Prefixing the tenant into the key — reindex:t_northwind:doc_9f3a2c — costs nothing and removes the class.

That fix reads like hygiene, and the mechanism under it is worth one more sentence, because knowing the mechanism is what lets you spot the next instance. A deduplication store answers exactly one question: have I seen this key before? It has no other input. Two pieces of work are therefore the same work precisely when their keys match, which makes the key the store's entire definition of identity. Leave the tenant out of the key and one tenant's job and another's are, as far as the store can tell, literally the same job — at which point the store does what it was built to do, which is run one of them and hand its result back for both. The general form is the one the previous chapter gave for caches — a stored intermediate identified by less than the decision depends on lends one caller's result to the next — with the word "cache" taken out of it.

When the vendor issues one key for the whole install#

Some vendors do not offer per-tenant grants at all, and it is worth saying so rather than pretending the clean design is always available. A product that only issues one install-level API key per integration leaves you holding a credential that spans tenants no matter how you pass it around, and the boundary has to be built somewhere you control. The usual answer is an egress proxy that holds the shared key, accepts requests only with a tenant assertion signed by the server, and rewrites or rejects the query so that the organisation filter cannot be widened — moving the shared credential out of the process that runs tool code and into one that does nothing else. That is more machinery than most teams want, and it is the honest price of a vendor who has not modelled multi-tenancy.

Why that arrangement helps is the store's signature argument applied to a process boundary instead of a function boundary. The shared key is dangerous because any code that can reach it can widen the query, and inside a process that runs tool handlers the set of code that can reach it is not something you can write down: it includes every transitive dependency, every parser, and every string a model produced. Moving the key into a process whose only input is a tenant assertion signed by the server shrinks that set to whatever the proxy itself parses, and a proxy that does nothing else is small enough to read end to end. The boundary has not vanished. It has been moved to somewhere you can enumerate both sides of, which is the only kind of boundary worth claiming.

The distinction that gets conflated in review is isolation against scope. A per-tenant credential with documents:* is correctly isolated and over-privileged at the same time; isolation says whose data the credential can touch, scope says which operations. Fixing one tells you nothing about the other, and a review that reports "credentials are per-tenant" has answered exactly one of the two questions asked of it.

The two are easier to hold apart with all four combinations in view, because only one of them is the state anyone wants. A credential can be broad in whose data it touches and broad in what it may do, which is the install key above. It can be narrow in whose data and broad in what it may do, which is a per-tenant grant carrying documents:* — correctly isolated, over-privileged, and the state most integrations reach first. It can be broad in whose data and narrow in what it may do, which is a read-only install key, still one borrowed request away from every tenant's records. Only the fourth combination is the target. Because the two are separate axes, a fix on one moves nothing at all on the other, which is why a report that names one of them leaves the other entirely open rather than half-answered.

Carrying credentials as arguments, from a store that will only answer to an identity, with the identity written into anything that outlives the request, is a design in which the authority a piece of code holds is visible in its signature and bounded by the caller who invoked it. That is worth a great deal against the ordinary failure, which is a path nobody noticed was privileged.

It does not change what one compromised process can reach, because the store will still produce any tenant's client for any identity that is presented, and code running inside the process can present whatever it likes. The immediate gap, though, is evidence. Everything so far is a design, and a customer's security team will not accept a description of a design — neither should you, because every one of these properties can be true in the code you read and false on the path that runs.