Stored credentials under process compromise

Remote code execution, abbreviated RCE below, is any defect that lets an outsider run instructions of their choosing inside your server. It is the threat that separates the controls in this topic into two groups: the ones that constrain what your code does, which is most of them, and the ones that constrain what any code in the process can do, which is almost none of them.

The distinction matters because of what a multi-tenant server increasingly holds. When each tenant connects their own third-party account — a CRM, a helpdesk, a document store — through OAuth, the server stores the resulting refresh tokens so that it can act for that tenant when a conversation requires it. Those are not rows the process reads under a tenant predicate. They are secrets the process decrypts on demand, and the question this chapter is about is what one RCE in that process is worth, which is a different question from what one bug in a handler is worth.

The functions the server already exposes to itself#

Attacker code inside the process does not need to find a SQL injection, forge a token, or defeat any check, because it can call the functions the server already exposes to itself:

# already imported, already configured, already authorised
for tenant in await store.list_tenants():
    for provider in ("salesforce", "zendesk"):
        creds = await store.decrypt_refresh_token(tenant.id, provider)
        await exfil(creds)

The decrypt_refresh_token call is the same one the request path makes. KMS sees a decrypt request from a principal that is allowed to make decrypt requests, which is the pod's role, which was granted that permission because the server's job is to decrypt these tokens. At an average of 25 ms per decrypt, eighty tokens take about two seconds. The exfiltrated payload is a few kilobytes, which is below the threshold of every egress-volume alert anyone builds, and it can leave through the same outbound HTTPS the server uses to talk to the vendor.

Both figures in that sentence are worth doing rather than accepting, because doing them shows what would have to be different for the answer to change. The count is a multiplication: a tenant that has connected both integrations contributes one refresh token per provider, so forty tenants across two providers is forty pairs, which is eighty tokens. The elapsed time is that count times the cost of one decrypt, because the loop above is sequential and each call waits for the one before it — eighty lots of twenty-five milliseconds is two thousand milliseconds, which is the two seconds. Multiplication is the right operation in both places because nothing here is shared or amortised: decrypting one tenant's token makes the next no cheaper, and adding a provider does not reduce the number of tenants. The one direction the estimate can move is downwards, since nothing obliges attacker code to run the loop sequentially.

What follows is worse than the theft. The attacker now holds credentials that work from anywhere, so the subsequent access does not traverse the compromised server at all. Requests arrive at the third-party system from the attacker's infrastructure, bearing a valid refresh token for a connected app — the OAuth client registration an administrator approved on the tenant's side — and they look like integration traffic because they are integration traffic. Detection has to come from each tenant noticing anomalous API usage in their own account, and rotation has to come from each of them revoking the connected app. The incident is not one company's; it is every tenant's, and the vendor's ability to end it consists of asking each of their security teams to act.

The route in is the ordinary one. Over the last several years the reliable sources of remote code execution in server-side applications have been dependency vulnerabilities in the transitive tree, unsafe deserialisation, template injection in a rendering path, and server-side request forgery, where a URL the attacker supplies is fetched by your server from inside your network, reaching an internal service or an instance metadata endpoint — the fixed local address from which a cloud machine hands out its own credentials to anything running on it. An MCP server adds surface to that list rather than removing any: it parses model-influenced arguments, it frequently shells out or evaluates expressions on behalf of a tool, and it is designed to accept instructions that originated in text it retrieved. Assigning this a low probability is a judgement about your dependency tree, and the probability is not zero for anyone.

What row-level security constrains#

Row-level security deserves a fair hearing, because it is the strongest control on the usual list and it is genuinely valuable for a reason unrelated to the paragraphs above. It moves tenant enforcement out of application code and into the database, so that a repository method someone writes next quarter without a tenant predicate returns nothing instead of everything:

ALTER TABLE documents ENABLE ROW LEVEL SECURITY;

CREATE POLICY tenant_isolation ON documents
  USING (tenant_id = current_setting('app.tenant_id')::uuid);

-- per transaction, from the validated identity
BEGIN;
SET LOCAL app.tenant_id = '5b2c…';
SELECT * FROM documents WHERE title ILIKE $1;
COMMIT;

That is a real improvement over a convention, and it converts an entire category of future mistake into an empty result set. Three deployment details decide whether it holds. The setting must be SET LOCAL inside a transaction, because a plain SET on a connection borrowed from a pooler in transaction mode persists into the next borrower's work and reproduces, at the database layer, precisely the session-versus-request defect from the chapter on per-request identity. The role the application connects as must not own the tables and must not hold BYPASSRLS, since table owners bypass policies by default unless FORCE ROW LEVEL SECURITY is set. And the migration role, the analytics replica, and any reporting tool that connects with a different role are all outside the policy unless someone checked.

The first of those three is the defect from the chapter on per-request identity, re-derived one layer down, and the chain is short enough to check link by link. The policy compares a row's tenant_id against current_setting('app.tenant_id'), so the value of that setting is the enforcement — the policy has no other input. A plain SET attaches the value to the session. A pooler running in transaction mode hands a session back after each transaction and gives it to whoever asks next. Therefore the value one request set is still attached when a different request borrows the same connection, and therefore that request's queries are filtered by the previous request's tenant unless it happens to set its own first. SET LOCAL breaks the chain at the second link, by attaching the value to the transaction rather than to the session, and a transaction ends when the work does. The policy is unchanged; what changed is the lifetime of the thing the policy reads, which is the same correction this topic opened with.

Row-level security constrains what a SQL statement issued on that connection can see. It does not constrain what the process can ask for, and it says nothing at all about refresh tokens. Attacker code that can execute SET LOCAL app.tenant_id to any value it likes is not subject to the policy in any meaningful sense; it is the thing the policy was configured by.

Every row in the first column is worth building, and the last row is worth building for the reason the third column gives: an audit trail written to a destination the process cannot rewrite is the one control here that materially improves the outcome, because eighty decrypt calls in two seconds is an anomaly that a rule can catch if anyone is looking.

Per-tenant keys earn a second mention for a benefit that is easy to overlook while dismissing them. One key per tenant makes every decrypt attributable in the KMS audit trail, so the signal "the principal mcp-server decrypted forty distinct tenant keys within two seconds" exists and is machine-readable, whereas with a single shared key the same event is forty ordinary uses of the one key the service always uses. Under normal traffic the server's decrypt pattern is a slow trickle across whichever tenants are conversing, so an alarm on distinct key ids per minute has a workable threshold — set it around three times the observed peak and it fires on the loop above while staying quiet in production. That is detection with an expected response time of minutes, and calling it prevention in an architecture document is the error this chapter is aimed at.

The word doing the most work in a design like this is "on demand". A server that can act for a tenant only while that tenant's user is present in a conversation has a much smaller requirement than one that must act at any time, and the two get conflated in almost every architecture document that describes such a server, because both are written down as "the server can act on the tenant's behalf". Asynchronous features are what create the stronger requirement: a nightly sync, a webhook handler that reconciles tickets, a scheduled digest. Each of those turns "decrypt while the user is here" into "decrypt whenever", and each is usually added later by someone who does not know they are changing the threat model. Listing which features actually require unattended access is a short exercise that frequently shrinks the set to two or three, and those two or three can be the only integrations whose refresh tokens are stored at all.

Depth in the wrong dimension#

"Defence in depth" means independent layers such that defeating one does not defeat the next. The word carrying the weight in that definition is independent, and it is doing arithmetic rather than rhetoric. Layers whose failures are independent compose by multiplication: if one layer lets an attempt through some fraction of the time, and the next layer is unaffected by whatever defeated the first, then the fraction getting through both is the product of the two fractions — and a product of small fractions is very much smaller than either of them. That product is the entire promise of depth, and it is available only while the independence assumption holds. Layers resting on a shared assumption do not multiply at all. The moment the shared assumption is false, every layer that rested on it is false in the same instant, so the fraction getting through the whole stack is just the fraction that defeats the assumption: no product, and no smaller than a single layer on its own.

The controls above are layered, and they share a single assumption: that the code executing in the process is the code you wrote. When that assumption fails, all of them fail together, which makes them one layer wearing five names. The honest way to describe such an architecture is that it has strong controls against application defects and no control against process compromise, and those are different threats with different frequencies.

There is a useful comparison here. A payment processor could hold every merchant's card data and protect it with excellent application controls, and the industry did not accept that arrangement; the design that won removes the data from the merchant's process entirely and replaces it with a token that is useless elsewhere. The argument was never that merchants write bad code. It was that the value concentrated in one process was worth more than any set of controls on that process could be worth, so the concentration itself had to go.

That comparison is an analogy and it has a seam worth naming, because the half that does not transfer is the half people reach for. Card numbers are interchangeable across merchants, which is what made a single substitute workable: one scheme could define what a card token means everywhere, and every merchant could adopt the same replacement. Refresh tokens for forty different third-party products are not interchangeable, because each vendor defines its own grant, its own lifetime and its own revocation path, so there is no industry-wide substitute anyone can adopt on your behalf. What transfers is the argument rather than the remedy — that the concentration was the problem and not the quality of the code around it. The remedy has to be built locally, which is why the next chapter costs three specific arrangements instead of naming a standard.

The likely response is that per-tenant isolation in code is the industry norm, and that the alternative does not scale. The first half is true. Most multi-tenant SaaS runs shared processes with logical isolation, and it is a reasonable posture when the data at stake is the vendor's own application data, because a compromise then yields access to a system the vendor operates, monitors and can shut off. A design that stores tenants' credentials is different in kind: the assets are the tenants' keys to third-party systems that the vendor does not operate, cannot monitor, and cannot shut off. A norm developed for the first case has been carried into the second without anyone re-deriving it.

What survives from every earlier chapter is worth stating rather than implying. Per-request identity, credentials as arguments, a scoped store and an adversarial suite make application defects rare and catchable, and application defects are the common case. The limit is that their protection is expressed in your code, and code execution is exactly what an RCE takes away, so the yield of one is set by what the process is permitted to hold rather than by how carefully it was written.

The second half of the objection — that the alternative does not scale — has to be answered with something specific, because an objection without a proposal is a delaying tactic. Refusing a design obliges you to name what you would build instead, and to be honest about what that costs in operational surface, in user friction, and in the number of services someone has to run at three in the morning.