Revoking one service beats rotating a shared secret

At 09:14 a security engineer got an alert that a container image published by the pricing-worker build pipeline contained a decrypted environment file. The image had been in an internal registry for eleven days and pulled by 340 distinct hosts, most of them CI runners. Inside the file was the service's database password, an S3 credential, and a provider API key beginning sk-ant-api03-. Nobody knew yet whether the key had been used from outside the network. The vendor's usage dashboard breaks spend down by key, and the key in question was shared by four services, so the dashboard could not answer the question either.

The interesting part of that morning was not the compromise. It was the next decision, and how long it took. The provider key had to be treated as burned, which meant rotating it, which meant finding everyone who held it.

The objection worth taking seriously#

A staff engineer reviewing the gateway design had asked, two months earlier, why issuing each service a gateway token — a credential the gateway itself mints, is the only thing that accepts, and can withdraw at any moment — was better than issuing each service its own provider key from the same secret manager, meaning the encrypted store such as HashiCorp Vault or a cloud equivalent that holds the company's credentials and releases them to services at start-up. The question deserves a straight answer, because on the surface the two designs look identical in every property that security reviews usually check.

Both keep the secret out of source control. Both store it encrypted at rest with access mediated by a policy engine. Both can be rotated on a schedule. Both produce an audit log of who read the secret and when. Both inject the value at deploy time rather than baking it into an image, at least when everything works, which in this incident it did not. If the review checklist is "is the secret in a vault", the two designs score the same and there is nothing further to discuss.

What separates them shows up only in the shape of the recovery, and specifically in whether containing one caller requires the cooperation of every other caller.

The rotation dance, with timings#

Rotating a provider key that four services hold is not one operation. Most vendors let an organisation hold several active keys at once — a common limit is 25 — which is what makes a safe rotation possible at all, because you can create the replacement before destroying the original. The sequence is: create key B, write B to the vault path each holder reads, trigger a restart or a secret refresh on every holder, confirm every holder is now signing with B, and only then delete key A. Delete A before the last holder has picked up B and you have taken that holder's feature down during a security incident.

Confirming that last step is where the time goes. The provider's dashboard reports per-key usage with a lag of several hours, so "has anything used key A in the last ten minutes" is not a question you can answer from the vendor. You end up asking each owning team to check their own deployment, which means finding four teams on a Wednesday morning. In this incident, two teams responded within twenty minutes; the third owned a service with a manual deploy approval and got to it after lunch; the fourth had a scheduled job that only ran at 02:00 and could not be verified until the following night. Key A was finally deleted 31 hours after the alert, and for all 31 of those hours a credential that an unknown number of people might hold was live against the vendor's API with no per-service attribution and no rate limit of your own.

The variant of that story that platform teams actually fear is worse: someone deletes key A on schedule at hour four, the 02:00 job wakes up, fails on 401, retries its whole batch, and the incident acquires a second incident.

What the same morning looks like with a gateway#

The gateway holds one key per vendor. pricing-worker holds a gateway token, and that token is the thing in the leaked environment file. Containment is a single call.

curl -X POST https://ai-gw.internal/admin/tokens/revoke \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -d '{"principal": "svc:pricing-worker", "reason": "SEC-4471 image leak"}'

Nothing else in the company is affected, because no other service uses that token. The provider key is untouched, so there is no rotation, no coordination, and no window in which a leaked vendor credential is live. pricing-worker is down for models until its owners deploy with a fresh token, which is the correct outcome: the compromised thing is the thing that stopped working.

The second property that matters on that morning is attribution. Because every request carries the caller's own token, the gateway can name the principal behind each one — the identity a request is made under, of which there is one per calling service rather than one per vendor account — and its log therefore answers the question the vendor dashboard could not.

{"ts":"2026-03-04T02:11:08Z","principal":"svc:pricing-worker","src_ip":"10.4.19.22",
 "model":"anthropic/claude-sonnet-4-6","input_tokens":1180,"output_tokens":402,
 "status":200,"feature":"margin-explainer"}
{"ts":"2026-03-04T02:11:09Z","principal":"svc:pricing-worker","src_ip":"203.0.113.44",
 "model":"anthropic/claude-sonnet-4-6","input_tokens":9,"output_tokens":1,
 "status":200,"feature":null}

The second line is what you are looking for during an incident: the same principal, from an address outside the cluster, making the small probe request that someone makes when testing whether a stolen credential works. With four services sharing one provider key and no gateway, that request is indistinguishable from legitimate traffic in the only log that exists.

There is a third property that only becomes visible once the tokens exist, which is that a per-service credential is a place to hang policy. A provider key carries exactly one grant: full access to the account's models at the account's rate limits. A gateway token can carry a scope, and the scope is where a great deal of ordinary risk gets removed without anyone writing a security control.

{
  "principal": "svc:pricing-worker",
  "models": ["anthropic/claude-haiku-4-5"],
  "max_output_tokens": 1024,
  "rate_limit_rps": 20,
  "monthly_budget_usd": 300,
  "data_classes": ["internal"],
  "expires": "2026-06-01T00:00:00Z"
}

With that in place, an attacker holding the leaked token cannot reach the expensive long-context model, cannot run 500 requests per second, and cannot spend more than $300 before the request is refused. None of those limits are expressible in a provider key, because the vendor has no idea which of your services is calling. The data_classes field is the one that tends to earn its keep in review meetings: it lets the gateway refuse a request from a principal that is not approved to send customer data to a vendor whose contract does not permit it, which is a control the security team would otherwise have to implement as a code review policy in twenty-two repositories.

Revocation is only as fast as the check that enforces it#

The single-call story assumes the gateway asks, on every request, whether this principal is still allowed. That assumption is worth checking in your own design, because the convenient token format breaks it.

Signed tokens, of which the JSON Web Token or JWT is the usual format, are attractive because validating one requires no state: the gateway verifies a signature and reads the claims the token carries about who it belongs to and when it stops working. The cost is that a signed token is valid until the expiry stamped inside it — its TTL, or time to live — and the issuer has no way to take it back. Revoking a principal whose tokens carry a one-hour lifetime means up to an hour during which the leaked credential still works, unless the gateway also consults a denylist, at which point the design is stateful again and the reason for choosing signed tokens has evaporated.

The workload-identity row is where new deployments tend to land, because it gets the credential out of the environment file entirely. The service presents a platform-issued identity document — a Kubernetes projected service account token, or a SPIFFE SVID — and the gateway exchanges it for a five-minute access token. The leaked image in this story would then have contained a credential that expired eleven days ago. This is what "use short-lived credentials where supported" means in practice, and the five-minute window is the honest cost: revoking a principal takes effect within one token lifetime, not instantly, unless you pay for the denylist as well.

Whichever row you pick, the number to write down is the revocation latency, because that is the number an incident commander will ask for while the incident is running.

Rotating the vendor key itself gets easier at the same time, which is the quieter benefit of having one holder. The gateway can be built to hold two vendor keys per provider and to prefer the newer one, so a rotation is: create key B, write it to the vault path the gateway reads as primary, move the old value to secondary, wait for the gateway's five-minute cache to expire, confirm from the gateway's own logs that every replica is signing with B, then delete A. The confirmation step that took 31 hours in the incident above takes as long as the cache TTL, because there is exactly one process to confirm and it reports its own key fingerprint in a health endpoint. Scheduled 90-day rotation stops being a project somebody dreads and becomes a cron job, which is the difference between a rotation policy that exists on paper and one that has actually run four times.

What this does not contain#

A gateway token is still a bearer credential, and an attacker who holds one can send requests. What they cannot do is take the credential somewhere else and use it against the vendor directly, and what they cannot do is spend beyond the budget attached to that principal, and what they cannot do is act without appearing in a log that names them. That is containment rather than prevention, and the difference matters when someone describes central key management as making the service "secure".

The central key is also a more attractive target than any individual service's key was, which is a real objection and has a real answer: it lives in one place, that place is on the platform team's on-call rotation, it is fetched over an authenticated channel and cached in memory rather than written to disk, and there is exactly one process to audit rather than 22. Concentrating a secret is only an improvement if the concentrated location is meaningfully better defended than the average of the places it replaced. If the gateway logs request bodies to a shared observability cluster that half the company can read, you have moved the incident from credentials to prompts and customer data, which is not obviously an improvement.

And none of this removes the secret manager. The gateway fetches the vendor key from a vault with a short cache; the vault still holds it, still rotates it, still audits reads. The design does not replace central secret storage — it reduces the number of principals that need a copy from twenty-two to one.

There is a larger assumption underneath the whole chapter, and it is the one the 02:00 batch job in the first story quietly violated. Containing a compromise with one revocation only works if the compromised service had no other way to reach the provider. The moment any workload in the company holds a provider key of its own, the rotation dance is back for that key, and the audit log has a hole exactly the shape of that workload. Which raises the question of how much coverage is enough, and whether a number like 94% means what it sounds like it means.