A stateless proxy is still a total outage for nine teams

The deploy was version 1.14.2 and the change was a log field. It rolled at 11:47 on a Tuesday, and between 11:47:06 and 11:47:46 every model request in the company failed. Nine teams, 14 features, roughly 16,000 requests returning 502 and then 500. The cause was two ordinary mistakes compounding: the deployment manifest had maxUnavailable: 100% inherited from a template written for a stateless image resizer, so all three replicas terminated at once, and the new replicas' readiness probe — the check the orchestrator runs to decide whether a replica may be given traffic — returned 200 as soon as the HTTP server bound its port, which happened about eight seconds before the vault client had fetched the vendor keys. The load balancer sent traffic to replicas that were up and could not sign a request.

Both mistakes were fixed the same afternoon. The interesting part of the postmortem was a sentence from the search team's engineer: they had assumed a gateway problem would show up as slower responses or a worse model, and had never considered that it would show up as their feature returning an error to users. The recommendations team said the same thing in different words. Two of the nine teams had built no handling for the gateway being absent, because they had not understood that absent was one of the states it could be in.

Where the wrong mental model comes from#

The belief was not stupid. Everything teams read about gateways is about routing, fallback, caching and policy — the vocabulary is all about a component that chooses between options, and a component that chooses between options sounds like one that will choose a worse option rather than none. The onboarding docs at this company reinforced it: they described automatic failover between providers, a response cache, and per-team quotas, and said nothing about what happens when the gateway is not running.

A gateway is a synchronous, in-path dependency for every model call, in the same category as an internal auth service, or the sidecar proxy that a service mesh runs alongside every workload and routes all of its traffic through. Its unavailability is total for the capability it fronts. If it can also fall back to a cheaper model, that only helps when the gateway is running and the provider is not. Teams need both facts stated together, and the second one stated first, because it is the one that changes their code.

The state that is worse than absent, and that no team plans for, is a gateway that is slow rather than down. During the 40 seconds above the callers got errors quickly and their own timeouts never fired. A month later the same gateway spent nine minutes with a saturated connection pool, answering at p50 of 14 seconds, which is under the callers' timeouts often enough that requests succeeded slowly and over them often enough that some failed. Health checks passed the whole time, because the health endpoint does not open an upstream connection. Partial failure is harder to detect, harder to fall back from, and lasts longer, so a caller's degradation logic should trigger on latency as well as on errors — a client-side circuit that opens when p50 over the last 30 seconds exceeds a threshold catches the case that the error-rate breaker never sees.

The thing that would have made the outage predictable rather than surprising is a published dependency contract with numbers in it — a service level objective, or SLO, which states the availability the owning team is committing to and the latency the component adds — in the same form the platform team would expect from any other internal service.

# ai-gateway/SLO.yaml — published to every onboarding team
availability:
  target: 99.9%             # 43 min/month of total unavailability
  measured_as: successful_responses / requests, per minute, at the LB
  worst_month_observed: 99.87%   # Mar 2026, two incidents
added_latency:
  p50_ms: 6
  p99_ms: 41
  p999_ms: 310
failure_mode: total          # requests fail; no degraded path is provided
planned_disruption:
  deploys_per_week: 4
  expected_impact: none, but deploys are the largest source of incidents to date
caller_requirement:
  must_declare: behaviour_when_gateway_unavailable

Publishing 43 minutes a month makes the conversation concrete in a way that "highly available" does not. A team whose feature is a nightly batch job reads that and does nothing. A team whose feature is on the checkout page reads it and calculates that 43 minutes of checkout without their fraud explanation is either acceptable or not, and either way they now have a decision on record.

Redundancy answers the deploy, not the class#

The proposal that arrives in every postmortem of this kind is a second cluster in another region with health-based routing between them. It would have prevented this outage, since a rolling deploy can be staggered across clusters and a bad rollout in one leaves the other serving.

Whether it prevents the next one depends on what the next one is. Across 18 months of gateway incidents at this company the distribution was: six caused by configuration changes, two by bad builds, two by infrastructure or zonal failure, and one by an expired certificate that was deployed identically everywhere. A second cluster addresses the two infrastructure incidents and the deploy-shaped part of the builds, provided the deploys are genuinely staggered and somebody watches the first cluster for long enough to notice. It does nothing at all for a config change that propagates to both, which is the most common category, and nothing for a certificate that expires simultaneously in both places.

Redundancy is also expensive in a way that is easy to underestimate: a second cluster means a second budget-store replica or a shared one across regions with its own consistency problem, a routing layer with its own failure modes, and roughly double the infrastructure cost of a component whose entire job is to add 6 ms. Buying it before anyone has decided what a caller should do when the gateway really is down means spending money to reduce the frequency of an event whose consequences remain undefined.

The ordering that works is to define the consequence first, cheaply, and then decide how much frequency reduction is worth buying. Cell-based deployment — partitioning tenants across several small gateway fleets so that a bad config released into one cell first, the way a canary deploy exposes a change to a small slice of traffic before the rest, affects two teams rather than nine — is generally a better return than geographic redundancy for this class of failure, because it bounds the blast radius of the incidents that actually happen.

Three answers a caller can have, and the one that is not an answer#

Every caller needs a declared behaviour for the gateway being unavailable, and there are only a few sensible shapes.

The last row is what the two surprised teams had, and it is what every team has by default. Retrying against the gateway is not on the list because chapter 5 dealt with it: a gateway that is down does not come back faster for being asked repeatedly, so the caller's allowance is one immediate retry against a second replica and then a decision from the table.

The search re-ranking row is the one that most often turns out to be available and unconsidered. That team had a perfectly good BM25 ordering, the classical keyword-scoring rank that search engines used long before models arrived, which the model re-ranked on top of; falling back to it costs measurable quality and produces no error at all. Teams frequently have a pre-model version of the feature still in the codebase, and rediscovering it is the cheapest degradation available.

Break-glass, argued both ways#

The proposal on the table after the postmortem was a set of pre-provisioned provider keys plus a feature flag, so that a tier-1 path could call the provider directly during a gateway outage.

In its favour: it removes the gateway from the critical path for the small number of flows where 43 minutes a month is genuinely unacceptable, it is cheap compared with a second cluster, and it is honest about the fact that a checkout flow cannot wait for a platform team's rollback. Against it: every one of the earlier chapters argues the other way. The key is back in the calling service's reach, so containment is a rotation again. There is now a path outside the gateway, so audit completeness and the emergency block are conditional on a flag nobody is watching. A flag that is only exercised during incidents is a flag that does not work during incidents, because the SDK version has drifted, the model name was deprecated, and the key expired in April.

The version worth building keeps the benefit and pays down most of the objections. The break-glass key is not injected at deploy time; it is fetched from the vault at flip time by a service identity that is only granted that permission while the flag is on, so the ordinary state of the calling service is still keyless. It is scoped to one model with a low rate limit, so a stuck flag cannot spend without bound. Every call it makes is written to the same usage schema the gateway emits, from the caller, so the audit trail has an entry with a distinguishing outcome field rather than a hole. It is exercised on a schedule — quarterly, against production, for five minutes — and the exercise fails loudly if the model name or SDK has drifted. It is available to routes that have argued for it, which at this company was two of fourteen.

That last constraint is the one that decides whether this stays sane. Break-glass for every route is direct provider access with extra steps, and the argument of chapter 3 applies to it in full.

Onboarding is where the surprise gets removed#

The two teams who did not know were not negligent; they were onboarded with a page of API examples and a Slack channel. Making the surprise impossible means making the declaration a precondition of getting a token.

The onboarding form asks for the route's class from the table above, the declared behaviour, and the name of the person who agreed to it. Issuing the gateway token is gated on that record existing. Before go-live, the team runs one exercise: the gateway is made to return 503 for their principal specifically — a per-principal fault injection the gateway supports precisely for this — and the team demonstrates the declared behaviour actually happens. That exercise takes about twenty minutes and it is the entire difference between a documented intention and a tested one, since roughly half of first attempts at this company failed, usually because a client library's default was to raise an exception that nothing caught.

The exercise is then repeated on a schedule, because a fallback path with no traffic rots at the same rate as any other untested code. Running it as a quarterly game day, which is a failure staged deliberately in working hours with everyone watching, across all onboarded teams simultaneously has the additional property of measuring the thing the SLO cannot: what a total gateway outage costs the business, in whatever units the business uses.

None of this makes the gateway safe to depend on in the abstract. It makes the dependency legible, which is what lets a team decide whether to take it. For nine teams sharing eleven models across four vendors with a compliance obligation and a six-figure monthly bill, that decision is easy. For a team of two shipping one internal feature in a company where nothing else calls a model, the same dependency is the largest shared component in their system, and the answer is not obviously yes.