Latency routing and data residency imply different failover rules

The incident call is eleven minutes old. The EU deployment is rejecting 30 percent of requests against its quota, the emergency capacity increase has been filed and the platform team's estimate is four to six hours, and someone has just pointed out that the US deployment is running at 40 percent of a 600,000-TPM allocation. The obvious move is on the table before anyone has thought about it: point EU traffic at the US endpoint, take the extra 90 milliseconds of latency, restore the service, and sort it out afterwards.

About half the affected traffic — 110,000 TPM — belongs to customers whose contracts commit to processing their data in the EU. Someone checks and reports that the contracts do not mention outages. On the call, that fact gets read as permission.

It is the opposite of permission. Silence about outages means no degraded mode was ever agreed, which is a gap to close rather than a licence to interpret. Routing that traffic to the US would restore availability by breaking a processing commitment, converting a four-hour capacity incident into a contractual and regulatory one, and the disclosure that follows lands worse than the outage would have. The answer on the call is that the unconstrained half fails over now and the constrained half is shed or degraded in the EU until capacity arrives, with a written note to the affected customers explaining that their data stayed where it was promised to stay.

The reason this decision has to be made under pressure at 30 percent rejection is a sentence in a design document written eighteen months earlier: "we deploy the inference path to three regions for latency and for GDPR data residency."

Two goals, one topology, incompatible failover#

Data residency is a commitment that a named class of data will be processed and stored only inside a stated legal boundary, which for European customers usually traces back to the GDPR, the EU's General Data Protection Regulation, and then into a clause in the contract they signed. Latency and residency are both legitimate reasons to run in several regions, and they impose opposite constraints on what the router may do when a region is unhealthy — that is, on failover, the redirection of traffic away from a region that has stopped serving reliably and onto one that has not. Latency routing wants the largest possible set of destinations, ranked by proximity, so that a degraded region's traffic flows to the next-nearest healthy one. Residency wants the smallest possible set, fixed by contract, with everything outside it forbidden regardless of health. A design that states both goals in one clause has not chosen between them, and the router that gets built will implement whichever one the person writing the failover config had in mind that week.

The bottom row is the one that changes how the work is done. A latency target can be renegotiated by an engineer on a Tuesday; a residency commitment is a term in a signed agreement, and the degraded mode has to be negotiated with the customer in advance and written into the contract. The clause that is needed is short — during a regional capacity or availability incident, processing may be moved to another region within the same boundary, and if no such region is available the service will return errors rather than process outside the boundary — and it is the difference between an incident call that has a decision to execute and one that has an argument to have.

Make the constraint a property of the request#

The router cannot apply a policy it cannot see, so residency has to arrive with the request as data rather than being inferred from which endpoint the caller happened to hit. A residency class on the tenant, resolved at ingress and carried through every hop, turns the incident decision into a lookup.

residency_classes:
  eu_only:
    allowed_regions: [westeurope, swedencentral]
    on_exhaustion: shed          # never leave the boundary
    tenants: [acme-de, banque-fr, nordic-health]
  eu_preferred:
    allowed_regions: [westeurope, swedencentral, eastus]
    on_exhaustion: failover      # latency goal only
  global:
    allowed_regions: [eastus, westeurope, japaneast]
    on_exhaustion: failover

routing:
  order: nearest_healthy_within_allowed
  health: rejection_rate_10s > 0.15 OR ttft_p95 > 8s
  shed_response:
    status: 503
    body: "Capacity is constrained in your data region. Retry shortly."

The eu_preferred class is doing quiet but important work. Most tenants in a European deployment have no residency commitment at all; they were put in the EU region because their users are there. Separating them from the genuinely constrained tenants is what let half the traffic fail over on the incident call, and a system that treats "deployed in the EU" as equivalent to "must stay in the EU" would have shed all 220,000 TPM instead of 110,000. Getting that split right is a data-gathering exercise against the contract set, not an engineering judgement, and it is worth doing before it is needed because the answer takes days to assemble.

Residency multiplies the capacity you have to buy#

Partitioned capacity cannot be pooled, and that changes the arithmetic from the previous chapter. Consider a workload with a 400,000-TPM global peak split 50 percent EU, 30 percent US and 20 percent APAC. Under pure latency routing across three regions, surviving the loss of any one region needs each survivor to absorb its share, so the total commitment is roughly peak × n/(n−1), which is 600,000 TPM. Under residency, each boundary has to be self-sufficient, so the EU needs two EU regions each able to carry the full 200,000 EU peak, the US needs two US regions carrying 120,000, and APAC needs two carrying 80,000.

The 33 percent premium is the price of the commitment, and it is a number worth putting in front of whoever signs the contracts, because residency is frequently offered as a sales concession by people who have never seen it costed. The alternative to paying it is to be explicit that the EU boundary runs without in-boundary redundancy, in which case the degraded mode is shedding and the contract has to say so.

Two further regional facts break plans that were made globally. Quotas are granted per region and per subscription, so a "global TPM" figure in a capacity plan is an aggregate that no enforcement point ever sees; the crossover calculation from chapter 1 has to be done for each region separately, against that region's own limits. And model availability differs by region, so a failover destination may only offer an earlier model version, which tokenises slightly differently, produces different output lengths, and may respond differently to the same prompt. A failover that silently changes the model changes the token arithmetic of the region receiving the traffic, which is how a region with apparently sufficient headroom exhausts it faster than predicted.

What multi-region does not give you#

Running in three regions is not high availability by itself. It is a prerequisite, and it delivers nothing unless the router can detect an unhealthy region quickly, unless the destination has real headroom rather than a quota someone assumed, and unless the failover path is exercised. A three-region deployment where all traffic is pinned by DNS to the nearest region and no automated health check exists has three single points of failure rather than one.

Residency is also narrower than sovereignty. A commitment to process in the EU is a statement about where computation happens; it says nothing about who can access the data, which jurisdiction can compel disclosure, whether the provider's control plane sits elsewhere, or whether prompts are retained for abuse monitoring in another region. Customers asking for sovereignty want answers about all of those, and a region selector does not supply them. Conversely, latency routing is not a data-locality mechanism in the storage sense — routing inference to a nearby region says nothing about where the vector index or the conversation history lives, and a retrieval step that reaches back across the Atlantic makes the latency argument for the local region substantially weaker than the diagram suggests.

Failover is not free at the moment it fires, either, which is easy to miss when the destination has visible headroom. Moving 110,000 TPM into a region running at 40 percent of 600,000 is arithmetically comfortable and operationally abrupt: the shift happens over seconds, the receiving region's admission ledger sees its reserved tokens jump by a third, and any per-tenant budgets configured there were sized for its own tenants rather than for a doubled tenant set. Health checks that flap will move the traffic back and forth, so the router needs hysteresis — a region marked unhealthy stays unhealthy for a minimum period, and returns to service by taking a fraction of traffic first. Without that, the failover mechanism becomes its own load generator, in the same way that an unjittered retry policy does.

The last thing the incident call surfaces is a question that will be asked afterwards by someone senior: why was capacity sitting unused while customers were being rejected. The answer that holds up is that the unused capacity was in a region we are not permitted to process those customers' data in, that using it would have converted a capacity incident into a breach of a signed commitment and a reportable transfer, that the traffic which could legally move had already moved within four minutes, and that the gap being fixed is the absence of an agreed degraded mode in the contract and a second EU region to fail into. Every clause of that answer is a decision made before the incident, which is the only time any of them can be made well.

All of this — the ladders, the reservations, the deadlines, the breakers, the routing classes — is a plan, and a plan whose failure paths have never executed is a set of assertions nobody has tested. The last chapter is about deliberately crossing the ceiling to find out which of them are true.