Fallback chains and graceful degradation across providers

Read these first

• Multi-provider abstraction
• Error handling

What you will know

• Fail over to a second provider when the first is down, without a deploy

  Failing over should not need a deploy, which means the routing decision is configuration, not code.

• Decide what degradation is acceptable: a weaker model, a cached answer, or an honest error

  Decide in advance whether a weaker answer beats no answer for this specific feature.

• Run a representative prompt suite against both providers and measure quality, schema adherence, latency, and cost

  Provider-specific tool formats, safety behavior, tokenization, and output limits can make an apparently equivalent prompt fail or regress.

• Test the fallback path deliberately rather than discovering it in an incident

  An untested fallback path is a second outage waiting inside the first one.

• Run failure scenarios and verify the system returns an explicit error instead of silently degrading for consequential requests

  A stale or plausible-but-wrong answer can cause more harm than an unavailable feature, so callers must be able to distinguish degraded results from normal ones.

Chapters

1. 1

   Retry, circuit breaker and fallback

   Three mechanisms sit in the same gateway function and get treated as one, which is how a primary returning 500s on 40% of requests holds availability at 99.7% while p99 latency moves from 2.4 to 10.2 seconds. Separate them by the assumption each makes: retry treats a failure as transient and specific to this request, the breaker treats it as shared and durable and stops paying per request to rediscover it, and the fallback only says where traffic goes. The practical test is which of the three you could afford to lose, and the answer follows from pricing each one rather than ranking them by strength.

   Read chapter
2. 2

   Classifying provider errors

   A 400 or a schema-validation failure re-sent to the secondary fails identically, a content-filter block is deterministic for the same input and gets refused again while spending the fallback's quota, and a 429 against your own quota is a limit on you rather than an outage, so failing over moves your load instead of routing around a fault. Only a sustained rate of provider-side failures is the case a breaker exists for. This chapter covers which responses count as failures at all, how to classify on the error body rather than the status code, how a timeout converts a slow provider into something countable, and the four numbers that define the open state.

   Read chapter
3. 3

   Where the fallback chain lives

   The ordering of providers in a fallback chain is a routing decision, and storing it in application code puts it on the deploy pipeline's schedule: one 02:07 page produced a switch to the secondary that took 48 minutes to go live, by which time the primary had recovered. This chapter covers where that value has to sit for a running process to observe a change in seconds, how it is validated on load once code review is no longer doing that job, the three-layer read path that stops a config-store outage from becoming a routing change, and why the on-call action should be selecting one of four reviewed presets rather than composing a chain.

   Read chapter
4. 4

   Measuring provider equivalence

   Ten sample prompts read for plausibility cannot see a 30% schema-validation failure rate, and the first real failover is where that arithmetic lands: each vendor expresses constrained output through a different mechanism, and tool arguments can arrive as a JSON-encoded string rather than an object. This chapter builds a 300-case suite drawn from production traffic that scores schema adherence, deterministic facts, refusals, latency and cost per request on both providers, covers the tool-loop cases a single-request suite never reaches, and attaches the resulting run to the chain entry so that a model version bump invalidates it.

   Read chapter
5. 5

   Per-feature degradation policy

   Whether a weaker answer beats no answer is a per-feature question, and the properties that decide it are reversibility, whether the recipient can tell the answer might be wrong, and whether a human sits between the answer and the action. A search summariser scores well on all three and can take a cached or weaker-model answer; an automated refund-eligibility decision scores badly on all three and cannot. Reason through a 24-hour cache at a 12% hit rate and a small self-hosted model, and settle per feature, in writing with product, which requests are allowed to degrade.

   Read chapter
6. 6

   Exercising the fallback path

   Configuration that has never carried a request tells you nothing about credentials, quota, rate limits or how a secondary behaves on your prompts, because configuration does not authenticate and does not have a rate limit. Confidence comes from firing the path deliberately: a small named slice of live traffic routed to the secondary continuously, or a game day that blackholes the primary while the people who would handle the real incident watch. This chapter compares what each exercise proves, where in the request path the failure gets injected, and what both still leave untested.

   Read chapter
7. 7

   Marking the serving tier

   A successful failover produces no errors, which is what it is for, and therefore produces no signal on any graph that watches errors. Carry the serving tier in the response payload and on the persisted answer record, alert on the mix of tiers rather than on the error rate, and put the field in the tools support and legal actually use. That turns a six-day reconstruction into a query, and a vague estimate of harm into a list of specific records.

   Read chapter