Where the fallback chain lives

A fallback chain is the ordered list of places a request can be served from — the primary vendor first, then a second vendor, then whatever sits below that, which is often a cached answer or a smaller model you host yourself. That ordering has to be stored somewhere, and the choice is usually made early, as a question about code organisation. It turns out to have been a question about how fast a human can change routing while a vendor is failing.

What counts as configuration#

The usual first home for a chain is a constant in the application:

PROVIDER_CHAIN = ["openai_gpt5", "anthropic_opus5", "self_hosted_8b"]

Changing that order means a branch, a pull request, an approval from someone asleep, a CI run that takes 11 minutes because it runs the integration suite, and a canary deploy that holds at 10% for 15 minutes before rolling the rest of the fleet. Everything about that path is correct for code. None of it is correct for a value that has to move faster than a human can write a commit message.

It is worth being precise about why each of those steps is correct for code, because the reason is what tells you which changes may skip them. Every step in that pipeline buys a guarantee about a change nobody has seen before. Review asks whether the change means what its author thinks; the integration suite asks whether it breaks anything else; the canary asks whether it breaks anything under real traffic while only a tenth of that traffic is exposed. All three are answering the same underlying question, which is what does this new code do, and that question is genuinely open for arbitrary code. It is not open for selecting one of four orderings of three providers that somebody already read, tested and approved last month. The pipeline's duration is the price of resolving uncertainty, and a change with no uncertainty left in it is being charged for a service it is not receiving.

Calling something configuration is not the same as it being configuration. A YAML file baked into the container image at build time is code with a different file extension: changing it still requires a build, an artefact push and a rollout, and it inherits every minute of the sequence above. The property that matters is not the format, it is that a running process can observe a new value without being replaced.

The mechanics underneath that sentence explain the whole table below, and they are worth a paragraph because most of the disagreement about where config belongs is really a disagreement about them. A container image is a stack of immutable layers addressed by a content hash, and a file inside one of those layers is part of what the hash is computed over. Editing it produces a different image, which is a different artefact, which has to be built, pushed and rolled out — there is no operation that edits a file inside a running image, because the identity of the image is its contents. An environment variable is stiffer still for a less obvious reason: the environment block is handed to a process when it is created and copied into its address space, and the operating system offers no interface by which one process can rewrite another's. Neither of those is a policy somebody chose and could relax. They are properties of the mechanism, and they are why a value that lives in either place can only change by replacing the process that holds it.

Three things have to be true before an on-call engineer can be trusted to change a fallback chain during an incident. The change has to take effect in seconds, because a change that takes 48 minutes will be applied after the failure it was meant to address and will then need reverting. The revert has to be a single action rather than the inverse of a multi-step procedure, because the second-worst moment to compose a new change is while the first one is misbehaving. And the space of possible changes has to be small enough that a tired person cannot express something catastrophic — reordering three known providers is a bounded operation, while free-text editing of a routing expression is not.

The first of those three is stronger than "the fix arrives late", and the difference matters enough to state. Compare two durations: how long the disturbance lasts, and how long your correction takes to arrive. When the correction is faster than the disturbance, it lands in the world it was designed for and does what it was meant to do. When it is slower, it lands in a world that has already moved on, and it is then applying a decision about conditions that no longer hold. A provider incident that resolves in forty minutes and a change path that takes forty-eight produce, reliably, a routing change made for a broken primary and delivered to a healthy one. That is not a late fix; it is an unrequested change pushed into production during an incident, and it has to be undone through the same path that was too slow the first time.

Environment variables are the trap in that table, because they feel like configuration and behave like code. Changing one requires the process to restart, which means dropping in-flight requests during an incident when in-flight requests are already suffering, and on a large fleet a rolling restart is slower than it sounds. They are fine for values that change monthly. A fallback chain changes during the worst ten minutes of a quarter.

The change latency in the last row is a TTL, the time-to-live on the service's cached copy of that row, after which it discards the copy and reads the value again, so the number in that column is one you choose and pay for in queries per second. The bottom two rows are what most teams converge on, and the practical difference between them is who writes the audit record. A remote-config product gives you the actor, the previous value, the timestamp and a one-click revert without you building any of it. A database row gives you the same capability only if you build the history table, and teams routinely do not, which is how a chain ends up in a state nobody can explain three weeks later.

Validating a chain you are allowed to change at 02:00#

The moment the chain becomes editable outside code review, the two protections that review was providing — someone checks it is well-formed, someone checks it is sane — have to move into the service. The service already knows which providers it can actually construct, so it can reject a chain naming anything else instead of discovering the typo when a request arrives.

const ChainSchema = z.object({
  version: z.number().int(),
  chain: z.array(z.object({
    provider: z.enum(REGISTERED_PROVIDERS),   // compiled-in, not from config
    model: z.string().min(1),
    timeout_ms: z.number().int().min(500).max(120_000),
  })).min(1).max(4),
  updated_by: z.string().min(1),
  reason: z.string().min(1),
});

function applyChain(raw: unknown): void {
  const parsed = ChainSchema.safeParse(raw);
  if (!parsed.success) {
    metrics.increment("routing.chain_rejected");
    log.error({ issues: parsed.error.issues }, "invalid chain, keeping current");
    return;                       // keep serving the last good chain
  }
  const next = parsed.data;
  if (next.version <= active.version) return;   // ignore stale reads
  log.warn({ from: active.chain, to: next.chain,
             by: next.updated_by, reason: next.reason }, "chain changed");
  active = next;                  // single atomic reference swap
}

Three decisions in that function are worth stating explicitly. The provider names are validated against a compiled-in registry rather than against a list in the same config, so a typo cannot invent a route that silently matches nothing. The registry has to be compiled in for the check to mean anything: a list of legal provider names sitting in the same document as the value being checked is not a constraint, because whoever mistyped one can just as easily mistype the other, and the document would still be internally consistent. What makes the check work is that one side of the comparison comes from a place the person making the change cannot reach.

An invalid document leaves the previous chain in place rather than crashing the process or applying a partial update, because the failure mode of a bad config push should be "the change did not take" and not "the fleet is in a restart loop during an incident". And updated_by and reason are required fields validated like any other, which is the cheapest possible incident timeline: the log line that says who moved traffic and why is the one you will want at the retrospective.

The version comparison two lines below those is the least obvious line in the function and the one that prevents the worst failure. A remote config store is usually several machines behind one address, and a value written to it takes a moment to reach all of them. During that moment two polls a few seconds apart can be answered by different replicas and return the new chain and then the old one, in that order, through no fault of anybody's. A process that simply applies whatever it last read will apply the new chain, then revert to the old one, then apply the new one again, and a fleet doing this independently is split across two routing decisions for as long as the disagreement lasts. Refusing any document whose version is not higher than the active one makes the sequence of applied values monotonic regardless of what order the reads arrive in, which is a property you get from one comparison and cannot get from any amount of care about how the value was written.

The config store is now on the critical path#

Making the chain remotely readable introduces a dependency that did not exist when it was a constant, and incidents correlate. The morning your provider is degraded is a plausible morning for your cloud region to be unhappy, and a service that blocks startup on fetching its routing config will fail to start at exactly the wrong time.

The pattern that holds up has three layers. The compiled-in default is the chain that ships with the binary, and it is what runs if nothing else is available; it should be the ordinary production chain, not something clever. The last-known-good copy is written to local disk every time a config is successfully applied, and is read at startup before any network call, so a pod that restarts during an outage of the control plane — the management services that store and hand out configuration, as against the data plane that serves your traffic — comes up with the chain that was live thirty seconds earlier. The live value is polled, every 10 seconds being a reasonable cadence for something that must move within seconds, and a failed poll leaves the current value untouched, with no TTL-based expiry that could drop you back to the default.

Serving a stale chain forever is the correct behaviour when the config store is unreachable, and the argument for it is an asymmetry between two costs rather than a preference. Compare what each option is wrong about. A process that keeps serving the last value it read successfully is wrong only if the chain changed during the outage of the config store, and the damage is that it routes the way it did a few minutes ago — which is how it was routing when somebody last looked at it and was satisfied. A process that expires the value and falls back to its compiled-in default is wrong whenever the live value differed from that default, which is precisely the situation you are in if anybody has made a change; and it becomes wrong on every pod at once, within one TTL of the others, triggered by an event that has nothing to do with routing. The first failure is bounded and resembles yesterday. The second is a synchronised fleet-wide routing change caused by an unrelated outage, arriving in the middle of that unrelated outage. Expiry looks like the careful option because expiring stale data usually is the careful option, and here the stale data is the only surviving record of a decision somebody made deliberately.

The property is directly measurable. In a staging environment, change the chain and measure the wall-clock time between the change and the first request served by the new head of the chain. If that number is under ten seconds and the revert is one action, the property holds. If it is two minutes because the process has to restart, it does not, whatever the file extension says.

Named presets and the expiry on a pin#

The uncomfortable part of moving the chain out of code is that code review was providing something real. Someone with context looked at the ordering and asked whether it made sense. Losing that entirely, in exchange for speed, is a bad trade, and the way to keep both is to move the review earlier: write the chain configurations you might want during an incident now, review them properly, and let the on-call action be selecting one rather than composing one.

{
  "presets": {
    "normal":          ["vendor_a/large", "vendor_b/large", "self_hosted/8b"],
    "pin_secondary":   ["vendor_b/large", "self_hosted/8b"],
    "drop_vendor_b":   ["vendor_a/large", "self_hosted/8b"],
    "local_only":      ["self_hosted/8b"]
  },
  "active": "normal"
}

The change surface is now a single field with four legal values, and a runbook can say "set active to pin_secondary" instead of describing an edit. A tired person at 02:00 cannot produce a chain with a typo, a duplicated entry, or a model name that was deprecated last quarter, because none of those are expressible. Each preset was reviewed by daylight, and adding a fifth one goes through the normal pipeline, which is fine because adding a preset is rarely urgent while selecting one always is. Presets also give you a place to encode the decisions from the rest of this topic: local_only is a chain that will produce noticeably worse answers, so the preset can carry the marking policy that goes with it rather than relying on someone remembering to set two things.

A manual selection needs an expiry. Every team that has run a few incidents has a story about a pin that stayed in place for weeks, quietly serving from a more expensive or weaker provider, discovered by an unrelated bill review. Attach expires_at to any non-default selection, default it to four hours, revert automatically when it passes, and alert when a pin has been active for more than an hour so that the choice to extend it is a choice someone makes rather than a thing that happens. The auto-revert is safe, because reverting to normal puts the primary at the head of the chain, and if the primary is still broken the breaker (chapter 2, the shared state that counts recent failures and stops sending a failing provider traffic) will open within its window and route around it exactly as it would have done anyway. The pin exists to override the breaker's judgement for a while, and "a while" should be a bounded quantity.

The reason a pin needs an expiry when a breaker does not comes down to which of them re-derives its verdict. A breaker holds a conclusion that it is continuously re-testing: the cooldown ends, probes go out, and if the primary answers them the breaker closes itself. Its verdict has a mechanism for becoming false again. A pin is a value somebody typed, and a typed value has no such mechanism — it stays exactly as true on Thursday as it was at 02:00 on Tuesday, because nothing in the system ever re-asks the question it answered. Left alone, an override that was correct for twenty minutes silently becomes the configuration, and it will keep being the configuration until an unrelated bill review or a curious engineer stumbles over it. An expiry supplies the missing mechanism from outside: it converts the default state from stay pinned unless somebody notices into revert unless somebody decides again, which puts the burden of a continuing override on the person who wants it to continue.

The breaker still makes the automatic decision#

A configurable chain does not decide anything. The breaker handles the automatic case, and it does it faster and more consistently than a human can; the manual override exists for the failures the breaker cannot see — a vendor serving 200s with visibly worse output, a pricing incident, a status page that says a fix is rolling out over the next hour, a regulatory instruction to stop sending data to a particular region. If your on-call engineers are flipping the chain for ordinary 5xx storms, the breaker is not doing its job and the fix is upstream of this chapter.

A destination that has never carried a request#

Making a chain change fast does not make it safe. That is a different property, and it is only useful when the destinations are known to work. Moving the head of the chain to a provider that has been in the config for eight months and has never carried a request is a fast way to turn one outage into two, and the argument that it must be fine because it is in the list is the argument that chapter 6 takes apart, by firing the fallback on purpose.

Where the chain lives decides how fast routing can change. A value the running process reads gets you the speed; validation on load gets back the well-formedness check that review used to do; and a short list of reviewed presets with an expiry keeps the space of expressible changes small enough to hand to somebody at 02:00. All of it has to exist before the page fires, because the pipeline you are trying to avoid is the same pipeline you would have to use to build it.

Between the automatic decision and the manual one lies an assumption that has been sitting unexamined since the first chapter: that the providers in the chain are interchangeable, so that moving traffic between them changes who answers and not what the answer looks like. That assumption is testable, it is usually false in at least one specific way, and finding out which way during a failover is expensive.