Per-feature degradation policy

A fallback chain is an ordered list of places a request can be served from when the one above it fails — a second vendor, a cached answer, a smaller model you host yourself — and serving from further down that list rather than returning an error is graceful degradation. This chapter is about the decision that mechanism cannot make for itself: whether a feature's users are better off with the weaker answer or with an honest failure.

The instinct is to settle that once for the whole system, usually with some version of "never show an error — always show something". That is a product position rather than an engineering default, and which features it fits depends on properties unrelated to how much anybody cares about them.

It is worth being fair to why that position is so durable, because it is not laziness. An error is a visible, attributable failure: somebody sees it, it lands in a graph, and a support ticket names it. A wrong answer produced by a weaker model is none of those things. It arrives in the same shape as a right one, nobody reports it, and if it does surface it surfaces weeks later attached to a customer rather than to an incident. So the two outcomes are compared with one of them fully priced and the other invisible, and the cheaper-looking option wins every time it is compared that way. The whole of this chapter is an attempt to price the second one before an incident makes the comparison for you.

Sort by what acts on the answer, not by how important the feature is#

The instinct is to rank features by importance and let the important ones have the better tier. That produces the wrong assignment, because importance does not predict how a wrong answer propagates. The properties that do are three.

Where those three come from is worth deriving rather than accepting, because the derivation is what lets you apply them to a feature that appears in none of the tables below. What a degraded tier actually costs is not the wrong answers it produces but the harm those answers do, and harm has three factors multiplied together: how often the answer is wrong, what it costs when something acts on a wrong one, and how likely it is that nothing catches it before the acting happens. The first factor belongs to the model and is the same for every feature served by that tier. The other two belong to the feature, and each of the three properties below is one of them turned into a question you can answer about a route without running anything. Importance appears nowhere in that product. A feature can be the most important thing you ship and still be a perfectly safe place to degrade, because it is checked by a human who has the source in front of them.

Reversibility: if the answer turns out to be wrong an hour later, what does undoing it cost? A summary the user re-runs costs nothing. A refund already paid costs the refund plus a support conversation plus, in some jurisdictions, an argument about whether it can be clawed back at all.

Detectability by the recipient: can whoever receives this answer tell it might be wrong? A user reading a search summary has the source documents on the same screen and a strong prior that summaries are approximate. A support agent reading "this contract renews on 14 March 2027" has no way to evaluate it and every reason to paste it into an email.

Automation: does a human sit between the answer and the action? An answer feeding a rules engine, a workflow trigger or a database write has no such gate, and the model's uncertainty — which the model does not report and your gateway does not measure — is spent silently.

Score a feature on those three and the assignment falls out. High reversibility, high detectability, human in the loop: degrade freely. Irreversible, undetectable, automated: do not degrade at all, and fail with an honest error so the caller's own error path runs.

What "always show something" costs when you do the arithmetic#

A ladder proposed for one contracts assistant ran primary, secondary, a 24-hour response cache, then a small self-hosted model, erroring only if all four missed. It sounds like exactly this topic, and the numbers underneath it do not hold up.

The cache runs at a 12% hit rate, which is roughly one question in eight. Over a 40-minute outage carrying about 1,000 questions, that is 120 answers from cache and 880 falling through to the 8B model — eight billion parameters, small enough to serve from your own hardware and a long way below the primary in capability. On the same 300-case suite from chapter 4 (a representative suite scoring quality, schema adherence, latency and cost on both providers), that model gets the deterministic contractual facts right on 61% of cases. So the ladder, executed exactly as specified, produces around 340 confidently-worded wrong answers about contracts in 40 minutes, and by construction it produces them in a form the user cannot distinguish from the real thing.

The three steps that produce that figure are all one line of arithmetic each, and doing them out loud is what turns a plausible-sounding ladder into a number somebody has to defend. A hit rate of 12% applied to about 1,000 questions leaves 120 served from cache. Everything the cache misses falls to the next tier, which is 880 questions. That tier is right about the deterministic facts 61% of the time, so it is wrong the remaining thirty-nine times in a hundred, and thirty-nine hundredths of 880 is about 340. The only step in that chain worth arguing with is the 61%, which was measured on the eval suite rather than on the questions people ask during an outage; if outage traffic is harder than the suite, the figure moves up rather than down.

The cache tier has a second problem that is unrelated to model quality. A 24-hour-old answer about a contract that was amended this morning is wrong in a way no model can be blamed for, and the staleness is invisible for exactly the same reason the model's weakness is: the response carries no indication of where it came from. Cache freshness is a separate axis from model strength, and a degradation policy that treats "cached" as simply a weaker tier of the same thing will serve a confidently stale answer in preference to a fresh weak one, which is often the worse trade.

There is a third option that gets skipped because it is not on the ladder. "Never show an error" can be honoured without answering the question at all, by degrading what you claim rather than how good the model is: show the three contract clauses that matched the query with the query terms highlighted, say that the assistant cannot summarise them right now, and offer the document. That is not an error state and it is not a guess; it is the raw material, which is available even when the inference is not, and for a large fraction of question types it is what the user was going to check anyway.

Writing the policy down where the gateway can read it#

Once the classification exists it belongs next to the route, in the same configuration surface the chain lives in (chapter 3, where the chain becomes a value a running service reads), so that it is inspectable and changeable on the same timescale.

routes:
  search_summary:
    degrade_to: [secondary, cache_24h, local_8b]
    on_exhausted: error
    marking: inline_notice

  contract_qa:
    degrade_to: [secondary, cache_1h_unamended]
    on_exhausted: retrieval_only     # show clauses, no generated answer
    marking: inline_notice

  refund_eligibility:
    degrade_to: [secondary_verified]
    on_exhausted: queue_for_human
    marking: blocking

  lead_enrichment:                    # async, nobody is waiting
    degrade_to: []
    on_exhausted: retry_with_backoff_for: 6h

The last row is the one people forget exists. When nothing is waiting on the answer, the axis to degrade along is time rather than quality: hold the work and run it when the primary is back. A batch pipeline that quietly switched to a weaker model during a 40-minute outage has permanently written weaker data into a warehouse in exchange for finishing 40 minutes sooner, which nobody asked for.

A weaker model can be asked to abstain instead of guessing#

The 61% figure for the local model assumes it is asked the same question in the same way as the primary, and that assumption is worth attacking, because a fallback tier does not have to run the primary's prompt. A smaller model instructed to answer only when it can quote the clause its answer comes from, and to return an explicit not_determinable otherwise, produces a different distribution: on the same suite subset, the 8B model moved from 61% correct and 39% wrong to 58% correct, 6% wrong and 36% declining to answer.

Read that as two subtractions and one ratio, because the ratio is the whole argument. Wrong answers went from 39 in a hundred to 6, which is thirty-three fewer. Correct answers went from 61 to 58, which is three fewer. Those two losses add up to the 36 that now decline, as they must, since a refusal has to have come from somewhere. So eleven wrong answers per hundred turned into refusals for every one correct answer that was lost, and refusals are recoverable in a way that a confident wrong renewal date is not: a route that returns not_determinable can be retried later, queued for a human, or answered from the clauses themselves, while a wrong date has already been believed.

The mechanism behind that trade is worth naming, because it is not the model getting better. Nothing about the weights changed. What changed is where the decision boundary sits between answering and not answering, and moving it converts errors of one kind into errors of another. The instruction to quote a clause supplies a check the model can fail before it commits, so cases where no clause supports an answer stop producing one. The cost is that some cases where a clause did support an answer also stop producing one, which is the three per hundred, and the exchange rate between those two effects is a property of the task rather than something you set.

That change costs something real. The fallback tier now has its own prompt, which means it has its own rows in the eval suite and its own maintenance, and the abstention rate has to be calibrated or the tier becomes useless in a different way — a model that declines 80% of the time is an error path with extra latency and a bill attached. It also only works where the task has a checkable grounding, which contract extraction has and open-ended summarisation does not.

The general form is that degradation has more than one axis. You can degrade the model, the freshness, the scope of the claim, or the confidence required before answering at all, and the last of those is usually the cheapest to implement and the least often considered.

Degraded output that gets written down outlives the outage#

Everything above treats a degraded answer as something a user reads and then forgets about. The expensive cases are the ones where the answer becomes durable state, because the outage ends and the state does not.

A conversation summary generated by the fallback tier during a 40-minute window is carried into every subsequent turn of that conversation for as long as it lives, so a weak summary produced at 09:30 is still shaping answers at 16:00 from a fully healthy primary. An extraction written into a CRM field stays there until someone contradicts it. A document classification that drives routing has already routed the document.

Embeddings are the sharpest version, and they deserve a rule rather than a judgement. An embedding is the list of numbers a model produces to stand in for a chunk of text, positioned so that texts with similar meaning land near each other, and vector search retrieves by comparing those positions. Because the comparison is between embeddings rather than between texts, a fallback that computes vectors with a different model puts points from a different geometry into the same index, where they are not merely lower quality — they are meaningless relative to their neighbours, and they stay meaningless until the affected range is recomputed. Even a same-dimension model from another vendor produces a different space. The correct fallback for an embedding pipeline is time, not an alternative model: queue the work, alert if the queue depth exceeds what you can drain, and accept that new documents are unsearchable for the duration.

Two steps in that argument are the ones to hold on to. The first is why matching dimensions do not help. A model is not asked to put "renewal" in a particular place; each of its dimensions means whatever the training run made it mean, and two models trained separately assign meaning to their axes independently. Two lists of the same length are therefore two lists of coordinates on unrelated axes, and a distance computed between them is arithmetic performed on numbers that were never in the same space.

The second is why nothing complains. A vector search takes the query's list of numbers, computes a distance to every stored list, and returns the smallest few. That computation succeeds on a foreign vector exactly as it succeeds on a native one — the lengths match, the arithmetic is defined, and a number comes out. What comes back is a ranking, and a ranking has no field for "these entries were measured against different axes". So the search returns results, the call succeeds, the dashboard stays flat, and the only symptom is that some documents stop being retrievable for queries they should answer. That is why this case gets a rule instead of a judgement: every other degraded tier in this chapter announces itself eventually, and this one is designed by construction not to.

For the write paths where degradation is genuinely acceptable, the mitigation is a recompute queue. Every artefact produced by a fallback tier gets enqueued for regeneration once the primary is healthy, the regenerated version replaces it, and the queue is drained as part of closing the incident. That turns a permanent quality dent into a temporary one, and it depends entirely on being able to identify which artefacts came from which tier — which nothing in this chapter has yet provided.

A policy is a statement of intent about code that may not exist#

Writing the classification down does not make the tiers work. The cache_1h_unamended tier above requires a cache that knows about amendments, and the secondary_verified tier requires the measurement evidence from the previous chapter attached to the chain entry. marking: inline_notice is a word in a YAML file until something in the response payload carries it, which is the subject of the final chapter.

More immediately, every tier in that configuration is a claim about a code path that may never have executed. The refund route's secondary_verified entry, the retrieval-only fallback for contract Q&A, the local 8B model: each is written down, each is deployed, and if the primary has been healthy since the day they shipped, none of them has served a real request.

Where the three properties stop separating features#

The ranking assumes each feature has one recipient and one downstream, and features that fan out do not. An answer surfaced in a UI and simultaneously written to an audit log has one recipient who can evaluate it and one who cannot, and the policy has to be set by the worse of the two.

The classification tells you what each route may do while a provider is failing, and it is worth exactly as much as the tiers it names. The next chapter is about what a never-exercised fallback path is worth, which is close to nothing, and about the two exercises that change that.