Marking the serving tier

A serving tier is which entry in the fallback chain actually produced a given answer: the primary vendor, a secondary vendor, a cached response, or a smaller model you host yourself. During normal operation that is always the first entry and nobody thinks about it. During a failover it is not, and the question this chapter is about is how anybody — a dashboard, a support agent, a reviewer, whoever writes a timeline up three days later — finds out which answers were produced by which tier.

The default answer is that nobody finds out, because a response from a weaker model is byte-for-byte the same shape as a response from the primary. It has the same fields, the same content type, and the same absence of any statement about where it came from. Everything downstream therefore treats it identically, which is convenient right up until somebody needs to know which answers to distrust.

It is worth following that from the other direction, because the fix is not where people look for it. Monitoring works by reading what a request and a response contain: a status, a duration, a body of a certain shape. Which weights produced the text inside that body is not one of those things. It is not absent from the dashboards because nobody built the panel; it is absent because the fact was never written down anywhere a panel could read it. There is exactly one moment at which the information exists — the instant the gateway picks an entry from the chain — and one process that holds it, which is the gateway itself. Everything after that point is handling text that no longer knows its own origin. So the label is not a reporting feature to be added later at the observability layer. It has to be manufactured at the routing decision, because that is the only place it is available to be manufactured at all.

The response has to say who answered it#

The smallest change with the largest reach is to make the serving tier part of the response, everywhere the response goes.

{
  "answer": "This contract renews on 14 March 2027 unless notice is given by 14 January 2027.",
  "serving": {
    "tier": "secondary",
    "provider": "vendor_b",
    "model": "vendor-b-large-2026-03",
    "degraded": true,
    "reason": "breaker_open:primary",
    "eval_ref": "extraction-v4/eval_2026_04_11_a"
  }
}

degraded is a boolean the caller can branch on without knowing anything about your provider inventory, and it is derived from the route's policy rather than from the tier index — a route whose chain declares two verified-equivalent providers is not degraded when it uses the second one, while a route falling back to a 24-hour cache is degraded even though the cache is technically tier three. reason distinguishes a breaker-driven failover from a manual chain pin from a cache hit, which is the difference between an incident and a deliberate change. eval_ref points at the measurement that justified putting this provider in the chain, so that a person reading a suspect answer six days later can find out in one click what the suite said this model gets wrong.

The field is worth very little if it stops at the gateway boundary. It has to survive every hop that the answer itself survives:

The branch that matters most is the persisted one. A log line carrying the tier is useful for two weeks and then falls off retention, while a contract answer lives for years and can be questioned at any point in them. Storing serving on the answer record turns a hand reconstruction into a query:

SELECT conversation_id, user_id, created_at, answer
FROM assistant_answers
WHERE serving_degraded = true
  AND created_at BETWEEN '2026-04-16 09:12' AND '2026-04-16 09:52'
ORDER BY created_at;

That query is also the precondition for remediation. Once you can enumerate the answers, you can re-run all of them against the primary once it is healthy, diff the extracted fields, and find the ones that materially changed. At the primary's measured cost of $2.10 per thousand requests, enumerating and re-running a 40-minute window of 1,027 answers costs about two dollars and twenty cents of inference and finishes in under ten minutes. Cost is never what stops this from happening.

That last figure is one multiplication and it is worth doing, because the instinct that re-running a window of production traffic must be expensive is what stops the conversation before it starts. A thousand and twenty-seven answers is a little over one thousand, and the primary costs $2.10 per thousand requests, so the whole re-run is a little over $2.10. The reason the number is that small is that inference is priced per request and the window is bounded by the incident, so the bill scales with forty minutes of your traffic rather than with anything about the severity of the problem. What actually blocks remediation is not knowing which rows to re-run, and that is a schema decision taken months earlier.

Answers produced by two models#

Streaming complicates the field in a way worth deciding once rather than per incident. A stream that dies at token 900 and is restarted on the secondary produces a single answer the user reads as one thing, assembled from two models with different views of the preceding context. The record for that answer needs tier: "mixed" with both entries listed, because it is neither primary nor secondary and averaging it into either bucket makes the retro query lie. For routes the degradation policy of chapter 5 marks as consequential, mid-stream failover should not be attempted at all: abandon the partial output and return the honest error, since a document whose first half was written under one set of assumptions and whose second half was written under another is harder to audit than no document.

From enumeration to remediation#

Identifying the affected answers is the beginning of a process, not the end of one, and the steps are mechanical enough to script before you need them. Re-run each degraded request against the primary once it is healthy. Diff the structured fields rather than the prose, since a summary reworded is not an incident and a date changed by a year is. Classify the differences into immaterial and material using the same schema the feature already has, and count them.

On one 40-minute window of 1,027 degraded answers the arithmetic came out as 812 identical after re-running, 173 differing only in wording, and 42 differing on a date or a monetary amount. Of those 42, telemetry showed 11 had actually been surfaced to a user rather than generated in a background pass, and 3 had been forwarded into an outbound email. That is a list of three people to contact urgently, eight to contact within a day, and thirty-one records to correct quietly — a completely different conversation from six days of uncertainty followed by an estimate.

Both partitions in that paragraph are worth checking, because the whole value of the exercise is that the pieces add up and nothing is left in a residual bucket. The first splits the window: 812 identical, 173 reworded and 42 materially different account for all 1,027 answers, so every degraded response has been placed in exactly one category and none is unaccounted for. The second splits the 42 by whether anybody saw them, which is a different question from whether they were wrong. Eleven reached a person, and the remaining thirty-one were generated in a background pass and have harmed nobody yet. Of the eleven, three were forwarded outside the company and eight were not, which leaves the eight inside a support tool where they can be corrected before they travel. Those three groups are the three different actions in the sentence, and each is addressed to a different person on a different clock — which is why the classification is worth doing rather than reporting a count of forty-two.

Two things make that pipeline real rather than aspirational. The recompute queue from chapter 5, where every artefact produced by a fallback tier is enqueued for regeneration once the primary is healthy, handles the write-path half by replacing degraded artefacts in durable state on the same pass. And the decision about who gets notified belongs to product and legal rather than to engineering, which is precisely why engineering owes them the classified list within hours: a notification decision made on a list of 42 specific records is a decision, while the same decision made on "sometime last Thursday, some answers may have been wrong" is a gamble that usually resolves as doing nothing.

Alert on the mix, not on the errors#

An error-rate alert cannot see a successful failover, because a successful failover has no errors — that is its purpose. The signal to watch is the composition of your traffic by tier, which is flat at approximately 100% primary in normal operation and moves sharply the moment anything degrades.

sum(rate(llm_responses_total{tier!="primary"}[2m]))
  / sum(rate(llm_responses_total[2m]))  > 0.05

Held for two minutes, that pages. It fires on a breaker opening, on a manual chain pin, on a quota problem pushing traffic down the chain, and on a bad deploy that changed the route policy, all of which are things you want to know about within two minutes. Routes that the degradation policy marked as non-degradable get a stricter rule with no threshold at all, since a single degraded response on the refund path is by policy impossible and its appearance is a bug.

Two properties of that expression are choices rather than conventions, and both are worth understanding before the numbers get tuned. It is a ratio rather than a count, so it means the same thing at four in the morning and at peak: a fixed count of degraded responses would be alarming at the quiet hour and unremarkable at the busy one, while a share of traffic is comparable across both, and a degradation that scales with your traffic is caught either way. And the threshold is held for two minutes rather than evaluated instantly, which keeps a brief excursion — one breaker probe, one pod reading a new config a moment late, a single cache hit on a route that rarely takes them — from waking anybody, while a real failover sits above the line continuously and pages on the schedule you chose.

Expressing it as a budget makes the conversation with product concrete in advance rather than during an incident. If a consequential route is allowed 30 degraded minutes a month, that number is small enough to argue about, it is measurable from the same counter, and its exhaustion is a defined event rather than a judgement call. It also gives the answer to "should we make the secondary stronger" a shape: strengthening the secondary buys you more degraded minutes at the same risk, which is a trade someone can price.

Distinguishable to whom#

A label serves three audiences and they need different things, which is where teams either under- or over-deliver.

Your own systems need it always, in the payload and in the record, without exception, because that is what makes enumeration and remediation possible. Internal humans — support agents, legal reviewers, anyone whose job involves acting on the output — need it visible in the tool they actually use, which usually means a field in the admin view and a filter in the case list, and not a log they have never opened. End users need it only where it should change what they do. A persistent "AI may be degraded" banner on every response teaches users to ignore banners within a week, so reserve the user-visible form for the routes and the tiers where the honest recommendation is to double-check, and let the rest be invisible to them and fully visible to you.

The internal-tooling half is the one that gets deferred and should not be. The support agent's case view needs the tier on the same screen as the answer, with the model version and a link to the eval reference, so that "was this a degraded answer" takes a glance instead of a Slack thread. The case list needs a filter on it, so an agent working a batch of complaints can see whether they cluster in one window. A legal review queue needs it as a sortable column, because a reviewer deciding how much scrutiny a machine-generated redline deserves is making exactly the decision this field informs. None of that is difficult; all of it requires someone to treat the support and review tools as part of the system rather than as somebody else's product.

On the metrics side, tier, provider, model and route are all low-cardinality labels — each takes a handful of distinct values, so attaching them to a counter multiplies the number of time series by a small factor — and they belong on the counter. User and conversation identifiers do not, and the instinct to add them so that a retrospective query can run in the metrics system is the instinct to solve a data-warehouse problem with a time-series database. The enumeration lives in the answer records, where the cardinality is free and the retention is years.

The multiplication is the part to hold on to, because it explains why one of those additions is harmless and the other is not. A time-series database stores one series per distinct combination of label values, so adding a label with four possible values to a counter that already has ten series produces forty, and adding one with a handful of values on top of that produces a few hundred. Adding a user identifier adds one series per user, and then one per user per tier, per model and per route, because the combinations multiply rather than add. The cost is not paid in disk alone; every query has to scan the series it might match, so the whole system slows for everybody. A warehouse row, by contrast, costs the same whether its identifier column has four distinct values or four million, because nothing about a row's identity is precomputed. The two stores are not competing options for the same data — they are answers to two different questions, one about how traffic is behaving right now and one about which specific records were affected.

A label is provenance, not confidence#

What the label does not do is make a wrong answer safe. It is metadata about where an answer came from rather than a statement about how likely it is to be right, and a degraded answer marked as degraded is still a wrong renewal date if someone acts on it. That is why the marking sits alongside the per-feature degradation policy of chapter 5 rather than replacing it: routes where something automated will act on the answer should fail rather than degrade, and the label is what protects the routes where degrading was the right call. A team that ships the label and skips the policy has built excellent forensics for a harm it chose not to prevent.

It is also not a substitute for measurement. Knowing that vendor B answered tells you nothing about how vendor B behaves unless somebody ran the suite, and eval_ref in the payload above is an admission of that dependency rather than a decoration.

The test that tells you the whole chain works#

There is a single exercise that checks everything in this topic at once, and it fits inside the game day from chapter 6, where the primary is blackholed on a schedule while the on-call rotation watches. Blackhole the primary, let the system serve a real request through the fallback, and then take the resulting response — the JSON, the database row, the screen the support agent sees — and ask whether a person holding only that artefact, with no access to the incident channel and no knowledge that anything was wrong, could tell that it was produced under degraded conditions. Then check that on a route the policy marks as consequential, no artefact was produced at all, because the system returned an honest error and something downstream queued the work for a human.

If both hold, the chain is doing its job. It moves traffic without a deploy, it moves it to a provider whose differences somebody has measured, it moves it only for the features where a weaker answer was agreed in writing to be better than none, and it leaves a mark on everything it touches.

If either fails, the next 40-minute window looks exactly like the one in this chapter: a flat error graph, a closed ticket, and a support agent six days later holding a renewal date that does not match the contract. The deciding factor between those two outcomes is whether anything that survived the request said which model produced the answer.