A verifier that times out must escalate, not wave through

At 09:14 the verifier's p99 went from 240 milliseconds to the 900-millisecond client timeout, and by 09:20 roughly a third of verifier calls were expiring. The code had a try/except around it that had been written in a hurry eight months earlier:

try:
    verdict = verifier.check(question, candidate, timeout=0.9)
except TimeoutError:
    log.warning("verifier_unavailable", request_id=req.id)
    metrics.increment("verifier.timeout")
    return candidate  # unverified, flagged in the log

The on-call engineer looked at it at 09:35, decided the behaviour was reasonable, noted that answers were being flagged for reconciliation, and went back to the provider's status page. The incident lasted four hours and twelve minutes.

Work out what that return candidate did. The surface handles about 1,670 requests an hour, so 6,970 requests passed through during the outage and about 2,370 of them hit the timeout path. The verifier's measured rejection rate on this traffic is 8.7%, so roughly 205 answers that the system was built to catch and re-run on the strong model went to customers instead, unchecked. Each one was well-formed, fluent, returned in 412 milliseconds with a 200 status, and — per chapter 4 — indistinguishable in telemetry from a correct answer. The metrics.increment call produced a number on a dashboard. The word "flagged" produced nothing at all, because no process consumed the flag, no queue received the request, and no human ever looked at the 2,370 log lines.

A flag is a promise that something else will happen#

Flagging is a legitimate pattern, and it is legitimate exactly when three things exist: a durable record of the flagged item somewhere other than the log stream, a named process that consumes those records, and a deadline by which they are consumed. An asynchronous document pipeline can flag an unverified extraction, write it to a needs_review table, and have a job re-run it against the strong model when the verifier recovers, because nothing has been delivered to anyone yet and the record can be corrected in place. That is reconciliation, and it works.

An interactive request has already been answered by the time the flag is written. There is nothing to reconcile; the customer read the answer and closed the tab. Under those conditions "flag and continue" and "ignore the failure" are the same code path with different log volume, and the choice being made is to accept unverified output on a third of traffic in exchange for not spending money.

The alternative costs a startlingly small amount. Escalating on the verifier's deadline means the timeout is treated as a rejection, which pushes those requests to the strong model. During the outage the escalation rate would have gone from its usual 12% to about 42% — the two thirds of requests whose verifier answered still escalated at the normal rate, plus the whole third that timed out — and per-request cost from $0.00167 to roughly $0.00251. Across 6,970 requests that is an extra $5.86. The incident's entire quality exposure was traded away to avoid six dollars, because nobody had decided in advance what the failure should cost and the default in an exception handler is always to continue.

The fallback that is already measured#

Escalating on timeout has to be bounded, since an escalate-on-failure rule combined with a total dependency outage means escalating 100% of traffic, which is the always-strong bill arriving without warning. The bound is a budget with a defined behaviour at the edge, and the useful observation is that a cascade has an unusually good option available: its safe degraded mode is always-strong, the arm it was measured against in chapter 5. Falling back to it costs the pre-cascade bill, which is a known and previously approved number, and produces the quality the design was calibrated to match.

The transition into AllStrong deserves a moment. When the escalation budget is breached the system is already paying most of the strong-model bill while still spending a cheap call and a doomed verifier call on every request, so cutting straight to the strong model is cheaper than the degraded state as well as safer. The one thing the transition must never be is back to serving unverified cheap answers, which is the direction an unthinking rate limiter would take it — capping escalations at 45% and letting the remainder through unchecked converts a cost control into a quality control nobody intended.

Those states live in a circuit breaker, which is a small piece of shared state that counts recent failures against a dependency and, once they cross a rate, changes what every subsequent request does without waiting for each one to fail on its own. The breaker is consulted before the verifier call rather than wrapped around it:

async def answer(req):
    candidate = await cheap.generate(req, timeout=0.9)

    if not schema_ok(candidate):           # free, deterministic, no dependency
        return await strong.generate(req, reason="schema")

    if breaker.state == "all_strong":
        return await strong.generate(req, reason="degraded_mode")

    try:
        verdict = await verifier.check(req, candidate, timeout=0.9)
        breaker.record_success()
    except (TimeoutError, ProviderError):
        breaker.record_failure()
        metrics.increment("escalation", reason="verifier_timeout")
        return await strong.generate(req, reason="verifier_timeout")

    if verdict.rejected:
        return await strong.generate(req, reason="verifier_reject")
    return candidate

The alerting follows from the metric labels. Escalation rate is now an observable proxy for verifier health, which means a dependency outage announces itself as a cost anomaly — escalation_rate > 2× the trailing 7-day baseline for 10 minutes fires within twelve minutes of 09:14 and names the reason label in the alert body. That is the property worth engineering for: the failure mode of a silent quality drop has been converted into the failure mode of an unexpected invoice, and only one of those has an alert that can be written against it.

There is one arrangement that removes the outage class entirely rather than handling it, which is to run the verifier in-process. A 1–3B classifier or a fine-tuned cross-encoder — a small model that reads the question and the candidate answer together in one pass and emits a single score, rather than embedding each separately — scoring a candidate locally has no network dependency, a p99 in the low tens of milliseconds, and no provider status page. It is weaker than a hosted small model at open-ended judgment and it never times out, and for schema-shaped or grounding-shaped checks the gap is small enough that the trade usually favours the local model.

When the escalation path's model is taken away#

The second event that week was a deprecation notice: the pinned strong-model version retires in sixty days. The tempting response is to repoint the string and carry on, reasoning that the strong model is the safe side of the cascade and a newer version can only be better.

The cascade's justification is a measured comparison against one specific model, and both of the things that comparison established are stated in terms of that model. The label from chapter 4 was a conjunction — the cheap answer scored below the bar and this strong model scored at or above it — so the definition of "needed escalation" moves when the second conjunct's subject changes. The quality gate from chapter 5 was a per-slice delta against that same arm. A newer model that is better on aggregate can be worse on a slice, which is the whole lesson of the extraction workflow, and it can also be better in a way that changes which requests are worth escalating.

That decomposition is also what makes the migration cheap, because only one conjunct moved:

Re-running one arm of a 5,000-row replay is an afternoon and the price of a takeaway lunch. The reason it usually does not happen is that nobody wrote down which conclusions depended on which pinned version, so the choice presents itself as "repoint the string" against "redo the whole evaluation", and the first option wins. Recording the pinned versions in the evaluation output, as chapter 5 required, is what turns a sixty-day deprecation into a half-day task.

Two operational habits fall out of this. Pin explicit versions on both paths, since a floating alias means the boundary between your arms moves without a notice arriving at all. And keep the replay harness runnable on demand by anyone, because an evaluation that takes a week of setup will be skipped under a deadline, which is precisely when the model on the escalation path is changing.

Deleting the dependency versus relocating the check#

The proposal that closed out the incident review was to drop the verifier and escalate on a cheap request-side classifier instead, on the grounds that one fewer provider is one fewer outage.

The case for it is real. It removes 180 milliseconds from every request, removes $0.0002 per request, removes an entire class of incident, and removes the timeout semantics that took four hours to get wrong. If the workload's failures are dominated by malformed structure and unsupported citations, the deterministic checks already catch those for free, and the verifier is earning less than its complexity.

The case against is that a classifier evaluated on the request is chapter 1's router wearing the cascade's clothes. It never sees the candidate, so nothing examines the cheap model's output, and the failure it cannot catch is the confident wrong answer — the same one the self-reported confidence could not catch, for the same reason. On the labelled set from chapter 4, the verifier caught 61 of 80 missed escalations; a request-side classifier trained on the same labels caught 24. Adopting it moves the missed-escalation rate from 1.2% back toward 3%, which at 1.2 million requests a month is around 21,000 additional wrong answers, and the measurement that would reveal this is the quarterly labelled run that the proposal does not remove.

Framed as a choice between the two, the verifier wins on the numbers. Framed as an architecture question the answer is better, because the dependency and the check are separable: run the classifier in front to send obviously-hard requests straight to the strong model without a cheap attempt, which cuts verifier volume by the escalation-certain fraction and reduces exposure to its outages, and move the remaining verification in-process so there is no provider to fail. The dependency gets removed by relocating the check rather than by deleting it, and the thing being traded away is a little judgment quality rather than the entire ability to notice a bad answer.

What to have in place before the first request is routed#

The order that keeps this honest is short. Pick one surface and one task where the cheap model plausibly suffices, and run the two-arm replay from chapter 5 on a stratified sample before writing any routing code, so you know the per-slice delta you would be accepting. Choose an escalation signal that looks at the answer, starting with whatever deterministic checks the task admits, and label 2,000 requests to get its false and missed rates before it goes anywhere near production. Name a price for a wrong answer, since the threshold encodes one whether or not it is written down. Wire the fallback to always-strong and the alert on escalation rate before the first percent of traffic, because both are unfixable during an incident. Then publish the saving and the per-slice quality delta in the same table, every quarter, and let the ratio decide which surface goes next.