Classifying provider errors

A circuit breaker is a piece of shared state in front of a provider that counts recent failures and, once too many arrive in too short a window, stops sending that provider traffic for a cooldown period so requests skip to the next entry in the fallback chain — the ordered list of places a request can be served from. Both mechanisms consume a single input: a verdict, per response, about whether that response was a failure. This chapter is about producing that verdict, which is usually a single comparison against status >= 500 and which decides whether the rest of the machinery fires on a vendor's bad hour or on your own bug.

Three questions, not one status code#

Would sending this exact request again, to this same provider, plausibly succeed? Would sending it to a different provider plausibly succeed? And does this response tell you anything about whether the provider is healthy for other requests?

The reason a single status code cannot answer all three is structural rather than accidental, and it is worth following because it says which field you should be reading instead. What each of the three questions is really asking about is where the fault lives: in this particular request, in your account, or in the provider's fleet. Status codes were designed to carry a much coarser version of that — 4xx meaning the caller sent something wrong, 5xx meaning the server failed — and that division was drawn for a web server holding documents, not for a metered API in front of a shared cluster. There is no HTTP code for "you are over the limit you bought", and none for "we are over the limit we have", so a provider is obliged to send the same 429 for both. The information you need was not deleted; it was never expressible in the field you are looking at.

A single status code therefore cannot answer all three, which is why a resilience layer that branches on status >= 500 is making one decision where three are needed. A 400 answers no to all three: the request is wrong, it will be wrong at the secondary, and the primary is fine. A 500 usually answers yes, yes, yes. A 429 answers yes-after-a-delay, then it depends, and usually no to the third — because the most common 429 is your own organisation's rate limit, which is a statement about your traffic rather than the provider's condition. Failing over on that one does not route around a fault; it takes the load that exceeded your primary quota and points it at a secondary account that is typically provisioned at a fraction of the primary's limits, so the overflow arrives somewhere less able to absorb it.

The sizes make that worse than a coin flip, and the reason is worth working through once. The traffic that fails over is not a share of your load, it is the excess: precisely the part that your primary quota — the larger of the two ceilings, because it was bought for production — could not accommodate. That excess is then presented to an account whose limit was set when somebody opened it for a proof of concept. The secondary rejects it with a 429 of its own, which your classifier is about to read as another provider failure, and the chain runs out. The general shape is that a limit attached to your account moves with your traffic and stays put when the provider's health changes, so a mechanism built to route around provider faults has no useful response to it at all.

Content filtering is the case that catches teams who have only ever thought about infrastructure failures. When a provider refuses a completion on safety grounds, the refusal is deterministic for the same input under the same policy version, so a retry produces the identical refusal, and a failover produces a refusal from the second vendor's classifier — with the second vendor's tokens billed and its quota spent. One document that trips a filter can, in a system that counts refusals as failures, walk the entire chain and end with an exhausted-chain error, having paid three vendors to say no.

The word doing the work there is deterministic, and it is the property that separates a refusal from every other non-200 in this chapter. A capacity failure is a fact about the provider at a moment in time, so the same request a second later meets a different fleet and may well succeed. A refusal is a verdict computed from the request itself. Nothing about waiting, sleeping, or asking a different machine changes the input, so — for as long as the policy behind the classifier stays where it was, which is a schedule the vendor owns and does not announce per request — nothing about it changes the answer. Retry and failover are both mechanisms for varying when and where a request is asked, and a fault that depends on neither is invisible to both — which is also why the same reasoning covers a malformed body, a request that exceeds the context window, and every other error that travels inside the request rather than waiting at the destination.

The two rows at the bottom are the ones that get missed. A stream that dies at token 900 of an expected 1,100 has cost you the full generation, and a blind retry pays for it twice while doubling the load on the thing that just dropped you; it belongs in the breaker's counts but not in the retry path. A 200 that returns prose where you asked for JSON is a model behaviour rather than a health signal, and treating it as a provider failure will trip a breaker on a hard prompt rather than on an outage.

Classify by error body, and admit the ambiguity#

Status codes are overloaded, and vendors overload them differently. A 429 might be your org quota, a per-model capacity ceiling, or fleet-wide load shedding, and the three are distinguished only by an error type string in the body and sometimes by headers that are not part of any contract. A 503 might be a genuine outage or a deliberate rejection during a deploy. Providers add new error types without a version bump, so any classification map you write is out of date on some schedule you do not control.

That argues for a specific structure: an explicit table for the cases you know, a defensible default for the ones you do not, and an alert on the default so the table gets maintained.

class Verdict(NamedTuple):
    retry: bool
    counts_as_provider_failure: bool
    failover: bool

KNOWN = {
    ("429", "rate_limit_error"):     Verdict(True,  False, False),
    ("429", "overloaded_error"):     Verdict(True,  True,  True),
    ("403", "content_filter"):       Verdict(False, False, False),
    ("401", "authentication_error"): Verdict(False, True,  True),
    ("400", "invalid_request_error"):Verdict(False, False, False),
}

def classify(status: int, error_type: str) -> Verdict:
    known = KNOWN.get((str(status), error_type))
    if known:
        return known
    metrics.increment("provider.unclassified_error",
                      tags=[f"status:{status}", f"type:{error_type}"])
    # 4xx defaults to our fault: do not move it, do not blame the provider.
    # 5xx defaults to their fault: count it and let the chain move.
    if 400 <= status < 500:
        return Verdict(False, False, False)
    return Verdict(True, True, True)

The unclassified_error counter is the load-bearing line. Without it the map silently rots and the defaults quietly become the policy; with it, a new vendor error type shows up on a dashboard within a day of its first appearance and someone adds a row.

One structural note about where this code sits. The classification map is per vendor, because the status-plus-type tuples are vendor-specific, which means the adapter layer has to surface the vendor's error type rather than flattening everything into a generic exception with an HTTP status. An abstraction that hands the gateway ProviderError(status=429) and nothing else has thrown away the only field that distinguishes "you are over your quota" from "we are shedding load", and no amount of tuning downstream recovers it. Adding a new vendor to the chain therefore includes adding its classification map, and a vendor shipped without one runs entirely on the defaults.

Turning slowness into a countable failure#

The taxonomy above is built on responses, and the most damaging provider incidents produce no response at all for a long time. Vendors shed load in two ways: some reject, which gives you 429s and 503s to count, and some queue, which gives you 200s that arrive 45 seconds later. Under the second kind, a breaker fed only on status codes sees a perfectly healthy provider while your worker pool fills with requests that are each holding a slot for twenty times their normal duration, and the first thing that breaks is an unrelated endpoint that could not get a thread.

The step from "the provider is slow" to "the breaker cannot see it" is short and worth making explicit, because it is the reason a timeout belongs in this chapter at all. Everything the breaker knows arrives as a verdict about a response. A request that has not come back yet has produced no response, so it has produced no verdict, so it has incremented no counter. Slowness is not a quiet signal to the state machine; it is the absence of any signal, and the counters cannot distinguish a provider that is taking forty-five seconds from a provider that is merely not busy. Meanwhile every one of those pending requests is holding a worker, which is a cost being paid in a currency the breaker does not measure.

Timeouts are what convert slowness into something the classifier can see, which makes the timeout part of the failure-detection design rather than a detail of the retry policy. What a timeout does, precisely, is manufacture a response at a moment you choose, so that a request which would otherwise never produce a verdict produces one on your schedule. Set it from the measured distribution rather than from a round number: if p99 for this route is 4.1 seconds, a 12-second per-attempt timeout is generous enough not to fire on ordinary variance and tight enough that a queueing provider is detected in seconds.

The two directions that setting can be wrong in are not symmetrical, which is what the measured distribution is protecting you from. A timeout below the real latency distribution fires on healthy requests, so you pay for a generation, throw it away, and then feed a fabricated failure into the breaker's counts — the breaker is now opening on your configuration rather than on the vendor. A timeout far above the distribution never fires on a healthy request and also never fires soon enough to be informative, so the worker pool drains while the counters stay clean. Sitting at roughly three times the measured p99 puts the threshold in a region that essentially no healthy request reaches and that a queueing provider reaches immediately, and the only way to know where that region is for a given route is to have measured it.

Streaming needs two timeouts instead of one, because a 90-second stream is normal for a long generation while a stream that has produced no token for 10 seconds is not — a time-to-first-token limit around 5 seconds, measured from sending the request to the arrival of the first fragment of the answer, and an inter-token stall limit around 10 seconds covering the gaps between fragments after that, together catch the hang without truncating legitimate long outputs.

Several breaker implementations also support a slow-call rate as a second trip condition, counting any call above a latency threshold as a partial failure and opening when that ratio crosses its own threshold. Where it is available it is worth configuring, because it opens the breaker on a degrading provider before the timeouts start firing, which is typically 20 to 40 seconds earlier. Where it is not, a timeout that is tight enough to be informative is the substitute, and its value goes into the same counters as everything else.

The four numbers that define the open state#

Once you know which responses count, the breaker itself is a small state machine with four parameters.

The window needs a minimum call count as well as a duration, because a rolling 30-second window that has seen three calls can hit 66% on two failures that mean nothing. Twenty calls is a common floor. The threshold wants to sit well above your steady-state error rate: providers run a baseline of transient 5xx in the 0.2–0.5% range, so a 5% threshold is an alarm rather than a decision, and 50% over a populated window is a claim you can defend. The cooldown is how long the primary is left alone, and 20 to 30 seconds is a reasonable start because provider incidents rarely resolve faster than that and your fallback is paying for every second of it. The half-open probe budget decides how you find out whether it is over: allow a handful of requests through, require most of them to succeed before closing, and on failure re-open with a doubled cooldown up to a cap of a few minutes. Without that backoff on the open state, a provider that is failing 60% of calls will produce a breaker that flaps open and closed every twenty seconds, which is the worst of both worlds because a third of your traffic keeps paying the full failure path.

One property of the threshold is easy to miss and shows up in the scenario below. A failure rate is a fraction, and the four numbers above constrain only its numerator. The denominator is every call that shares the breaker's key, so the rate can cross the threshold with the provider's behaviour completely unchanged, purely because the mix of traffic under that key moved toward whatever is failing. An endpoint failing a fixed share of its own calls contributes more to a vendor-wide rate on an afternoon when that endpoint is a larger part of the vendor's traffic. The graph rises, the breaker opens, and nothing at the vendor did anything. Reading a rate as a measurement of the provider only works when the denominator is stable, and traffic mixes are not.

Two scoping decisions matter as much as the numbers. A breaker keyed only by vendor conflates a specific model's capacity problem with the vendor being down, so key it by the tuple you actually route on — provider, model, and region — and you can fail one model over while the rest of the account keeps serving. And a breaker's counters live somewhere. Per-process counters on a fleet of 40 pods handling 300 requests per minute give each pod 7.5 requests per minute, which is under 4 calls in a 30-second window, so the minimum-volume floor is never reached and no breaker ever opens no matter how badly the provider is failing.

That arithmetic is three divisions and every team gets it wrong the same way, so it is worth stepping through. Traffic is measured for the service and the counters are kept per process, which means the rate each counter actually sees is the fleet's rate divided by the number of processes: 300 requests a minute across 40 pods is 7.5 a minute at any one of them. The window is half a minute, so halve that again, and each pod's rolling window contains under 4 calls. The floor of twenty exists to stop two failures out of three from looking like a catastrophe, and here it is never approached at all — the condition for opening is a conjunction, and one of its terms is permanently false. Every pod is independently certain it has not seen enough evidence, while the fleet as a whole has seen a great deal. The load-testing environment, where one pod carries all 300 requests a minute, passes the exercise and proves nothing about production.

Either share the counts through Redis, or scale the window and volume floor to per-instance traffic, and check that arithmetic against your real pod count rather than against the one machine you tested on.

The belief that produced that Thursday is one almost every team holds, and it is worth stating in the form somebody would defend it. It is that a breaker protects the system from provider failures, so installing one makes provider failures a solved category. The first half is true and the second half does not follow, because the breaker has no access to the thing the sentence is about. It cannot tell a provider failure from any other non-200; it can only count the responses your classifier hands it and compare that count against a threshold. Point it at a set of responses that includes your own bugs and it will faithfully protect you from your own bugs, in the only way it knows, which is by moving them somewhere else. A breaker is a fast, consistent executor of a judgement made somewhere upstream, and the judgement is the map.

A breaker used as a rate limiter#

Using a breaker to shape traffic is tempting because both mechanisms stop requests. A rate limiter shapes healthy traffic to a known ceiling and should engage constantly; a breaker is an assertion that something is broken and should engage rarely. If your breaker opens every afternoon at peak, it is telling you that you are over quota and you have built a very expensive rate limiter that also disables your primary provider for twenty seconds at a time.

Nothing in the state machine reads the answers#

The counters know whether a response came back, and nothing more. A provider that has silently switched to a degraded serving tier and is returning 200s full of shorter, worse completions will keep the breaker closed all day, because every one of those responses is a success by the only definition the state machine has.

The four numbers are the part of a breaker that gets tuned in a retrospective, and the classification is the part that decides whether tuning them was ever the problem. Write the map per vendor, default 4xx to yourself and 5xx to them, count what the map says to count, and alert on what it does not recognise.

What the state machine does decide is when traffic moves, and it moves that traffic to whatever the chain names next. In the Thursday failure the chain was correct and the classification was wrong. The next failure mode is the reverse: the classification is right, the breaker opens exactly when it should, and the chain it consults turns out to be a constant compiled into the binary that nobody can change at two in the morning.