Classifying provider error responses

A service that uses a hosted model talks to it over HTTP, the same way it talks to any other web API: it sends a request carrying the prompt and the parameters, and it gets a response back. A fraction of those responses are failures rather than answers, and each one arrives with an HTTP status code — the three-digit number at the top of the response — and a structured error body underneath it. Error handling is the layer that reads those failures and decides what happens next: wait and try again, give up and surface the error, or stop calling the provider for a while. It lives in the client wrapper that every service eventually grows around its provider SDK, and it is the code path that decides whether a provider's bad half-hour becomes your outage.

The first decision that layer makes is classification: which kind of failure is this. Three classes account for nearly all of the traffic, and the rest of this topic — how long to wait, how many times, whether waiting is safe at all — branches on the answer. This chapter defines the three, shows where the provider states which one you are in, and is honest about the fact that classification is a labelling step that reduces your error rate by exactly zero requests.

The three constraints and where each one lives#

A rate limit is a ceiling the provider puts on how much of its capacity one account may consume in a rolling window, measured along more than one axis at once: requests per minute (RPM) and tokens per minute (TPM) are the usual pair. The ceiling exists because the GPUs serving the endpoint are finite and shared between every customer, so without a per-account cap a single heavy tenant would crowd everyone else out. A request that exceeds the cap is rejected with HTTP status 429, "Too Many Requests". An overload is the opposite situation, where aggregate demand across all customers exceeds the provider's capacity, your own allowance is irrelevant, and the service sheds work to stay up; Anthropic reports this as 529, a status number outside the registered HTTP set that several model providers have adopted for the condition. A malformed request is a payload the API refuses to process at all — an unknown field, a value above a documented ceiling, a tool schema the validator rejects — and it comes back as 400, "Bad Request", decided before any model runs.

Two properties of that first ceiling are worth deriving rather than accepting, because both of them explain failures later in this topic. The first is why the ceiling is per account at all. The hardware is finite and shared, so something in front of it has to decide whose request runs next. A queue with no per-account meter serves whoever arrives, which means a caller that opens more connections gets more of the machine, and a caller running an unbounded loop gets nearly all of it. Metering per account is what stops the allocation being settled by whichever tenant is most aggressive.

The second is why the meter runs along two axes at once rather than one. What an account actually consumes is time on a GPU, and that time is dominated by how much text a request reads and writes rather than by the fact that a request arrived. Counting only requests would price a short prompt and an enormous one identically, and an account sending the second kind could saturate the hardware while sitting comfortably inside a requests-per-minute ceiling. Counting only tokens would leave the per-request overhead unpriced — admission, scheduling, the fixed cost of starting a generation — and an account sending a flood of tiny calls would sit inside a tokens-per-minute ceiling while doing the same damage. Two meters, checked independently, close both gaps, and either one of them alone is enough to reject a request. That is why one status code ends up carrying two unrelated conditions, which the last section of this chapter comes back to.

These are not three severities of the same problem. A 429 is a statement about your own consumption: you have an allowance and you used it, the provider has capacity, and you are simply not entitled to more of it right now. The fix is on your side — fewer concurrent workers, a queue, a higher tier, or spreading the load across the window — and time resolves it because the window rolls. A 529 is a statement about the provider's capacity. Nothing you configure changes when it clears, and the same 529 is being returned to every other customer in the same region at the same moment. A 400 is a statement about the payload: a field that does not exist, a max_tokens above the model's ceiling, a tool schema missing additionalProperties, messages that do not alternate, a parameter removed in the model generation you just upgraded to. There is no window that rolls and no capacity that returns; the same bytes will be rejected the same way in an hour.

The severity reading is tempting for a reason worth naming. Status numbers are ordered, larger numbers read as worse, and across most of the web a 5xx really does describe something more serious than a 4xx. Here the number encodes whose constraint was hit, not how bad the situation is. A 529 arriving in the middle of a capacity crunch may clear in seconds without anyone doing anything, while the 400 that reads as the mildest of the three is the only one of them that will still be failing next week. Sorting the three by their status numbers sorts them by nothing.

What makes two of the three clear on their own is a mechanism worth stating once, because every retry schedule later in this topic is built on it. The allowance is metered over a rolling window: the provider holds what the account spent across the trailing minute, and as the clock advances, spend that has aged past the far end of the window stops counting against you. Doing nothing therefore returns headroom, which is the unusual property that makes waiting a repair rather than a stall. A 529 clears by the same kind of process one level up, except that the demand ageing out of the window belongs to other customers and nothing you configure moves it. A 400 has no window and no meter anywhere in its path — the request was rejected by a validator that read the bytes — so there is nothing for the passage of time to act on.

The row that matters most for alerting is the last one. A 529 is shared: if you are seeing it, so is everyone else, and the best move is usually to wait and let the provider's own oncall handle their own incident. A 429 is yours alone, and a sustained one means something in your fleet changed. Treating both as "the provider is unhealthy" throws away the distinction that says whether there is anything for you to do at all.

That same row is why a provider's status page cannot adjudicate your error rate, which is the first place most runbooks send an engineer. A status page reports degradation of the service as a whole, which is the aggregate condition a 529 comes out of. An account that has spent its own allowance is not a degradation of anything: the service is behaving exactly as configured, for one customer, and no aggregate view will ever contain it. Checked against a wall of 429s the page can only come back green, and green then gets read as evidence that the errors must have come from somewhere else entirely.

The status line is a summary, not the message#

Reading only response.status_code is where classification quietly breaks, because the same number covers cases that need different handling and different numbers cover the same case. Every error the Claude API returns carries a typed body alongside the status, and the type is the more precise signal:

{
  "type": "error",
  "error": {
    "type": "rate_limit_error",
    "message": "Number of request tokens has exceeded your per-minute rate limit"
  },
  "request_id": "req_011CSHoEeqs5C35K2UUqR7Fy"
}

The reason the status under-determines the case is structural rather than an oversight anybody could fix. HTTP statuses are a small fixed vocabulary, standardised long before hosted model APIs existed, and an API of this kind can be in far more distinct conditions than there are numbers available to name them with. Any mapping of many conditions onto few codes has to collapse distinctions somewhere, so one number arrives carrying several causes, and occasionally one cause arrives under more than one number depending on where it was detected. The provider's answer is to leave the status as the coarse routing signal that every proxy, load balancer and HTTP library already understands, and to put the precise condition in a field of its own.

The error.type string is the field to branch on. A 403 can be a permission_error or a billing_error, and those want different alerts — one is a key scoped to the wrong workspace, the other is a card that expired. A 400 is always invalid_request_error, but the message distinguishes "your tool schema is malformed" from "your organization's data retention setting makes every request to this model invalid", and only one of those is fixed by editing the request. The SDKs expose the field directly:

import anthropic

client = anthropic.Anthropic()

try:
    message = client.messages.create(
        model="claude-opus-5",
        max_tokens=4096,
        messages=[{"role": "user", "content": prompt}],
    )
except anthropic.APIStatusError as e:
    # e.status_code is the HTTP status; e.type is the documented error type.
    log.warning(
        "provider_error",
        status=e.status_code,
        error_type=e.type,
        request_id=e.request_id,
    )
    raise

The request_id in that log line is worth as much as the classification. It is the only handle the provider can trace, and an escalation carrying twenty of them gets a different answer than one describing an error rate.

Rate limit responses carry headers as well, and they are the difference between knowing you were throttled and knowing why:

HTTP/1.1 429 Too Many Requests
retry-after: 12
anthropic-ratelimit-requests-limit: 4000
anthropic-ratelimit-requests-remaining: 0
anthropic-ratelimit-requests-reset: 2026-08-08T03:47:24Z
anthropic-ratelimit-input-tokens-limit: 400000
anthropic-ratelimit-input-tokens-remaining: 212480
anthropic-ratelimit-input-tokens-reset: 2026-08-08T03:47:36Z

Reading that block is a subtraction. The -limit header is the ceiling for the current window, the -remaining header is what is left of it, and the difference between the two is what has already been spent. On the requests axis, a ceiling of 4,000 with nothing remaining means the whole allowance is gone. On the input-tokens axis, a ceiling of 400,000 with 212,480 remaining means a little under half the budget has been spent and more than half of it is still there. The two -reset timestamps sit twelve seconds apart, which is the ordinary case rather than a symptom: they are separate meters over separate windows, and there is no reason for two windows that started at different moments to refill together.

Requests and tokens are consumed by different things, and that is what makes the pair informative in a way neither meter is alone. Every call spends exactly one unit of the request budget whatever it carries, so that meter is counting callers. The token meter is counting how much text those callers sent. Reading the two side by side therefore separates how many from how much, and both of those are facts about your own fleet rather than about the provider.

Those two blocks say something the status code cannot: the request allowance is exhausted and the token allowance is only half-used. That points at concurrency rather than at prompt size, and it points there in one glance. The reverse pattern — requests remaining, input tokens at zero — points at a caller sending very large prompts, and the remediation is completely different. What the reset timestamps are worth once you have them is the subject of a later chapter in this topic; the point here is that they exist, and that a classifier reading only the first line of the response throws them away.

Where the taxonomy leaks#

Three categories with clean boundaries is the version that fits in a table. Production has edges, and the most common one is that not every failure has a status at all. A DNS failure, a TLS handshake timeout, a connection reset mid-request, or a client-side timeout produces no HTTP response, and the SDKs surface these as a distinct exception class — APIConnectionError in Python and TypeScript, an AnthropicIoException in Java. A classifier written as a switch over status codes has no branch for these and falls through to whatever the default is. If the default is "retry", you are retrying requests whose outcome you do not know, which is the subject of the last chapter of this topic. If the default is "fail", you are failing on blips that a single retry would have absorbed.

The belief that makes the first default attractive is that a call which did not come back is a call that did not happen, so trying again is free. The second half does not follow from the first. A connection reset or a client-side timeout tells you that the client stopped hearing about the request. It says nothing about whether the provider received it, ran it, and is at that moment writing an answer into a socket nobody is listening on. That is a materially different position from a 429, where the provider is explicitly telling you it declined to run the request. What to do when the outcome is genuinely unknown is the subject of the last chapter of this topic; what matters here is that "no status" is a fourth class with properties of its own, not a milder shade of one of the three.

The second edge is that intermediaries lie. If your traffic goes through a corporate proxy, an API gateway, a service mesh sidecar, or a CDN, any of them can synthesise a 502, 503 or 504 that has nothing to do with the provider. Those responses do not carry the provider's error body, and the giveaway is usually that the body is HTML rather than JSON. A classifier that assumes any 5xx came from the model provider will happily attribute your own mesh's connection-pool exhaustion to provider overload, and you will spend an afternoon reading someone else's status page.

The HTML body is not a coincidence, and knowing why makes it a dependable tell rather than a rule of thumb. An intermediary that fails before it reaches the provider has no provider response to relay, so it must compose one itself, and what it composes is whatever its own error handling produces — a gateway's default page, in the format gateways default to. The typed JSON body in the block above can only be produced by the provider, because only the provider knows which of its documented error types applies. A response whose body does not parse into that shape is therefore evidence about who answered, which is a different question from what went wrong. A classifier that branches on error.type gets this check without paying for it: the field is absent, so whatever sent this was not the provider.

The third edge is that 400 is not a single fault class. A request rejected because the payload was built wrong is permanent. A request rejected because it exceeds a limit that depends on the conversation — the context window filled up as the history grew — is also permanent for that exact payload, but the caller can do something about it: compact, truncate, or split. Both are correctly non-retryable and both are correctly your problem, but one is a bug and the other is a workload characteristic, and lumping them together means the deploy that introduced a real schema bug hides inside a metric that is normally non-zero.

The context-window case is the one that defeats the reflex, and it is worth following through. The payload is well-formed. Every field in it is one the API documents, every value is of the right type, and the code that built it has no defect. What failed is the interaction between a correct request and a ceiling that depends on how much conversation has accumulated behind it, which means the same code path emits valid requests for weeks and an invalid one the moment a thread gets long enough. A team that reads every 400 as "somebody shipped a bug" therefore goes looking at the most recent deploy, finds nothing wrong with it, and loses the afternoon. The deploy that made this possible may be months old, and there may not have been one at all — the input that crossed the line was written by a user.

Finally, a 429 can mean two different things behind one status. The requests-per-minute allowance and the tokens-per-minute allowance are separate budgets that refill on different timescales, and both report the same status code. A later chapter takes that apart, because a retry policy that treats them identically works for one and fails reliably for the other.

Routing by failure class#

Classification is a labelling step. It does not reduce your error rate by a single request, and it is worth being clear about that before it gets sold internally as a fix. It does not solve capacity planning: knowing that 331 of 412 failures were 429s says you are over your allowance, not whether the answer is a higher tier, fewer workers, a queue with admission control, or moving a batch job to a window where interactive traffic is quiet. It is not the same thing as a retry policy either, though the two are usually written in the same file — a classifier wired to a policy that retries everything is exactly as harmful as no classifier at all, and just produces better logs while doing the damage. It also does not survive without maintenance, because providers add error types, deprecate parameters, and change which conditions produce which status. A classifier with a hard-coded list of retryable statuses written in 2024 will, sooner or later, treat a newly introduced retryable condition as fatal or a newly permanent one as transient. Branching on the documented error.type rather than on the status number narrows that exposure, since the type is the field the provider documents as stable.

What classification does buy is the ability to route, and routing is what turns one undifferentiated error rate into three signals with three owners and three appropriate responses. Of the two transient classes, the 429 is the one that deserves the three-in-the-morning page, because it is the one where somebody on your side can act; a 529 at that hour usually means waiting is the correct response, and waking a human to wait is a cost with no benefit.

The label is the input to every decision in the rest of this topic, which is why it comes first. Knowing that a 400 is permanent already tells you not to retry it, and that is the easy half of the answer, because the correct number of attempts is zero and no schedule can improve on it. What it does not yet tell you is what a retry of a permanent error costs when a wrapper does it anyway. That number is larger and stranger than the token bill suggests, and it is what the next chapter measures.