Error classification across providers

Five conditions account for nearly every provider failure worth distinguishing: a malformed request body, a prompt longer than the model's context window, a generation stopped by a content filter, an exhausted account quota, and ordinary rate limiting or backend overload. They call for completely different responses — wait and retry, shrink the input, tell the user, page a human, fix a bug — and they arrive over the same HTTP connection wearing whatever status code each vendor chose for them.

The trouble is that the vendors chose differently, and that some conditions are not reported in the status at all. A retry policy keyed on the status alone, of the kind every client library ships and every hand-written wrapper copies, will therefore retry a condition that can never succeed on one provider while treating it correctly on another, using identical code. This chapter lays the five conditions against two vendors' reporting, builds a classification defined by what your code does next rather than by the provider's vocabulary, and then looks at what the typed exception hierarchy a client gives you carries — which is exactly the information the status code carried, and no more.

Five conditions, two vendors, no agreement#

Before reading the rows it is worth knowing why they could not have come out the same, because the disagreement is structural rather than careless and that changes what you should expect from it. HTTP statuses are a small, fixed vocabulary, standardised long before any of this existed and designed to describe what happened to a document request. The conditions in the left column are not document conditions. A prompt exceeding a context window, a generation halted by a classifier, and an account with no credit are three situations with no established status apiece, so each vendor picks the status that seems least wrong — and "least wrong" is a judgement call made by different people at different times.

That is why this is not a gap anyone is going to close. Neither vendor is misusing HTTP; both are compressing a larger set of conditions into a smaller set of codes, and compression loses information. The information does not vanish, it moves: into a type string, a code, a message, a stop reason. Which is the whole argument of this chapter stated as a mechanism rather than as advice — the status is what survived the compression, and the body is where the rest of it went.

Read the quota row and the rate-limit row together, because they are the pair that breaks status-based logic. On one provider those two conditions have different statuses, so a status-based rule separates them correctly and a client's built-in retry policy leaves the terminal one alone. On the other they share a status and are distinguished only by a string inside the body, which no status-based retry policy ever looks at. The identical code, pointed at the two providers, retries a hopeless request on exactly one of them.

It is worth granting that the provider giving both conditions a 429 has a case, because dismissing it leaves you with a rule you will misapply elsewhere. From the serving side the two situations are the same situation: a request arrived, and it will not be served. Whether waiting would help is a fact about your account's future, not about the request, and the status code is a statement about the request. Retryability, in other words, is not a property the protocol was ever designed to express — it is an inference you draw from knowing which condition occurred, and the condition is in the body. A rule reading "retry 429" has quietly promoted one provider's convention to a law of the industry, and the uncomfortable part is that it is right often enough to survive years of production before the day it is not.

The content-filter row is stranger, because on one path it is not an error at all. A content filter is the provider's own check on what is being generated, applied separately from any question of whether your request was well-formed, and it can stop a generation the model was otherwise willing to produce. A 200 with a normal-looking response object, an empty or truncated text field, and a stop reason indicating filtering will pass straight through any error handling you have, get stored as an assistant turn, and be replayed to the model on the next request as though the assistant had genuinely said almost nothing. The bottom row has the same property for a different reason: once a streaming response has sent its headers the status is committed at 200, so every failure after that point has to travel inside the body.

There is a defensible reason for that 200 as well, and it is the same reason stop_reason exists at all. A 4xx status says the request was wrong. A filtered generation involves no wrong request: the body was valid, the model was available, the work was scheduled and performed, and something about the output caused it to be stopped partway. Reporting that as a client error would be a false statement about your request, and reporting it as a server error would be a false statement about the provider. What actually happened is that a generation ended for a particular reason, and there is already a field whose entire job is to say why a generation ended. So the condition goes there — correctly, and invisibly to every handler that treats a status as the answer.

Filling in a table like this for your own providers is an exercise in deliberately causing each condition, which takes about an hour and is worth doing once per provider. A 200,000-word string provokes the context overflow. A model identifier with a typo provokes the not-found case. A body carrying "messages": "hello" instead of an array provokes the schema error. Rate limiting can be provoked with a burst of concurrent requests on a low-tier key. Quota exhaustion is the one condition you generally cannot provoke on purpose, which is why it is also the one that reaches production unhandled — the practical substitute is to read the provider's error reference for the code it uses and write the branch from documentation, accepting that it is the one branch your tests do not cover.

The classification your code actually needs#

The taxonomy you want is not the provider's. It is a small set of buckets defined by what your service does next, and the useful shape has five of them.

The shape of that flowchart carries the argument: the status code is the first discriminator and never the last one. Every path either reads a type string, an error code, a message, or a stop reason before it reaches an action, and the two branches that look most alike from the outside — a 429 that will succeed later and a 429 that never will — are separated by a single string in the body.

Each bucket has one action attached, and the actions are what make the distinctions worth drawing. Backoff-and-retry is the only bucket where a second attempt has any chance of succeeding. The billing bucket must page a human, because no amount of code fixes it and every second of delay is complete unavailability. The shrink-input bucket is recoverable inside the request: a batch job that hits it can chunk the document and continue. The bug bucket is the one that should never appear in production, so its rate is a meaningful alert. The user-visible bucket needs a message rather than a retry, because the system is working correctly and the answer is that the request will not be served.

Written against status and type strings rather than against exception classes, this classification is about forty lines and moves between providers by changing a lookup table.

from enum import Enum

class Bucket(Enum):
    RETRY = "retry"; QUOTA = "quota"; INPUT_TOO_LONG = "input_too_long"
    BAD_REQUEST = "bad_request"; USER_VISIBLE = "user_visible"; OK = "ok"

def classify(status: int, body: dict) -> Bucket:
    err = body.get("error") or {}
    kind = err.get("type") or ""
    code = err.get("code") or ""
    text = (err.get("message") or "").lower()

    if status == 200:
        stop = body.get("stop_reason") or _finish_reason(body)
        return Bucket.USER_VISIBLE if stop in {"refusal", "content_filter"} else Bucket.OK
    if status in (401, 403) or code == "insufficient_quota" or "credit balance" in text:
        return Bucket.QUOTA
    if status in (429, 500, 502, 503, 529):
        return Bucket.RETRY
    if status in (400, 422):
        if code == "context_length_exceeded" or "context window" in text or "too long" in text:
            return Bucket.INPUT_TOO_LONG
        return Bucket.BAD_REQUEST
    return Bucket.BAD_REQUEST

The buckets are also the right unit for your alerting, which is the reason to define them in terms of actions rather than in terms of the provider's vocabulary. A rise in the retry bucket is a capacity signal and should feed a dashboard. A single event in the quota bucket is a page. A rise in the input-too-long bucket is a change in the shape of your traffic and usually means someone upstream started sending bigger documents. A single event in the bad-request bucket is a bug you shipped. Those four alerting policies are completely different from one another and completely undifferentiated if the only thing you record is that the provider returned a 4xx.

Matching on message substrings is unpleasant and it is what the providers leave you when a condition has no dedicated code. Treat those lines as the fragile part: pin them with a test that sends an over-long prompt to each provider and asserts the bucket, so that a wording change surfaces as a failing test rather than as a batch job reporting the wrong cause.

Two cases fall outside the function's signature and need their own handling. A connection error or a client-side timeout produces no status and no body at all, so it cannot be classified by inspecting a response; it belongs in the retry bucket with the caveat that a timeout on a request the provider may have completed is not safely repeatable for anything with side effects. And a failure that arrives inside a stream has a 200 on the outside and an error object on the inside, so the classification has to be applied to the payload of the error event rather than to the HTTP exchange — the same function works, called with the status you would have expected rather than the one you got.

Whatever the classification decides, log it next to the request identifier the provider returns in its response headers. A structured line carrying the bucket, the status, the provider's type string and the request id turns the whole taxonomy into something queryable, and the first question after any provider incident — how many of these were retryable and how many were terminal — becomes a group-by rather than a reading exercise across thousands of stack traces.

What the SDK's exception hierarchy gives you, and what it does not#

Both major Python clients raise a typed exception hierarchy: BadRequestError, AuthenticationError, PermissionDeniedError, NotFoundError, ConflictError, UnprocessableEntityError, RateLimitError, InternalServerError, all under a common APIStatusError, alongside APIConnectionError and APITimeoutError for failures with no response. The two hierarchies are nearly identical, which is convenient and slightly misleading, because the similarity comes from both being generated by the same toolchain rather than from the providers agreeing about anything.

Understanding where that hierarchy comes from tells you the ceiling on what it can ever contain. These clients are generated from a machine-readable description of the API, and in that description an error response is typed by its status: one schema for a 400, another for a 429. The generator turns each of those into a class. What separates a context overflow from a malformed array is a string inside the 400's body, and the description carries no schema for the values that string can take — it is prose written for a human reading a message. A generator has nothing to build a class from, so no class exists. The hierarchy is not shallow because somebody skimped; it is exactly as deep as the material it was generated from.

The hierarchy is a mapping from status code to class name, so it carries exactly the information the status code carried. BadRequestError is raised for a context overflow, a malformed message array, an unsupported parameter and a temperature out of range without distinguishing them, and RateLimitError is raised for both throttling and an exhausted account on the provider that gives them the same status. Everything the classification above separates has to be recovered from exc.status_code, exc.body and exc.response.headers, which the exception objects do carry. The practical consequence is that classify should take a status and a body — both of which an exception hands you — rather than being written as a chain of except clauses.

That also keeps one implementation working whether the response arrived through the client or through a raw HTTP call you made yourself, which matters as soon as some part of your traffic takes the second path. A team that has written its handling as except RateLimitError has coupled its error semantics to the library, so adding a single hand-made call means either duplicating the logic against status codes or wrapping the raw path in fake exceptions to satisfy handlers that expect them.

Two clarifications are worth making explicit, because both are habits rather than mistakes. Retryability is not a property of a status code; it is a property of a condition, and the mapping between them is per-provider — a rule expressed as "retry 429" is a rule about one provider's conventions that happens to be written in a vocabulary that looks universal. And a 200 is not a success, which is the more expensive of the two to unlearn, because the filtered response and the truncated stream both arrive with one.

The general form of that second habit is worth carrying past this chapter, because it explains three separate failures already met and one still to come. A status code reports the fate of an HTTP request: it was understood, it was authorised, it was routed, an answer was produced. It has never reported the fate of the work. Those two coincide in most APIs, because in most APIs the work is either done or refused. Here they come apart routinely — the request succeeds and the generation is filtered, the request succeeds and the stream dies halfway, the request succeeds and the answer stops at a ceiling — and every one of those is a 200 with the disappointing news somewhere in the body. Reading the body is not defensive programming. It is the only place the outcome is written down.

Classification, then, is the piece of provider integration that most repays being written once and written properly. It is forty lines, it takes a status and a body rather than an exception, its output is a small set of actions your service already knows how to perform, and it is the thing that makes the difference between a retry policy that is correct on one vendor by coincidence and one that is correct on both by construction.

There is one more condition that arrives on a 200 and is not an error at all, and it appears in no table above because no error object is involved. The request is accepted, the model runs, the response is well-formed, and the answer simply stops earlier than you wanted. The field that says so is one your classification code has already been reading for a different purpose, and reading it properly is what separates a plumbing fault from a genuine difference in model behaviour.