Four ways a generation fails, four different answers

The worker had one decorator on it, and the decorator was the standard one the team used everywhere else:

@retry(attempts=3, backoff=exponential(base=2), on=Exception)
def run_generation(job):
    return image_client.generate(prompt=job.prompt, **job.params)

For most of the services it wrapped, that was correct. For the image provider it turned every prompt that mentioned a musician by name into three requests, each of which was declined for the same reason in the same 900 milliseconds, and then a fourth when the user pressed the button again. One product-template bug — a caption field being interpolated straight into the prompt — put roughly 40 such prompts a day into the queue. The monthly line item for generations that never produced an image was small in absolute terms, around $70, and it was the diagnostic that mattered: 4,800 requests a month were being answered with the same deterministic "no" and nobody was reading it.

The premise the decorator encodes is that a failure is a failure and that time heals some of them. Image generation breaks in four ways that share nothing except the shape of a stack trace, and the correct response to each is different enough that collapsing them costs money in one case, corrupts data in another, and produces a user experience that is simply wrong in a third.

A refusal is an answer, not an error#

A safety refusal is deterministic with respect to the prompt. The same prompt and the same model version produce the same decision, so an identical retry is a request whose outcome you already know. Whether it is billed depends on where in the pipeline the decision was made: a prompt-stage rejection usually costs nothing but a request slot, while a refusal issued after the image was rendered and then withheld is a generation the provider performed, and several providers bill it. Even in the free case the attempt consumes rate-limit budget shared with the requests that would have succeeded, and it holds a user in a spinner for the duration.

The wire shape is a 400 rather than a 5xx, which is the signal that separates this class from the next one:

{
  "error": {
    "type": "invalid_request_error",
    "code": "content_policy_violation",
    "message": "Your request was rejected as a result of our safety system.
                Your prompt may contain text that is not allowed."
  }
}

That message is all you get, and it does not name the offending span. Passing it through to the user produces a support ticket, because "your prompt may contain text that is not allowed" reads as an accusation with no remedy attached. The useful response names the category and gives the user the substitution to make. Where you can identify the trigger — and for a bounded set of categories you can, with a cheap check of your own over the prompt before you send it — say so directly:

REFUSAL_HINTS = [
    (NAMED_PERSON_RE, "Prompts naming a real person are declined. "
                      "Describe them instead, e.g. 'a woman in her forties "
                      "in a lab coat' rather than the name."),
    (BRAND_RE,        "Prompts naming a brand are declined. Describe the "
                      "object: 'a red running shoe with a white sole'."),
    (GORE_RE,         "This request was declined for graphic content. "
                      "Remove the injury description and try again."),
]

def explain_refusal(prompt: str) -> str:
    for pattern, hint in REFUSAL_HINTS:
        if pattern.search(prompt):
            return hint
    return ("This request was declined by the safety system. Rephrasing "
            "in plainer, more descriptive language usually resolves it.")

The list is maintained by reading your own refusal log, which is the second thing this class needs. Refusals cluster: a healthy baseline in a marketing tool is somewhere around 0.5–1% of generations, and when the rate rises the cause is nearly always a single template or a single feature rather than a change in user behaviour. The template bug in the opening paragraph accounted for 31% of that month's refusals and was invisible until refusals were grouped by the code path that produced them rather than counted in total.

The class that retries are for#

Transport failures are the ones the decorator was written for and the only class where an identical retry has a real chance of a different outcome. They arrive as 500s, 502s, connection resets, TLS handshake failures, and 429s, and they also arrive as responses that were structurally fine at the HTTP layer and unusable above it: base64 that does not decode, a body that ends mid-stream, a Content-Type of application/json containing an HTML error page from an intermediary, or a PNG of 0 bytes.

Decoding the payload before you declare success is what turns the second group from a corrupt asset into a retryable failure. A worker that writes whatever it received into storage and marks the job succeeded produces a broken image in a customer's document and no error anywhere.

def decode_or_fail(payload: dict) -> bytes:
    raw = base64.b64decode(payload["data"][0]["b64_json"], validate=True)
    if len(raw) < 1024:
        raise MalformedResponse(f"suspiciously small: {len(raw)} bytes")
    img = Image.open(io.BytesIO(raw))
    img.verify()                       # raises on truncated or corrupt data
    return raw

Three attempts with exponential backoff and full jitter is a reasonable policy for this class. Full jitter means each attempt sleeps for a random interval drawn from zero up to the current backoff ceiling rather than for the ceiling itself, so two workers that failed in the same second do not wake in the same second either. That policy comes with one exception: a 429 carries Retry-After, and honouring that header beats any backoff curve you invent, because it reflects when your quota actually refreshes. Backoff without jitter across a fleet of workers reconverges the retries into the same second and reproduces the overload that caused the 429 in the first place.

The timeout is the ambiguous one#

A timeout is not a failure report. It is the absence of one, and the request it refers to may have succeeded, may still be running, and may already have been billed. Reissuing it blindly is how the duplicate-generation problem from the first chapter comes back through a different door: idempotency at the API edge protects against a client retrying, and it does nothing about a worker that timed out on the provider call and calls again.

What makes this tractable is capturing the provider's own job identifier before waiting on the result, where the API offers an asynchronous mode, and reconciling against it. The sequence is short and worth being explicit about:

def run_with_reconciliation(job):
    if job.provider_job_id:
        remote = image_client.retrieve(job.provider_job_id)
        if remote.status == "succeeded":
            return decode_or_fail(remote.payload)     # never re-billed
        if remote.status in ("running", "queued"):
            raise StillRunning(job.provider_job_id)   # requeue, do not resend
        # remote failed: fall through and classify it properly

    submitted = image_client.submit(prompt=job.prompt, **job.params)
    db.image_jobs.update(job.id, provider_job_id=submitted.id)
    return image_client.wait(submitted.id, timeout_s=90)

Where the provider offers only a synchronous call with no retrievable handle, the honest position is that a timeout leaves you unable to distinguish a lost result from a lost request, and the mitigation is to make the timeout long enough that it is genuinely rare. Setting the worker timeout at 90 seconds against a p99 near 60 puts the ambiguous case in the fraction of a percent, where a duplicate generation is an acceptable cost. Setting it at 20 seconds to "fail fast" moves roughly a tenth of your traffic into the class you cannot reason about, and every one of those is a paid generation thrown away.

The failure with a 200 on it#

The fourth class produces no exception at all. The request succeeded, the bytes decode, the file is a valid 1024×1024 PNG, and the image has garbled text on the sign it was asked to render, or a hand with six fingers, or a composition where the product is cropped at the edge, or a colour palette nothing like the brand. Nothing in the pipeline can tell, because from the pipeline's point of view this run is indistinguishable from the successful ones.

Calling this a failure at all is a category choice worth making deliberately. It is a quality outcome, and the only reliable judge is the person who asked. That has three consequences for the design. The user needs a regenerate control, because the alternative is that they leave. The regenerate has a price attached, since it is a fresh billed generation at full rate. And the rate at which people press it is a metric about the model and prompt rather than about the infrastructure.

Some of this class is mechanically detectable and worth catching, because these cases are cheap to check and embarrassing to ship. An aspect ratio that does not match what was requested, an image whose pixels are 98% one colour, or a mean luminance under 8 all indicate a run that went wrong in a way no user should be shown. That check is worth doing automatically and, unusually for this class, is worth retrying once, since it is closer to a malformed response than to a matter of taste.

The four, side by side#

The mapping from provider response to class belongs in one function rather than scattered across the worker, because the classification is the policy:

def classify(exc) -> str:
    if isinstance(exc, APIStatusError) and exc.status_code == 400:
        if exc.code in ("content_policy_violation", "moderation_blocked"):
            return "refusal"
        return "bad_request"            # our bug: bad size, bad params
    if isinstance(exc, (APITimeoutError, ReadTimeout)):
        return "timeout"
    if isinstance(exc, (APIConnectionError, MalformedResponse)) \
       or getattr(exc, "status_code", 0) >= 500 \
       or getattr(exc, "status_code", 0) == 429:
        return "transport"
    return "unknown"

Each class also wants its own counter, and the counters are more useful as ratios than as totals. Refusals as a proportion of submitted prompts, grouped by the feature that produced them, is the signal that found the template bug. Transport failures per attempt tell you whether your retry budget is being consumed by one bad hour a week or spread evenly, which decides whether the answer is more retries or a second provider. Timeouts as a proportion of completions tell you whether your worker deadline is set sensibly relative to the provider's tail. And regenerations per accepted image, the fourth class expressed as a rate, is the only one of the four that is primarily a statement about the model rather than about your infrastructure.

There is one control that sits above all four and is worth adding before any of them causes trouble: a spend circuit breaker per account and per feature. Every class except the refusal can, under the wrong conditions, produce a loop — a transport failure that retries three times against a provider having a bad afternoon, a timeout that reissues, a user holding down a regenerate button. A counter of billed generations per account per hour, with a hard stop and an alert at a threshold set well above normal use, converts every one of those from an invoice into a page. The threshold is not a product limit and should not be presented to users as one; it exists so that the worst outcome of a bug is a few dollars and a notification.

bad_request deserves the separate name it has here. A 400 that is not a policy code usually means an unsupported size, a quality tier the model does not offer, or a mask whose dimensions do not match the source image, and none of those are worth a retry either — they are your bug, and retrying them three times just delays the alert.

What this classification cannot do is tell you whether a failure is worth preventing. A 1% refusal rate might be a broken template or might be the normal cost of letting users type freely, and the fourth class is entirely silent about whether 2 regenerations per accepted image is fine or a crisis. Both questions turn into the same arithmetic once you notice that the number of generations behind each delivered image is the multiplier on every price the provider quotes you.