An image request is a job, and its result is your asset

The feature shipped on a Tuesday and looked healthy for about four hours. By Friday the team had three separate incidents open, and none of them was about the model. Roughly 4% of POST /covers requests were returning 502 with no log line on the application side. Support had a growing pile of tickets saying the cover image on a document had "gone blank" overnight, always on documents created the day before. And the image provider's dashboard showed 11,400 generations billed for a week in which analytics recorded 6,800 accepted covers, a gap far too large to explain as users regenerating for taste.

The endpoint was forty lines and every line of it was reasonable:

// POST /covers  — the version that works in dev
app.post("/covers", async (req, res) => {
  const { docId, prompt } = req.body;

  const result = await imageClient.generate({
    model: "image-gen-1",
    prompt,
    size: "1024x1024",
    quality: "medium",
  });

  const url = result.data[0].url;
  await db.documents.update(docId, { coverUrl: url });
  res.json({ coverUrl: url });
});

In development this is a nine-second request against a warm provider and a developer who is willing to wait. In production it is a synchronous call to a third party with a latency distribution that nothing in your stack was sized for, a write of a borrowed address into a column you treat as permanent, and no memory of what it has already done.

What that one call actually is#

The imageClient.generate line is a single HTTPS request to somebody else's fleet of accelerators. An image generation API is an application programming interface with the same request-and-response shape as the chat API you have already met: you post JSON describing what you want and you get JSON back. What differs is the payload on both ends. The request carries a prompt, a plain-language description of a picture, plus a handful of output settings — pixel dimensions, a quality tier, how many images you want. The response carries a picture that did not exist before, either as base64-encoded bytes or as a link to bytes the provider is holding for you. Nothing about the model runs on your machine. The provider hosts a diffusion model, which produces a picture by starting from a field of random noise and removing a little of it at a time, steered at every step by your prompt, and bills you per picture it produces.

The same family of endpoints has a second mode, which is where the "editing" in the topic's name comes from. An image editing API takes an existing image as an input alongside the prompt, usually with a mask — a second image marking which region of the first one may be changed — and returns a modified version: a coffee cup removed from a desk, a product photograph set against a new background, a square image extended outward into a landscape one. Generation makes something out of a description; editing alters something you or your user already had, which is what pulls other people's photographs through your pipeline.

These arrive as APIs rather than as a library you import because the models are large, need hardware you would otherwise rent and keep warm around the clock, and are replaced by their authors every few months. You meet them wherever a product needs a picture nobody had time to commission: the hero-image button in a marketing tool, cover art and avatars, thumbnails for user-generated content, and background replacement in anything that accepts an upload. The interface is small enough to integrate in an afternoon, which is why the handler above exists in so many codebases in almost exactly that form, and why the four healthy hours are the ordinary outcome rather than bad luck.

The seconds are not where you think, and neither are the limits#

Image generation latency is not a mean with a bit of noise around it; it is a mean with a long right tail driven by queueing on the provider's side. On a representative production sample of 1024×1024 medium-quality generations, a shape you can expect is a p50 near 7 seconds, a p90 near 19 seconds, and a p99 somewhere between 40 and 70 seconds — percentile notation, in which p99 names the figure that ninety-nine out of a hundred requests come in under, so the p99 is the experience of the unluckiest 1% rather than a worst case. The p99 is not a broken request. It is the same request, accepted, billed, and eventually successful, arriving after every timeout in the path between the browser and your handler has already given up.

Those timeouts are worth writing down explicitly, because most of them are defaults nobody chose:

The 4% of 502s came from the load balancer, which is why nothing appeared in the application logs: the process was still waiting on the provider when the connection it was going to answer on had already been closed. Raising that timeout is the obvious move and it buys you a smaller number, not a fix, because there is always a layer above with its own ceiling and a user whose patience runs out well before 70 seconds regardless.

A retry you did not write is a generation you did pay for#

The gap between 11,400 billed generations and 6,800 accepted covers is the second failure, and it is mechanically caused by the first. When the load balancer returns a 502, the client-side fetch wrapper — the one every frontend has, with three attempts and exponential backoff — issues the request again. The original generation is not cancelled by the 502; the provider never learns that anybody stopped listening, so it finishes and bills. The second request is a fresh generation at full price, and if it is slow too, so is the third.

The correction is an idempotency key that the caller generates before the first attempt and reuses on every retry of the same user intent. On your side that means a uniqueness constraint, so a duplicate arrival is a lookup rather than a new job:

CREATE TABLE image_jobs (
  id                UUID PRIMARY KEY,
  idempotency_key   TEXT NOT NULL,
  user_id           UUID NOT NULL,
  status            TEXT NOT NULL,        -- queued | running | succeeded | failed
  prompt            TEXT NOT NULL,
  params            JSONB NOT NULL,       -- model, size, quality, seed
  provider_job_id   TEXT,
  asset_id          UUID,                 -- set once bytes are in our storage
  failure_kind      TEXT,
  attempts          INT NOT NULL DEFAULT 0,
  created_at        TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE UNIQUE INDEX image_jobs_idem
  ON image_jobs (user_id, idempotency_key);

The key must be derived from the intent, not from the request. A key generated inside the retry loop, or a uuid4() computed per HTTP attempt, produces a different value on every attempt and protects nothing. Some providers also accept an Idempotency-Key header of their own with a replay window measured in hours, which is worth sending as a second layer, but it does not cover the case where your own worker crashes after billing and before writing, so the constraint above is the one doing the real work.

The URL you stored is a lease, not an address#

The third failure is the quietest and the most expensive to discover late. Providers that return a hosted URL return a signed, short-lived one. A signed URL carries an expiry timestamp and a cryptographic signature in its query string, and the storage service checks both before it serves any bytes, so the address stops working on a schedule chosen by whoever issued it. An hour is a common window here, and the guarantee is often stated as "may expire" rather than as a duration at all. The row in documents.cover_url is still there the next morning. The bytes behind it are not, and the browser gets a 403 with an XML body, which renders as a blank space rather than as an error your monitoring notices.

A CDN in front of the image — a content delivery network, the layer of geographically distributed caches that most products put between a bucket and a browser — will hide this for a while, which makes the diagnosis harder: the covers look fine for everyone whose request hits a warm edge cache and blank for everyone else, so the bug appears intermittent and regional. The fix is to treat the provider response as transport rather than as storage. Request base64 bytes where the API offers them, or fetch the signed URL immediately and once, and write the result into a bucket you control before the job is allowed to report success.

async function persistGeneration(jobId: string, b64: string) {
  const bytes = Buffer.from(b64, "base64");
  const key = `generated/${jobId}.png`;

  await storage.put(key, bytes, { contentType: "image/png" });

  const assetId = await db.assets.insert({
    storageKey: key,
    bytes: bytes.length,
    sha256: sha256(bytes),
    jobId,
  });

  await db.imageJobs.update(jobId, { status: "succeeded", assetId });
  return assetId;
}

Storing the SHA-256 is cheap and pays for itself the first time you need to know whether two jobs produced the same output, or whether an asset in a customer's export is one you generated at all.

What the shape becomes#

Put the three corrections together and the endpoint stops being a call and becomes a job submission. The HTTP request that the user's browser makes now returns in about 40 milliseconds with a job id, and the work happens on a worker that has no gateway above it and can afford to wait 90 seconds.

Two operational details make the difference between a job table that works and one that accumulates rubbish. Jobs need a terminal deadline enforced by something other than the worker, because a worker that is killed mid-generation leaves a row in running forever and a client polling it forever; a sweep that moves any job older than five minutes in a non-terminal state to failed with failure_kind = 'abandoned' costs one indexed query a minute. And the queued state needs an honest estimate attached to it, since a user who is told "generating…" for 40 seconds with no further signal will reload the page, which is another way of saying they will retry. Returning the current queue depth and a p50 estimate with the 202 lets the client show something that decreases.

Polling and webhooks are interchangeable at this level of the design. Polling every 2 seconds from an open document costs one cheap database read per client per interval and works through every proxy and corporate firewall you will meet. A provider webhook — an HTTP callback the provider makes to an endpoint of yours when the job finishes, so the notification travels in the opposite direction from every other call in this chapter — removes the polling but adds a public endpoint you must authenticate, a replay policy, and a fallback for the deliveries that never arrive — which means you end up writing the polling path anyway as a reconciliation sweep. Starting with polling and adding webhooks when the read volume justifies it is the ordinary path.

Which one you fix first#

The three failures are independent bugs with independent fixes, and the order matters because it is an order of irreversibility rather than of severity.

Copying the bytes goes first because every hour it is not shipped destroys assets that cannot be recovered — the prompt may still exist, but a re-generation produces a different image, and for a customer whose brand deck used the old one that is not the same asset. Idempotency goes second because it is bounded loss in money rather than in data. The asynchronous job shape goes last despite being the most-discussed of the three, because it is the largest change and the only one whose damage is fully visible while you wait.

That ordering is also the answer to the version of this question that gets asked in interviews, where the tempting reply is to raise the request timeout and add a retry with backoff. That reply makes two of the three failures worse: a longer timeout keeps a connection open through the tail of the latency distribution without changing it, and a retry with backoff, without an idempotency key, is precisely the mechanism that generated the 4,600 extra billed generations.

What this shape does not buy you#

None of it is an argument for making every third-party call asynchronous. A geocoding lookup with a p99 of 300 milliseconds inside a request that already takes 200 is not improved by a job table; you would be adding a queue, a worker, and a polling loop to a problem that a 2-second timeout solves. The trigger for this shape is a latency tail measured in tens of seconds combined with a per-call price, and those two together are unusual outside generation.

The job record also does not make anything cheaper. It stops you paying twice for the same intent, which is a leak rather than the cost model, and the actual per-image economics are unaffected by it. It says nothing about whether the image is any good, and it does not decide what a worker should do when the provider returns a refusal rather than an image — a job that automatically retries every failure will burn money faster than the inline version did, so the failure_kind column above is doing nothing yet.

What has changed is ownership. The bytes now sit in a bucket you control, under a URL on your domain, with a row in your database naming the user who asked for them and the prompt that produced them. That is the position from which the next question becomes answerable and unavoidable: those bytes are now something you are hosting and serving, and deciding what may be in them is not a decision the provider makes for you.