Bulk work in a request handler

The shape is ordinary. An import endpoint accepts a CSV of product rows, summarises each one with a model call, and returns the summarised rows as JSON:

@app.post("/imports")
async def create_import(file: UploadFile) -> ImportResult:
    rows = parse_csv(await file.read())
    summaries = await asyncio.gather(*(summarise(client, row) for row in rows))
    return ImportResult(rows=[row.with_summary(s) for row, s in zip(rows, summaries)])

At an account bound of five calls in flight, 812 rows at 800 ms each is 130 seconds of work. A load balancer in front of a service typically has a 60-second idle timeout — the time it will hold a connection on which no bytes have moved — and this handler sends nothing until it is finished, so at 60 seconds the balancer closes the connection and answers the browser with a 504. The handler does not find out. It keeps running, completes the remaining 437 summaries, serialises a response, and writes it into a socket nobody is attached to. The bill for that request is 812 summaries and the delivered value is zero.

Every figure there comes from one rate. Five calls in flight, each holding its slot for 800 ms, complete five rows every 800 ms — a little over six rows a second — so a row costs about a sixth of a second of wall clock and 812 of them cost 130 seconds. The balancer's cut lands at 60 seconds, and sixty seconds at that rate is a little under three hundred and eighty rows, which is why 437 are left for the handler to summarise after the connection carrying their results has already been closed. Twenty rows costs 3.2 seconds by the same arithmetic, 200 rows costs 32 and 3,000 rows costs 480, which is the table further down and the reason the fixture passed.

That the handler does not find out is the mechanism from the disconnect chapters one layer further out. The balancer closes the connection it holds to the browser and the connection it holds to your service, and a closed socket delivers no interrupt into a coroutine. The handler is being driven by the event loop, which is waiting on asyncio.gather, which is waiting on provider calls; not one of those is watching the inbound socket, because nothing asked it to. The work therefore continues, and the only thing that has changed is that the eventual write has nowhere to land. All 812 summaries are charged, and the delivered value is zero rather than partial precisely because the results are assembled into a single response at the end.

What actually breaks, in the order it breaks#

The timeout is the visible failure and the least interesting one. Three things go wrong underneath it, and they get worse with the number of people using the feature rather than with the size of any one file.

The provider gates are shared. The call and token gates from earlier in this topic — CALL_GATE and TOKEN_GATE in the code of the chapter on token bounds — are process-level objects sized for the whole account, so an import holding all five call slots for 130 seconds means a chat endpoint in the same process gets none. Six merchandisers importing at once either queue behind each other on the gate, making each import six times slower, or — if someone sized the gate per request instead of per process — present 30 calls in flight against an allowance of five, and the chat traffic starts receiving rate-limit rejections caused by a completely different feature.

The request slot is occupied. On an async framework the handler yields at every await, so the event loop is free and what is pinned is the request slot rather than the thread: uvicorn --limit-concurrency 100 counts this request against that hundred for 130 seconds, and the memory holding 812 parsed rows, 812 prompt bodies and 812 pending coroutines stays resident the whole time. On a synchronous stack the situation is blunter, because a Django or Flask view under gunicorn sync workers occupies an operating system worker for the full 130 seconds, and four workers means four concurrent imports take the entire service offline for everything, including the health check.

Deploys and restarts destroy the work. A rolling deploy sends SIGTERM and waits for the graceful shutdown period, commonly 30 seconds, which is a quarter of what this request needs. Any import in flight during a deploy is lost, and since the results are only written at the very end, it is lost completely rather than partially.

The row that explains why this reaches production is the first one. There is no size at which the design starts being wrong; there is only a size at which the wrongness exceeds a timeout, and a staging fixture is chosen to make tests fast.

The queue is the architecture; the experience is a separate question#

Accepting the upload, writing the rows to durable storage, enqueueing a job and returning a job identifier is the correct shape, and it is where most versions of this proposal stop. The endpoint answers in about 120 ms, no request is ever long, deploys stop destroying work, and the import survives a file of any size.

@app.post("/imports", status_code=202)
async def create_import(file: UploadFile, idempotency_key: str = Header(...)) -> Accepted:
    payload = await file.read()
    import_id = await store.create_import(
        rows=parse_csv(payload),
        fingerprint=idempotency_key,       # a re-upload returns the same import
    )
    await queue.enqueue("summarise_import", import_id)
    return Accepted(import_id=import_id, status_url=f"/imports/{import_id}")

The fingerprint is doing real work there. A user whose first attempt died at the balancer will press the button again, and without one that second press creates a second import and pays for 812 summaries twice. Hashing the file contents together with the user id and returning the existing import when it matches turns the retry into a no-op, which matters more here than on a cheap endpoint because each duplicate has a direct token cost.

The part that gets skipped is what the product manager was actually asking for. "The synchronous experience is the product" is a statement about what the user sees on the screen — upload a file, watch results appear, read them without navigating anywhere — and none of that requires the HTTP request to stay open. Returning a bare job identifier and telling the user to check back later is a genuinely worse product than the broken version, which is why architecturally correct proposals get rejected and the timeout survives another year.

What preserves the experience is streaming the progress back over a connection designed for it, which in a browser is usually server-sent events — one long-lived HTTP response the server appends messages to as things happen, which the page consumes with an EventSource:

@app.get("/imports/{import_id}/events")
async def import_events(import_id: str) -> EventSourceResponse:
    async def stream() -> AsyncIterator[dict[str, str]]:
        async for event in store.watch(import_id):     # emits per completed row
            yield {"event": event.kind, "data": event.json()}
    return EventSourceResponse(stream())

Because each row is written to storage as its worker finishes it, the page can fill in rows one at a time starting about a second after the upload, which is visibly faster than the old version was even on files it survived.

An event stream is not the only way to deliver this. Polling GET /imports/{id} every two seconds is less elegant, works through every proxy ever built, and is the right choice where infrastructure treats long-lived connections badly. Both are better than the job identifier on its own, and the difference between them is an infrastructure decision rather than a product one.

Moving state out of the request also changes what a failed row means. In the inline version a single row that returns a malformed response or exhausts its retries raises out of asyncio.gather, and the caller receives an error for the whole import — 811 successful summaries paid for and thrown away because of one. At a per-row failure rate of 0.4%, unremarkable for a model call with a strict output schema, an 812-row import fails outright about 96% of the time. With per-row state the same three failures become three rows carrying status = 'failed' and an error message, alongside 809 rows the user can read and act on immediately, with a retry that costs three calls rather than 812.

The 96% deserves its own step, because 0.4% per row reads as harmless and the import-level figure does not. An import of this shape succeeds only if every row succeeds, so its chance of succeeding is the per-row success rate multiplied by itself once for every row — 812 times over. Compounding a number slightly below one that many times drives it close to zero, and what survives is a success probability of roughly four in a hundred. The general shape is worth carrying past this example: a failure rate that is negligible per item becomes close to certain per batch as soon as the batch is all-or-nothing, and the batch size at which it turns is approximately the reciprocal of the per-item rate. At 0.4% that is a couple of hundred rows, which is well below the file that actually arrived and slightly above the fixture that was tested.

Progress reporting gets easier for the same reason. The completed-row count and the elapsed time give an honest estimate without any modelling — 210 rows in 34 seconds is 6.2 rows per second, so 602 remaining rows is another 97 seconds — and it is worth showing that number rather than an indeterminate spinner, because a user who can see 97 seconds will wait and a user watching a spinner will refresh, and every refresh of the inline version started the work again.

That estimate needs no model of anything, which is the point of it. Two hundred and ten rows in 34 seconds is 6.2 rows a second, the remaining 602 rows divided by 6.2 is 97 seconds, and both figures come from counting rows that have finished rather than from predicting rows that have not. Per-row state is what makes that possible: once completion is recorded as it happens, progress is a measurement rather than a forecast, and a measurement can be put in front of a user without a hedge. The inline handler had nothing to count, because it filled nothing in progressively and held every result until the end, so a spinner was the only honest thing it could display.

When inline is still correct#

None of this argues that long requests are forbidden. A handler that makes one model call and returns in 3 seconds is fine inline, and wrapping it in a queue adds a storage round trip, a delivery guarantee to reason about and a second place for the work to get stuck, in exchange for nothing. The boundary is not a duration so much as a question of whether the number of provider calls is fixed in the code or comes from the size of something the user uploaded — one call, or three, is a handler; len(rows) calls is a job.

Streaming a single model response over a chunked or SSE response also keeps a request open for a minute or more quite legitimately, because bytes are flowing continuously and no idle timeout is triggered. That works precisely as long as the work is one call whose output arrives incrementally, and it stops working for a fan-out over 812 rows, where the natural unit of output is a completed row and the natural place to put it is durable storage.

The queue is also not free. It introduces at-least-once delivery, meaning a worker that dies after calling the provider and before writing the result will redo that row and pay for it twice, so per-row writes need to be idempotent on the row identifier. It introduces a class of stuck job nobody notices, because no user is holding a connection open to complain, so three alerts replace that feedback and none of them is optional: the age of the oldest unstarted job, which catches a worker fleet that has died or is undersized; the count of imports with unfinished rows older than ten minutes, which catches jobs that started and stalled; and the depth of the dead-letter queue, where a worker framework parks a message that has failed every retry, which catches rows failing systematically rather than incidentally.

What the migration buys beyond survival is that cost becomes attributable for the first time. Each row records the tokens its summary consumed, so an import has a price, and whether a 5,000-row import should be allowed at all becomes answerable with a number instead of a guess. That matters more than it sounds when the feature is exposed to customers, since the inline version's practical limit on file size was enforced by a timeout, and removing the timeout removes the only thing bounding spend per user.

The move to a queue is therefore two decisions, not one. Where the work runs decides whether it survives a file of any size, a deploy and a refresh. What the user sees while it runs decides whether the change is accepted at all, and that part is per-row state written as the work completes rather than a job identifier and a promise.

The queue also moves the work to a fleet whose size is now a capacity decision, and that is the consequence which undoes the improvement if it goes unexamined. The API is responsive, the imports complete, and the throughput is frequently identical to what the inline version achieved, because moving work to a worker changes who is waiting for it without changing how fast anything calls the provider.