Sequential and concurrent awaits

A support desk takes five hundred tickets on a busy day, and somebody has decided that each one should arrive at the on-call engineer with a two-sentence summary attached. The summary comes from a hosted chat API: one call per ticket, prompt in, summary out. Topic 2.1 owns what that call is and what it costs. The part that matters here is only its shape — a request leaves the process, crosses a network to somebody else's machine, waits there while a model produces tokens, and comes back.

Written the obvious way, the job is a loop. Take a ticket, call the API, wait for the summary, store it, take the next ticket. Nothing about that loop is wrong. It produces five hundred correct summaries, and on a desk that takes five tickets an hour nobody would ever look at it again. On five hundred, it takes long enough that the on-call engineer reads the last ticket the following morning, and the machine running it spends almost the whole of that time doing nothing at all: no CPU work, no memory pressure, a single open socket, and a process sitting still with one outstanding question to a server across the internet. The work is not slow. The waiting is serial.

Nothing forces it to be. No summary depends on any other, so no call has any reason to wait for the one before it — five hundred requests could be in flight at once, and the whole batch would take about as long as the slowest one of them. That gap — between the time a batch of calls takes and the time the calls themselves take — is what everything below is about.

The pieces asyncio gives you for running many calls at once#

Nine things close that gap, and they build on each other in a fixed order. A coroutine is a unit of work whose body can pause partway through. The event loop is the thing that actually runs coroutines and switches between them. A task lets the loop track a coroutine on its own rather than only when something is waiting on it. gather launches a batch and collects its results. A task group supervises a batch, so one failure does not leave the rest running unwatched. Fan-out is the name of the pattern once those pieces are in hand. A semaphore caps how many of them run at once. A bounded queue and a worker pool reach the same cap without building every coroutine up front. And once the hazard — whatever it is that holds the whole program still — stops being waiting and starts being computing, a blocking call, event loop lag, the Global Interpreter Lock, a thread pool executor and a process pool are what the problem turns into and what answers it.

They arrive in that order for a reason. The first pair is what makes a single call pausable and runnable at all, and until a call can pause there is nothing to interleave. The next group is what runs many waiting operations at once and decides what happens when one of them fails. The group after that is what bounds how many run at once, because a batch that overlaps perfectly and hits a provider's ceiling has traded one failure for another. The last group is for the moment something inside the loop refuses to wait cooperatively at all, which is a different problem wearing the same syntax.

Coroutines and the event loop#

The first of those nine things is the coroutine: a function written with async def whose body can pause at an await expression and pick up again later, without tying up the thread it runs on.

The smallest useful one takes a ticket and returns its summary.

async def summarize_ticket(ticket: Ticket) -> str:
    response = await chat_client.complete(
        model="claude-haiku-4-5",
        prompt=f"Summarise this support ticket in two sentences:\n\n{ticket.body}",
    )
    return response.text

The await on the middle line is the pause point. When execution reaches it, the request has gone out to the provider — the service on the far side of chat_client that runs the model and returns the summary — and the answer has not come back, and rather than standing there holding the thread, the function suspends: it keeps its local variables, keeps its place in the body, and hands control away until the response arrives.

Hands it to what, though. Calling the function is not enough to find out, because calling it does not run it.

>>> summarize_ticket(ticket_492)
<coroutine object summarize_ticket at 0x104f2a740>

No request went out. Nothing was sent to the provider, nothing was billed, and no socket was opened. What came back is an object representing the work, with the body still at its first line. A coroutine is a description of work rather than the work running, which is the first thing that separates it from an ordinary function, where calling and running are the same act.

Something has to take that description and drive it, and that something is the event loop: the single-threaded scheduler that runs ready coroutines and callbacks one at a time, switching between them at await points. Give it the coroutine and the request happens.

>>> asyncio.run(summarize_ticket(ticket_492))
'Customer cannot log in after a password reset. The reset email arrives but the link 400s.'

asyncio.run starts a loop, hands it the coroutine, drives the coroutine to completion, and shuts the loop down. Inside that, the switching is mechanical and worth following once, because "switches at an await point" is a phrase until you can picture it. The loop keeps a set of coroutines it is running and a record of what each paused one is waiting for, and it moves each one around this cycle.

Read off the arrows what the loop never does: nothing preempts anything. A coroutine runs until it chooses to yield, and choosing to yield is what await does. Corey Schafer's Python Tutorial: AsyncIO - Complete Guide to Asynchronous Programming with Animations walks the same cycle on a timeline, which is the other form it is easy to hold in your head.

The cycle looks inevitable once you have watched it, but the loop that runs it and the syntax that marks its pause points are both younger than the problem they solve, and the shape of them is an answer to how Python solved it before. Both pieces arrived together and replaced something. asyncio entered the standard library in Python 3.4 under PEP 3156, and the async/await syntax used above landed in 3.5, as one standard answer to a job Python was already doing two other ways. One way was a thread per in-flight request, which reads top to bottom and pays an operating system thread's memory and scheduling cost for work whose entire duration is waiting. The other was callback-chained I/O in the style of Twisted or Tornado, which kept that cost down and scattered the steps of one logical operation across handlers, so that following a single request meant jumping between functions that no longer sat in reading order. The coroutine bought the sequential reading order of the threaded version at roughly the cost of the callback version, with every pause point marked in the source by the word await instead of left implicit in where one handler hands off to the next.

Look at the size of what that has built, though. One coroutine, run by one loop, doing one thing. Nothing here overlaps with anything, because there is nothing else present to overlap with.

Awaiting five hundred chat calls in sequence#

One coroutine on its own has nothing to overlap with. A support inbox does: five hundred tickets and the obvious loop.

async def summarize_all(tickets: list[Ticket]) -> list[str]:
    summaries: list[str] = []
    for ticket in tickets:
        summary = await summarize_ticket(ticket)
        summaries.append(summary)
    return summaries

Put a number on the call. Take 700 milliseconds per chat call, and hold that figure fixed for everything that follows, so that every wall-clock claim later can be checked against it. Five hundred tickets at 700 milliseconds each, one after another, is 350 seconds: close to six minutes for a job whose actual computation is five hundred string concatenations.

The arithmetic is that simple because of what await does in this position. The loop body suspends at summarize_ticket, and the for loop cannot advance past a suspended body — the next iteration begins when the current one finishes, which is what a for loop means. So the pause hands control back to the event loop exactly as designed, the loop looks around for something else that is ready, and there is nothing else, because nothing else was ever started. Five hundred pauses happen and five hundred resumptions happen, and they happen strictly one at a time.

Which is worth saying plainly, because the syntax invites the opposite conclusion: writing a function as async def does not, by itself, make anything run concurrently. Every line in the loop above is inside async syntax, summarize_ticket is a coroutine, await is doing precisely what it is documented to do, and the result is exactly as sequential as the same loop written with a blocking HTTP client. The keyword makes a function pausable. It does not create anything to run during the pause.

So something else has to be true for these five hundred calls to overlap.

Tasks#

Awaiting each call in turn is the thing that stops them from overlapping. What would it take to hand a coroutine to the event loop and move on to the next one without waiting for the first to finish?

The answer is a task: a coroutine wrapped so the event loop schedules and tracks it independently, letting it run concurrently with others. asyncio.create_task does the wrapping. Using tasks turns out to require a particular shape of code: one loop to create them, another to collect what they produce.

async def summarize_all(tickets: list[Ticket]) -> list[str]:
    tasks = [asyncio.create_task(summarize_ticket(t)) for t in tickets]
    summaries: list[str] = []
    for task in tasks:
        summaries.append(await task)
    return summaries

Two loops now, and the split between them is the whole difference. The first loop never awaits anything. create_task registers the coroutine with the loop as work to be driven and returns immediately, so the list comprehension runs through all five hundred tickets in the time it takes to allocate five hundred objects. By the time the second loop starts, the loop is already holding five hundred pieces of work it is responsible for.

Now the awaiting behaves differently, and it behaves differently for a reason already established rather than a new one. The second loop awaits ticket 1's task, which suspends the calling coroutine, which hands control to the event loop, which looks around for something ready — and this time there are 499 other things, every one of them at its first line. It starts ticket 2, which runs to its own await on the API call and pauses. It starts ticket 3. It works through all of them, and by the time it has, ticket 1's response is arriving. The 700-millisecond calls now overlap almost entirely, so five hundred tickets finish in something close to 700 milliseconds plus the scheduling overhead of starting and settling five hundred tasks, rather than 350 seconds.

Note carefully what did not change to produce that. Adding tasks did not add a thread and did not add a CPU core. There is still exactly one event loop, running on the one thread it was started on, still running one coroutine body at a time and still switching only where a body yields. Five hundred tasks is five hundred pieces of bookkeeping on that single loop, and the speed-up came from the fact that 499 of those pieces were parked on a socket rather than using the thread. Overlapping waiting is free in a way that overlapping computing is not, and that distinction goes quiet for the whole middle of this subject before it comes back hard once the workload stops being about waiting.

The waiting came free, but the pattern that bought it did not, and what the rewrite cost is visible in its own shape. Creating five hundred tasks by hand and collecting five hundred results in a second loop works, and the create-all-then-await-all structure is exactly the shape that needs a name, a rule for what one failure does to the other 499, and eventually a rule for how many to create in the first place.

Gather#

Collecting five hundred results in a hand-written second loop is what asyncio.gather is for: a function that runs a set of awaitables concurrently and returns their results once all of them complete.

summaries = await asyncio.gather(*(summarize_ticket(t) for t in tickets))

Looking at the code, there's one line where there were seven. The results come back as a list in the order the awaitables were passed, not the order they finished, so summaries[137] is ticket 137's summary whether that call returned first or last. The overlap is the overlap already described — gather wraps each coroutine in a task, so the wall clock is the same slightly over 700 milliseconds.

Now change exactly one thing about the batch. Ticket 137 has a malformed attachment reference in its body, and the provider rejects the request with a 400. Everything else is as it was: the same five hundred tickets, the same coroutine, the same line.

try:
    summaries = await asyncio.gather(*(summarize_ticket(t) for t in tickets))
except ProviderError as exc:
    logger.error("ticket batch failed: %s", exc)
    await chat_client.aclose()

With gather's default arguments — which is to say with return_exceptions left at False — the 400 does not wait for its siblings. Roughly 700 milliseconds into the batch, ticket 137's task raises, gather sees the first exception among the awaitables it is watching, and that exception comes straight out of the await asyncio.gather(...) line to whoever called it. Here that is the except block, which logs and closes the shared HTTP client.

Which raises the question the next section has to answer: five hundred calls were in flight and one of them failed, so what happened to the other 499.

Orphaned tasks#

They are still running.

The reason they are still running is that gather propagated the first exception it saw. It did not touch the tasks behind the remaining 499 awaitables, and nothing in the language or the runtime interpreted the propagation as an instruction about them. Each of those tasks is still registered with the event loop, still parked on an open socket to the provider, still counted against whatever quota the account is billed on, and still going to be resumed when its response arrives. When each one finishes, it will write its summary into a result object that the gather call no longer holds a reference to, because gather already returned by raising.

Freeze it at a specific moment and it is a state you can check on a real service. One second after ticket 137's 400 came back: the caller is inside its exception handler and may already have finished running it; no cancellation has been requested on anything, because nothing requested any; 499 requests are open against the provider; and the summaries those requests produce over the next fraction of a second are written into objects nobody is looking at. Nothing in the runtime will move that state on its own. gather returning early is not gather stopping anything.

An orphaned task, then, is a task left running and unsupervised after the code that launched it has moved on — here because a sibling failed and the caller's await already returned.

The failures among them are worse than the successes. When a task raises, the exception has to go somewhere, and where it goes is onto the task object itself, where it sits until something asks for it by awaiting the task or calling result(). In this batch nothing ever will. So the exception waits on an object with no remaining references, and when the garbage collector destroys that object, asyncio notices that the stored exception was never retrieved and logs it — the Task exception was never retrieved line, with a traceback, arriving at whatever moment collection happens to run, disconnected from the code that launched the task and long after any handler that could have acted on it has returned.

That disconnection is a real cost rather than a curiosity, and the except block above shows why. It closes the shared HTTP client, which is the correct thing to do when a batch is finished and the wrong thing to do while 499 requests are still using it. Those calls now fail on a closed transport, and each failure surfaces as a connection error against a ticket whose own request was perfectly well-formed. The tickets that report a problem are not the ticket that had one.

Task cancellation#

Preventing that cascade means stopping the unwatched tasks before their failures spread, and asyncio has a mechanism for exactly that: cancellation is the means by which a running task is asked to stop, raising CancelledError inside it at its next suspension point.

task = asyncio.create_task(summarize_ticket(ticket_492))
await asyncio.sleep(0.1)
task.cancel()

try:
    await task
except asyncio.CancelledError:
    logger.info("ticket 492 summary abandoned")

task.cancel() does not stop anything by itself. It sets a flag on the task and schedules the delivery. The coroutine is a tenth of a second into its API call, suspended at the await inside summarize_ticket, and the next time the loop would have resumed it — with a response, or with an error, or immediately if it is already resumable — it resumes it by raising CancelledError at that await instead of returning a value. The coroutine's finally blocks and context managers run on the way out, which is how the HTTP connection gets released, and then the task finishes in the cancelled state.

The words "next suspension point" are doing real work in that sentence and are not a synonym for "instantly". A task in the middle of an uninterrupted stretch of code has no point at which the loop can hand it anything, so it keeps running to the end of that stretch, and the cancellation is delivered whenever it next yields. For a coroutine that is sitting on a network call, that is essentially immediate. For one that is not, it is not, and how far "not" goes is a question this comes back to.

Cancelling one task by hand is easy. Cancelling the right 499 tasks, at the right moment, only when one of them actually fails, and then reporting what happened once they have all finished unwinding, is not something anybody wants to write correctly for every batch.

Task groups#

Task groups solve that automatically: a task group is an asyncio construct that supervises a set of child tasks as a single unit, cancelling every remaining task the moment one of them raises. That makes it the second way to fan out a batch and wait for it, and the one gather is most often confused with.

Before the contrast, the thing both constructs have in common, because it is why they are so easy to confuse in the first place. gather and a task group take the same input and give back the same kind of thing: a batch of coroutines that should run concurrently goes in, and what comes back to the caller is one place in the code where the batch's outcome is known. Both wrap each coroutine in a task. Both let the loop overlap all of them. Both turn a fan of concurrent work into a single point the calling function can await.

They differ on what that single point does about failure. Where gather left the 499 open requests two sections back, a task group, given the same failure, cancels every task still in flight and raises only once the batch is fully settled.

async with asyncio.TaskGroup() as tg:
    tasks = [tg.create_task(summarize_ticket(t)) for t in tickets]
summaries = [t.result() for t in tasks]

The batch is the same: five hundred tickets, with ticket 137 the one that fails at roughly 700 milliseconds. The async with block is what changed: tg.create_task registers each coroutine as a child of the group, and the closing edge of the block is where the caller waits.

Follow ticket 137's 400 through it. Its task raises at roughly 700 milliseconds. The group is notified, and it calls cancel() on each of the 499 children still running. Each of those receives CancelledError at its next suspension point, which for a task parked on a socket is the next turn of the loop, so their connections are released and their tasks settle as cancelled. The async with block does not exit while any child is still finishing — that is what makes it a supervisor rather than a signal — and when the last one has unwound, the block raises an ExceptionGroup containing ticket 137's ProviderError.

Put that against where gather left the batch at the same second. One second after ticket 137's 400, the task-group version has nothing in flight, has stopped consuming provider quota, and the caller is either still inside the async with waiting for the final cancellation to land or already holding an ExceptionGroup that accounts for the whole batch. At that same second, the gather version had 499 calls open and a caller that had moved on.

Both the construct and the exception type are recent: asyncio.TaskGroup and the ExceptionGroup it raises were added in Python 3.11, specifically to give asyncio the all-or-nothing supervision that gather's default behaviour does not provide. On 3.10 or earlier the async with asyncio.TaskGroup() line above raises AttributeError, and the gather-based shape is what is available. The pyblog.in guide asyncio TaskGroup Patterns: The Complete Guide documents the cancellation sequence in more depth than belongs here, including what happens when a child swallows its own CancelledError.

State the difference as a fact about the batch rather than a preference between two APIs. What varies independently is whether the five hundred tickets are all-independent work, in which case one failure is one failure, or all-or-nothing work, in which case one failure ends the batch. That is a property of the job somebody is doing, not a property of asyncio, and the same five hundred tickets are legitimate work under either reading.

Fan-out#

Both constructs implement one pattern, and it now has enough behind it to be named. Fan-out is the pattern of dispatching many concurrent operations from a single point and then collecting or reconciling their results. The ticket batch is a fan-out whichever way it was written.

The choice between the two implementations follows from what the batch shares. Reach for a task group when the calls share something that makes an orphaned task expensive: a connection pool that the failure path will tear down, a budget that keeps being spent after the answer stopped being wanted, a downstream system that should stop being hit the moment something is known to be wrong. Reach for gather with return_exceptions=True when the calls are genuinely independent and every result is wanted regardless of any single failure — that flag stops gather raising at all, and instead places each exception object into its own slot in the returned list, so summaries[137] holds the ProviderError and the other 499 slots hold summaries. Five hundred support tickets are usually this second case: a 400 on ticket 137 is a fact about ticket 137's attachment, and cancelling 499 healthy summaries over one malformed ticket is the wrong outcome for the desk. What that costs you is that failures are now data rather than control flow, so something afterwards has to walk the list, separate the exception slots from the summary slots, and report per-ticket outcomes — a fan-out that reports one aggregate error makes a single bad record look like an outage.

Neither answer says anything about how many calls should be in flight at once. The five hundred tickets, run either way so far, launch all five hundred immediately.

Semaphores#

To answer that question, we turn to a semaphore: a counter-based primitive that admits only a fixed number of concurrent holders, making anything beyond that count wait.

sem = asyncio.Semaphore(50)

async def summarize_bounded(ticket: Ticket) -> str:
    async with sem:
        return await summarize_ticket(ticket)

summaries = await asyncio.gather(
    *(summarize_bounded(t) for t in tickets), return_exceptions=True
)

All five hundred coroutines still start. What changed is where the first fifty-one of them get to. Each one reaches async with sem:, which decrements the counter and proceeds if there is a permit left. The first fifty take the fifty permits and go on to their API calls. The fifty-first finds the counter at zero and suspends there, on the semaphore rather than on a socket, and so do the 449 behind it. When one of the running calls returns, its async with block exits, releases the permit, and the loop wakes one of the waiting coroutines, which acquires it and starts its own call. Fifty requests are in flight at any instant, the batch still completes every ticket, and it completes them admitted fifty at a time instead of all at once. The superfastpython.com reference Asyncio Semaphore in Python covers the primitive's fuller behaviour, including acquiring without a context manager and what a semaphore initialised at zero is good for.

Now change one variable and nothing else. Keep Semaphore(50), and give each call up to three attempts with a backoff sleep between them when the provider answers 429 — topic 2.11 owns what a 429 is and how backoff should be shaped, so treat the sleep here as given. There are two places to put the retry loop relative to the async with, and they are not equivalent. Say forty of the fifty in-flight calls get rate-limited at roughly the same moment and each backs off for four seconds.

Same function, same Semaphore(50), same three attempts, same sleep durations; the only difference is which of the two statements encloses the other, and it costs four fifths of the batch's throughput for the length of every backoff. Nothing about the primitive broke while that happened: the semaphore's count stayed at exactly the fifty it was configured with, and it admitted exactly fifty holders throughout, which is its whole contract. What collapsed is the useful concurrency, because a permit held by a sleeping task is a permit doing no work.

A four-second window in which almost nothing gets sent invites the wrong culprit, and the loop is the first thing a reader suspects, so be precise about what those sleeping tasks are and are not doing to it. asyncio.sleep is itself an await point, so a task sleeping through its backoff hands control straight back to the event loop for the whole four seconds, and the loop spends that time running every other ready coroutine exactly as it would otherwise. The loop is not stalled, not late, and not the thing that got worse. time.sleep in the same position would do something completely different to it, and that case has its own section further down. Here the resource being held is the permit, not the loop's thread.

Memory is the other thing a reader suspects, and it is also not what degrades. A retry is not a new unit of work: the backoff and all three attempts happen inside the same task that was already created for that ticket, so no additional coroutine or task objects are allocated however many times the batch retries, and the process's memory during a 429 burst looks like its memory during a clean run. The single thing that degrades is the rate at which waiting tickets get admitted.

Which is also the semaphore's real limit. It bounds how many calls can be in flight, and it says nothing whatever about how long any one of them takes once admitted.

Queue time#

While a coroutine waits for a free permit, that wait is itself a duration on a real request, and it needs a name and a way to see it. Queue time is the interval a unit of work spends waiting for a free semaphore slot or worker before it starts executing.

It is easy to miss for a structural reason. A per-request timer is almost always started where the request's real work begins, which in the code above is inside the async with, after the permit has already been acquired. A ticket that waited nine seconds for a permit and then took 700 milliseconds is recorded as a 700-millisecond request. Percentile latency — p50 and p99, owned by topic 15.3 — is computed from those recorded numbers, so queue time does not appear in either of them as itself. It appears as a gap opening between them, without either number naming the wait.

Sizing the permit count against that wait is arithmetic rather than feel. If you know roughly what rate of work you want to sustain and roughly how long one unit takes once it starts, Little's law gives the concurrency needed to avoid queueing at all: concurrency is approximately the target rate multiplied by the average latency. Run it on the numbers already in play. The desk wants its five hundred tickets summarised inside about seven seconds, which is a target of 70 summaries per second, and a summary takes 0.7 seconds once it starts. Seventy per second times 0.7 seconds is 49. Forty-nine calls have to be in flight at any instant to sustain that rate, which is where the Semaphore(50) of the previous section came from — not a round number somebody liked.

The law is more useful run backwards, because that is the direction a problem arrives in. A service bounds provider concurrency with Semaphore(20) because twenty seemed reasonable. Traffic doubles. Its p99 goes to 45 seconds while its p50 sits at 800 milliseconds, which is where it always sat.

Running the law backwards means starting from the permits rather than the target, so work the ceiling out first — the highest rate this service can complete work at, whatever arrives. Twenty permits, each occupied for about 800 milliseconds per request, turn over 1.25 requests per second each: twenty times 1.25 is 25 requests per second, and that is the most this service can complete no matter what arrives. Below 25 per second the permits are never all busy for long, and above it a backlog forms and grows for as long as arrivals stay above the ceiling. Now the two numbers fall out. A request that happens to arrive when the backlog is empty gets a permit immediately and finishes in its 800 milliseconds, and enough requests do that for the median to stay exactly where it was — which is the whole reason p50 did not move. A request that arrives behind a backlog waits for the permits to turn over ahead of it, and at 25 per second a backlog of a thousand requests is forty seconds of waiting before its own 800 milliseconds begins. The 45 seconds is queue time almost in its entirety.

The p50 that did not move is the diagnostic signal, and it points away from two things at once. The calls themselves did not get slower, so the provider is not degraded, and raising the request timeout would extend the wait without shortening it. Lowering the semaphore, which is the instinctive response to seeing 429s alongside this, cuts the service rate of the queue that is already the bottleneck and makes the 45 seconds longer.

So if twenty is wrong and lowering it is worse, the replacement number comes from the law used forwards again. To sustain the new arrival rate — say it settled at 40 requests per second — at 800 milliseconds each, concurrency has to be 40 times 0.8, which is 32, plus headroom for arrivals that come in bursts rather than evenly. And 32 is a candidate rather than an answer until it is checked against the provider's published requests-per-minute and tokens-per-minute limits, because that ceiling frequently sits below what the arithmetic alone suggests, and it is enforced per account or per key rather than per process — so a fleet of four replicas each holding Semaphore(32) is asking for 128.

None of which touches memory. Sizing the semaphore this way still means all five hundred coroutines exist from the moment the batch starts, each holding its captured ticket, and at ten thousand tickets with real payloads that is the whole corpus resident before the first call goes out.

Bounded queues and worker pools#

To hold fewer of those coroutines in memory at once, the shape has to change at the point where work is created. A bounded queue is a queue with a fixed maximum size that stops accepting new items, blocking or rejecting producers, once it is full.

On its own that is not the fix, and it is worth being clear why before building on it. A bounded queue holding ten thousand tickets has ten thousand tickets in memory just as a list would, unless the bound is small and something is draining it continuously. The bound is only a limit on what has been produced but not yet consumed; it does the work when there is a consumer.

So: a worker pool is a fixed number of long-lived consumers that pull work items one at a time from a shared queue instead of being spawned per item.

async def worker(queue: asyncio.Queue[Ticket], out: dict[str, str]) -> None:
    while True:
        ticket = await queue.get()
        try:
            out[ticket.id] = await summarize_ticket(ticket)
        finally:
            queue.task_done()


async def summarize_all(tickets: Iterable[Ticket]) -> dict[str, str]:
    queue: asyncio.Queue[Ticket] = asyncio.Queue(maxsize=100)
    out: dict[str, str] = {}
    async with asyncio.TaskGroup() as tg:
        workers = [tg.create_task(worker(queue, out)) for _ in range(20)]
        for ticket in tickets:
            await queue.put(ticket)
        await queue.join()
        for w in workers:
            w.cancel()
    return out

Twenty workers, each an ordinary coroutine in an infinite loop, each suspended on queue.get() whenever there is nothing to take. The producer loop pushes tickets in and suspends on queue.put() whenever a hundred are already waiting, which is the bound doing its job: a producer reading from a database cursor or a file cannot run ahead of the consumers by more than a hundred rows. queue.join() returns when every item put in has been marked done, and the explicit cancel() afterwards is what ends workers that would otherwise sit on queue.get() forever. A task group tolerates children that were cancelled deliberately, so the block exits cleanly.

That gives two ways to hold the same cap, so put the pool next to the semaphore before separating them, because they are built for the same number. Twenty semaphore permits and twenty workers both cap the work at twenty concurrent API calls, both let a finished slot be reused immediately, and both produce identical results in near-identical wall-clock time. A reader could reasonably expect them to be the same design written twice.

What varies independently is what exists in memory while that cap is being enforced. Against the semaphore's resident coroutine per ticket, twenty workers pulling from a bounded queue means exactly twenty coroutines exist at any moment, and the other 9,980 tickets are either plain values in a hundred-slot queue or not yet read from wherever they are coming from: peak memory set by the concurrency limit and the bound rather than by the size of the job. At a few hundred items with small payloads the difference is noise. At ten thousand items with real payloads, or anywhere the producer is a stream that should be slowed down rather than drained, the fixed footprint is the reason to prefer the pool.

Two other shapes get reached for before either of these, and both are worth ruling out concretely rather than dismissing.

The first is to skip the cap entirely: create all ten thousand tasks and let the provider's rate limiter shed whatever it cannot take. What that produces is ten thousand requests arriving within a few hundred milliseconds, the provider accepting whatever fits under its ceiling and rejecting the rest, and every rejection turning into a retry that lands as a second burst on top of traffic that was already over the limit. Two things make this different in kind from being merely slower. The limit being blown is the account's, not this process's, so an unrelated service sharing that key starts receiving 429s for requests it made at a perfectly reasonable rate, and the incident belongs to a team that changed nothing. And the work never got bounded at all: ten thousand coroutines and their payloads are resident the whole time, and a provider's rejection is not a queue — a rejected ticket is a ticket that has to be sent again, not a ticket waiting its turn.

The second is chunking: slice the ten thousand tickets into five hundred batches of twenty and gather each batch in turn. Grant what it gets right, because it gets two real things right — concurrency never exceeds twenty, and memory stays small, since only twenty coroutines exist per batch. The property it does not share shows up on the chapter's own latency figures. A gathered batch is finished when its slowest member is finished, so if nineteen tickets return in 700 milliseconds and one takes four seconds, nineteen slots sit idle for more than three seconds waiting for the batch to close, and no ticket from the next batch can start during that wait. Repeat that across five hundred batches and the idle time rather than the work is what sets the wall clock. Neither the semaphore version nor the worker pool has that behaviour: in both, a slot that finishes takes the next ticket immediately, and one slow call delays exactly one slot.

The pool leaves its own limit open. Twenty workers do not become thirty when the queue backs up, so a burst still waits — it waits in the queue instead of at a semaphore, which is a difference in memory and in backpressure, and not a difference in queue time.

Blocking calls#

Worker pool design rests on an implicit assumption that has gone untested until now: everything so far has assumed that whatever a ticket handler does eventually reaches an await and hands control back. Nothing in the async def keyword guarantees that. The next hazard is a call inside a coroutine that never yields at all.

Ticket processing grows a step: some tickets arrive with a PDF attached, and its text has to be pulled out and appended to the ticket body before the summary is any use. Topic 9.3 owns what PDF extraction does. What matters here is that the library doing it is synchronous and takes roughly 800 milliseconds on a typical attachment.

async def handle_ticket(request: Request) -> Response:
    ticket = await load_ticket(request.ticket_id)
    summary = await summarize_ticket(ticket)
    return Response(summary=summary)

async def handle_ticket(request: Request) -> Response:
    ticket = await load_ticket(request.ticket_id)
    ticket.body += extract_text(ticket.attachment)
    summary = await summarize_ticket(ticket)
    return Response(summary=summary)

One line different. extract_text is an ordinary function called in the ordinary way, and calling an ordinary function inside a coroutine simply calls it.

That line is a blocking call: a synchronous operation that occupies the thread running the event loop until it returns, during which no other coroutine on that loop can run.

The cost is not the 800 milliseconds on this ticket. It is 800 milliseconds on every other request currently on that loop. Suppose thirty handlers are in flight when this one enters extract_text. Some are waiting on the database, some on the provider, and one of them has had its response sitting in the socket buffer for a while and needs one more turn of the loop to serialise it and return. All thirty stay exactly where they are for the full 800 milliseconds, because there is no point inside a synchronous call at which the loop regains control and can pick something else — the loop's thread is inside extract_text, and the loop is not a thing that runs alongside its coroutines, it is the thing whose thread they borrow. Health checks scheduled during that window fire late. Timeout timers scheduled during that window fire late. The request that was one turn from finishing waits out the whole extraction.

The runtime does nothing about this by default, and it is worth knowing exactly what the one available mechanism is, because it is easy to over-read. Asyncio has a debug mode, enabled with asyncio.run(main(), debug=True) or the PYTHONASYNCIODEBUG environment variable, and what it does is log a warning of the form Executing <Handle ...> took 0.800 seconds when a callback overruns. Three limits make that a diagnostic rather than a rescue. It is off by default, so a production service that has not opted in reports nothing. The warning is emitted after the callback returns, so the 800 milliseconds of frozen handlers has already happened by the time anything is logged. And the loop takes no corrective action of any kind — it does not preempt the call, does not run anything else during it, and resumes afterwards as though nothing had occurred.

The reason this survives code review is in the exhibit. The function is still declared async def, still awaits two things, still sits in a file where everything is async, and the added line looks like every other line of ordinary Python in the codebase. Nothing in the syntax announces that the body has stopped being cooperative.

Event loop lag#

The symptom lands on requests that have nothing to do with PDFs, which raises the question of how anybody would notice it in a running service. Event loop lag is the delay between when a scheduled callback was due to run and when it actually runs, used as a signal that something is blocking the loop.

Measuring it needs nothing but a coroutine that expects to be woken on time.

async def heartbeat(interval: float = 0.1) -> None:
    expected = time.perf_counter() + interval
    while True:
        await asyncio.sleep(interval)
        now = time.perf_counter()
        lag = now - expected
        if lag > 0.05:
            logger.warning("event loop lag: %.3fs", lag)
        expected = now + interval

The task asks to be resumed every hundred milliseconds and records how much later than that it actually was. On a healthy loop the gap is a millisecond or two of scheduling overhead, because resuming a sleeping coroutine is nearly free and the loop reaches it promptly. Against the handler with the direct PDF call, the gap spikes toward 800 milliseconds, and it spikes during exactly the windows when a request carrying an attachment is being handled.

The measurement points at the loop rather than at anything upstream because of what it cannot be made late by. A slow provider call does not delay it at all: the handler waiting on that call is suspended, the loop is free, and the heartbeat is resumed on schedule whether the provider takes 700 milliseconds or 40 seconds. The only way to make a sleeping coroutine late is to occupy the loop's thread past the moment it was due, and the only thing that occupies the loop's thread is code that does not yield.

Three other numbers get reached for first, and the point of this one is that it discriminates where they do not.

The first of these three numbers is a count of concurrent in-flight tasks; it rises during a stall, because requests keep arriving and none of them complete while the loop is frozen. It rises identically when twice as many tickets simply showed up and everything is working correctly. The number is the same number in both cases and carries nothing that tells them apart.

The second of these metrics, CPU utilisation, fails in both directions. A loop pegged at one core is what a CPU-bound stall looks like, and it is also what a healthy service doing legitimately compute-heavy work looks like, so a high reading is not evidence. And a synchronous call that sits waiting on a disk read or a socket read blocks the loop just as completely with the CPU near idle, so a low reading is not evidence either.

Request p99 latency is the most tempting and the least useful of the three. It climbs during a stall — but it also climbs when the provider gets slower, when permits are scarce, and when arrivals outpace the configured concurrency, and each of those has a different fix. Worse, it is measured on the request, so it cannot distinguish time a request spent on its own work from time it spent frozen while some other request refused to yield. In this incident it actively misleads: the requests whose p99 moves most are the ones with no PDF in them, because they are the ones doing least and therefore the ones for which 800 milliseconds of freeze is the largest proportion of their total. Optimising the endpoint whose number moved is a week spent on the innocent handler.

Event loop lag is the one of the four that only moves for the thing being looked for.

What it gives back is narrow, and worth stating so nobody expects more from it. It says the loop stalled, roughly when, and for how long. It does not name the line, and finding that still means reading what was running during the window it points at.

The Global Interpreter Lock#

The instinctive fix is to move the blocking call onto a thread. Before that means anything, it matters what a thread actually buys in CPython, and that depends entirely on one piece of machinery: the Global Interpreter Lock, the lock in CPython that allows only one thread to execute Python bytecode at a time, regardless of how many threads exist.

Read carefully, that rules out less than it appears to. Two Python threads cannot execute Python bytecode at the same instant, so eight threads running a pure-Python loop take about as long together as they would one after another. But a thread holds the lock only while it is running bytecode, and it releases the lock whenever it is waiting on something outside the interpreter — a blocking read from a file or a socket, a call into a C extension that has explicitly given the lock up for the duration of its own work. During those intervals the thread is doing something real and another thread is free to run Python. That release-while-waiting behaviour is the whole reason moving a blocking call to a thread can free the event loop even though only one thread of Python bytecode ever runs at a time: the loop's thread is running bytecode, and the offloaded call is not.

It also means the answer for the PDF extractor is not yet settled. Whether that specific synchronous library releases the lock while it does its work is the question the fix depends on, and it is not something the lock's definition can answer.

Thread pool executors and process pools#

To answer both halves of that GIL question — keeping the loop free, and getting the PDF work to actually overlap — there are two tools, and it is worth naming what they share before separating them. A thread pool and a process pool both move work off the thread the event loop runs on, so that thread stays free to keep scheduling coroutines. Both are reached through the same kind of await, both return the result to the calling coroutine as though it had been computed locally, and both leave the loop responsive while the work happens elsewhere.

A thread pool executor is a fixed set of operating system threads used to run a blocking call off the event loop so the loop itself stays free. Put the PDF handler from earlier through one and a single line changes.

    ticket.body += await asyncio.to_thread(extract_text, ticket.attachment)

The same handler, the same extractor, the same attachment: only that one line differs from the direct call. asyncio.to_thread hands the function and its arguments to the loop's default executor and returns an awaitable, so the handler suspends there and the loop's thread comes straight back to scheduling. Run the heartbeat prober against this version and its gap stays near a hundred milliseconds through requests that carry attachments, because the extraction is no longer happening on the thread the heartbeat needs.

That answers the unrelated-requests problem and leaves the question the lock section opened. Whether this also makes PDF extraction itself faster to run many of at once depends on whether the library releases the lock during its own work, and that is checkable rather than guessable.

t0 = time.perf_counter()
extract_text(sample_pdf)
single = time.perf_counter() - t0

t0 = time.perf_counter()
with ThreadPoolExecutor(max_workers=8) as pool:
    list(pool.map(extract_text, [sample_pdf] * 8))
eight = time.perf_counter() - t0

If eight comes out near eight times single, the library held the lock throughout and the threads ran one after another. If it comes out near single, the library released the lock and the threads genuinely overlapped. Anything in between says it released the lock for part of its work. The same three-line comparison answers the same question for any library, and it is worth running rather than inferring from what the package is written in, because a wrapper around a C library can still spend most of its time in Python.

That is the whole of the fix stated abstractly, and it is worth landing it in the form the problem actually arrives in, because the diagnosis is the hard part. An endpoint serving 300 requests per second at a 40-millisecond p99 degrades to a 1.2-second p99 the week after the PDF step ships, and the degraded requests are mostly ones with no PDF in them. That shape is the whole clue — degraded requests that never touch a PDF are the shared thread showing itself in the p99 — and proving it rather than inferring it is one measurement: run the heartbeat prober and correlate its spikes with the requests that carry attachments.

Then answer honestly whether to_thread finishes the job, because it finishes one half of it. It does end the damage to unrelated requests: the loop's thread is free the instant the call is handed off, so a request that touches no PDF is no longer paying for one. It does not give the PDF step throughput. The default executor holds a bounded number of threads, so once concurrent PDF requests exceed that count they wait for a thread, which is the queue time from earlier arriving in a new place and invisible in the same way. And if the check above shows the extractor holding the lock, those threads execute one at a time regardless of how many there are, so the PDF work itself proceeds at exactly the rate it did before. When both of those bite, the honest fix is a process pool rather than a larger thread pool.

That case earns a second workload, because PDF extraction has a network call on either side of it and the argument is cleanest on work that has none. Somewhere in the same system, a job counts tokens across two million archived documents. Topic 1.1 owns what tokenizing is; here it is a workload whose cost scales with document count and which never leaves the process.

A process pool is a set of separate worker processes, each with its own interpreter and its own lock, used to run CPU-bound Python work in true parallel.

2,000,000 documents, 4 ms of pure-Python tokenizing each
  sequential baseline, one thread ................. 8,000 s

  500 asyncio tasks, one event loop ............... 8,000 s + task overhead
  ThreadPoolExecutor, 8 threads
      tokenizer holds the lock .................... 8,000 s
      tokenizer releases the lock ................. ~1,000 s
  ProcessPoolExecutor, 8 workers .................. ~1,000 s + serialisation

Each row follows from something already established. The asyncio row is the one that surprises people: count_tokens contains no await anywhere, so a task running it runs to completion before the loop can switch to the next one, and five hundred tasks on one loop execute one after another exactly as the sequential baseline does. The direction of that result is worth being exact about, because it is not a wash. Creating five hundred task objects, scheduling them and settling them is real work the plain loop never does, and none of it buys any overlap back, so the row is the sequential time plus a small overhead penalty. Asyncio applied to CPU-bound work costs little and never pays.

The thread pool row splits on the lock, which is why it is two rows. A pure-Python tokenizer holds the lock for the entire count, so eight threads take turns and finish in about the time one thread would. A tokenizer whose inner loop is C or Rust and gives the lock up during that loop lets the eight threads genuinely overlap, and the row collapses toward an eighth of the baseline.

The process pool row does not split, because each worker is a separate interpreter with its own lock and eight of them execute eight bytecode streams at once whatever the tokenizer is written in. Its cost is at the boundary. Every document handed to a worker has to be serialised, sent through a pipe, and deserialised on the other side, and every result comes back the same way. Put a number on it against the tokenizing: if serialising one document costs a millisecond against four milliseconds of counting, then sending documents one at a time spends a fifth of the run crossing process boundaries, and at two million documents that is over half an hour of pure overhead. Batching — handing each task five hundred documents rather than one — amortises one crossing over five hundred documents and is the direct answer to that cost.

Throughput#

To compare those strategies against one another — the process pool's serialisation cost against the thread pool's conditional gain against asyncio's near-free overlap — we need one measure that holds across all three. Throughput is the number of operations a system completes per unit of time, used to compare concurrency strategies running the same workload.

It is the quantity both worked exhibits were measuring, and naming it makes what they showed comparable. Wall-clock time for the ticket batches moved with the number of concurrent tasks, because the work was waiting. Wall-clock time for the token count did not move with the number of asyncio tasks at all, and moved with the number of processes — or of threads, conditionally on the lock — because the work was computing.

As a rule for choosing a strategy, that reads in three lines. Asyncio concurrency raises throughput on work that spends its time waiting on something outside the process, such as a chat API call or a database round trip, for the reason the ticket batches showed: there is waiting there to overlap. A thread pool raises throughput on blocking or CPU-bound work under the condition the extractor made concrete, that the code running inside it releases the lock. A process pool raises throughput on CPU-bound work regardless of what the code does, at the cost of serialising data across the process boundary.

Run that rule out loud on the two workloads picked in the same breath, since the question arrives that way: one service that has to make 200 concurrent calls to a hosted model API, and one job that has to count tokens across two million documents. The 200 calls go to asyncio with a bounded fan-out, and the fact that decides it is that every one of them spends its time waiting outside the process. The token count goes to a process pool, and the fact that decides it is that none of them does. The primitive follows from where the time goes, not from which library is fashionable.

The follow-up worth answering is whether knowing which tokenizer is in use changes either choice. It changes neither. A coroutine with no await in it is a coroutine with no await in it whatever the tokenizer is, so asyncio has nothing to interleave in the second job either way, and a process pool sidesteps the lock either way. What the tokenizer changes is whether a thread pool is a legitimate cheaper alternative for the second job: one with a C or Rust inner loop that gives the lock up can reach process-pool throughput without paying to serialise two million documents, and a pure-Python one cannot. That is a fact to establish by running the lock release check against the specific library rather than to assume from its reputation.

And the measurement that ties the whole rule together is event loop lag, because it is the one signal that separates the case a bigger semaphore or more tasks can help — a genuinely slow upstream call — from the case where only moving the work off that thread will do anything.

Choosing and sizing a concurrency strategy#

Everything above is a set of facts about where time goes and what holds what. Turning them into a decision needs the facts asked in an order, against a specific workload, with real numbers in the places where the arithmetic needs them. ArjanCodes' Next-Level Concurrent Programming In Python With Asyncio works the same decisions against real service code, which is the form in which they are easiest to copy.

Two things in there are deliberately not decided for you. The value of limit comes out of your own target rate, your own measured latency and your own provider's published ceiling, and copying 20 or 50 from these pages gives you a number with no derivation behind it. So does the choice between a semaphore and the worker pool above, which turns on the size of your item count and your payloads rather than on which is better.

Both of those undecided choices are choices between mechanisms, and the mechanisms come down to three moves: running many waiting operations at once, bounding how many of them run at once, and recognising and relocating the moment something stops waiting cooperatively. Each has a mechanism behind it and each has a cost, and the costs are what decides where they apply.

They apply narrowly and they apply hard. Asyncio concurrency earns its complexity on I/O-bound fan-out with real concurrency to exploit — many independent calls to something slower than the code making them — where the difference between 350 seconds and one is available for the price of a rewrite. On a CPU-bound loop, as the token-count timings showed, there is nothing there to overlap, and a process pool is the tool with a mechanism to justify it.

What none of this settles is the numbers. The semaphore size, the worker count, the serialisation overhead, whether a particular library gives the lock up during its inner loop: every one of those is specific to a workload, a payload size and a provider limit, and the arithmetic here shows how to derive them rather than what they are. Measured on your own system, they will be different numbers reached the same way.