Diagnosing effective concurrency

The division is arithmetic anybody can do in their head, and it settles the question that usually gets answered with a capacity request. Two hundred requests in flight, each completing in 0.8 seconds, produce 200 / 0.8 = 250 items per second. A job in that configuration doing 12 items a second is running at 4.8% of what its own settings should deliver, and no amount of extra capacity fixes a client using 4.8% of the capacity it already has.

Twelve items a second is also where the run time comes from, which is worth checking because the run time is the number people arrive with. Ten thousand items at 12 a second is a little over eight hundred seconds, and eight hundred and thirty-odd seconds is 13 minutes 53. The 4.8% is the other division, 12 observed against 250 expected. Neither figure needs instrumentation that is not already present — a count of items written and a timer around the HTTP call are the whole apparatus — which is what makes this the first thing to do rather than the thing to do once a profiler has failed.

Running it the other way is the more useful form, because it turns an observation into a number that can be compared against a config file:

effective_concurrency = throughput * service_latency
                      = 12 items/s * 0.8 s
                      = 9.6 requests actually in flight

Two hundred tasks exist and about ten requests are outstanding at any instant. Something between the task and the socket is admitting ten at a time, and the investigation is now a search for what that something is.

Why the second pod makes it worse#

If each process is internally serialised at ten concurrent requests, a second process gives twenty, so throughput goes from 12 to 24 items per second and a job that took 13 minutes 53 takes 6 minutes 57. Doubling the infrastructure bought a real 2× improvement against a target that is 20× away, which is the specific reason the proposal survives review: it works, measurably, and it works in the direction that makes the defect permanent.

The cost is more than the second pod's invoice. The team now believes the workload is horizontally scalable and has calibrated its capacity planning against a client running at 5% efficiency, so the next volume increase is answered with four pods and the one after that with eight. Sixteen pods and a 5% efficient client is 192 items per second, still short of what one correctly written process delivers, and by then the arithmetic has been buried under a scaling policy that appears to work. The version that is genuinely hard to recover from is a shared constraint: if the ten-at-a-time limit comes from a connection pool in a sidecar, a shared proxy, or a lock in a library both pods import, the second pod adds nothing at all — and if it adds a little, the improvement is enough to close the investigation without explaining the twenty-fold gap.

The trajectory is hard to interrupt because every step of it is confirmed by measurement. Ten thousand items at 24 a second really is 6 minutes 57, so the second pod delivered exactly what was predicted. Sixteen pods at 12 items a second each really is 192 a second, an improvement over every previous fleet size, and it is still under the 250 that one correctly configured process delivers on its own. Nothing in that sequence of observations ever contradicts the theory that the workload is simply large, which is why the arithmetic has to be done before the first pod is added rather than after the fourth.

Measuring what is actually in flight#

The effective concurrency computed above is an inference. Measuring it directly costs one gauge, and it converts an argument into an observation:

IN_FLIGHT = Gauge("provider_requests_in_flight", "Requests outstanding at the transport")
SERVICE = Histogram("provider_service_seconds", "Time inside the HTTP call only")


@contextlib.asynccontextmanager
async def tracked_call() -> AsyncIterator[None]:
    IN_FLIGHT.inc()
    started = time.perf_counter()
    try:
        yield
    finally:
        SERVICE.observe(time.perf_counter() - started)
        IN_FLIGHT.dec()


async def classify(client: httpx.AsyncClient, item: Item) -> str:
    async with GATE:
        async with tracked_call():
            response = await client.post(PROVIDER, json=prompt_for(item))
    response.raise_for_status()
    return response.json()["content"][0]["text"]

The gauge must wrap the HTTP call alone rather than the semaphore acquisition, because a gauge that includes queue wait counts coroutines waiting for permission, and those are not in flight anywhere. With the gauge inside and the semaphore outside, a healthy job sampling provider_requests_in_flight shows a value pinned near the configured bound.

The same placement rule applies to the latency in the division, and getting it wrong makes the whole diagnostic useless. Little's law relates the number in a system to the arrival rate times the time spent in that system, and it holds as an identity — so measuring latency end to end, including the time an item spent queued for a slot, returns the declared concurrency no matter how badly the job is broken. The gap only appears when the latency term is service time: the duration of the HTTP call and nothing else. That is why SERVICE is observed inside tracked_call and not around the whole function.

Why the end-to-end version returns the configured number is worth seeing rather than accepting, because a diagnostic that returns the same answer under every condition gets abandoned rather than fixed. Suppose 200 tasks each loop forever: take a slot, make a call, release it. Every one of those tasks is inside the system at every instant, so measured end to end the average number inside is 200 by construction, whatever the client is or is not doing. Little's law then forces throughput multiplied by end-to-end latency to equal 200, as an identity — and an identity cannot disagree with anything, which is why the measurement never reveals a fault. Restricting the latency term to service time breaks the tautology, because a task waiting for a slot is inside the loop and outside the HTTP call, and the difference between the two numbers is precisely the queueing being hunted.

Six things that serialise a fan-out#

An effective concurrency of about ten against a declared two hundred, with low CPU and no rejections, narrows the field considerably. Each candidate produces a characteristic effective concurrency, and each has a measurement that confirms or eliminates it in a couple of minutes.

The last row is the one that exonerates the client, and it is worth stating because the chapter is otherwise about blaming it. If the gauge sits at 200 and the service histogram shows 16 seconds, the client is doing exactly what it was told and the provider is the constraint, at which point the conversation is about quota, model choice or prompt size.

Low CPU does real work in narrowing the rest. A process running 200 genuinely concurrent calls, parsing 250 JSON responses a second and building 250 request bodies, would sit meaningfully higher on one core, so a reading of 15% is consistent with a process that is mostly idle and waiting. It also argues against a CPU-bound blocking call — a tokenizer or a PDF parser running on the loop would show a busy core and fire loop-lag warnings. What low CPU does not rule out is an I/O-blocking synchronous call, because a thread parked in requests.post or a synchronous database driver consumes no CPU while it holds the loop hostage. That is why the third row is confirmed by loop lag rather than by CPU: enabling asyncio debug mode and reading Executing <Handle ...> took 0.812 seconds in the log distinguishes a stalled loop from an idle one, and neither shows up as CPU.

Two more measurements are worth taking before concluding, because they cost nothing and each closes off a whole class of explanation. Sampling the in-flight gauge over the run says whether effective concurrency is stable at ten or oscillating between 200 and 0, and oscillation means batching somewhere — a producer that reads 200 items, waits for all of them, and reads 200 more, which is a different bug with the same average. And comparing the service histogram against the provider's own reported latency for the same window separates queueing inside your process from queueing at the provider's edge, which is the only way to be certain which side of the socket the missing time is on.

Where the division stops being reliable#

The arithmetic assumes every call has roughly the same service time, and it degrades when the workload is a mixture. A job where 90% of items take 400 ms and 10% take 12 seconds has a p50 of 400 ms and a mean of 1.56 seconds, and dividing by the p50 will report the job running at a quarter of its capacity when it is running correctly. Use the mean service time for this division and keep the percentiles for latency questions.

That mean is one weighted sum. Nine tenths of the items at 400 ms contribute three hundred and sixty milliseconds between them, one tenth at 12 seconds contributes twelve hundred, and the two add to the 1.56 seconds quoted — the long tail dominating a mean it makes up a tenth of. The p50 stays at 400 ms because nine items in ten really are fast, so the p50 and the mean differ here by close to a factor of four, and dividing throughput by the wrong one moves the answer by that same factor. That is the whole mechanism behind a healthy job appearing to run at a quarter of its capacity. The rule that falls out is narrow rather than general: percentiles answer questions about what one item experienced, this division is a question about total work, and total work needs the mean.

Streaming responses break it differently, because the useful notion of service time becomes ambiguous: a slot is held from the request until the last token, which can be twenty times the time to the first token. A streaming call that takes 400 ms to first token and 9 seconds to completion occupies concurrency for 9 seconds, so the division has to use the full duration even though every dashboard the product team looks at reports the 400 ms.

The measurement also has to survive the move to multiple processes, which means the in-flight gauge needs to be summed across replicas rather than read from one. A gauge scraped per pod and summed gives fleet-wide concurrency, and comparing that sum against the fleet-wide bound — per-process bound times replica count — is the same check at a different scale. It is the check that catches an autoscaler quietly tripling concurrency against a provider that was sized for the original replica count.

One caution belongs alongside those, and it concerns what the division cannot tell you even when it comes out right. Effective concurrency matching the configured bound says the client is issuing what it was told to issue. It says nothing about whether the bound is the right bound. A process running exactly ten genuinely concurrent calls against an undersized pool and a process running exactly ten against a correctly sized one are indistinguishable to this measurement, because both are saturated at ten. The division locates a bottleneck inside your process or rules one out; deciding what the number ought to have been is the work of the first three chapters of this topic, and the two questions want different instruments. Further refinements to the measurement — sampling the gauge over the run, comparing against the provider's own reported latency — sharpen the first question and never touch the second.

The ordering that falls out of all this generalises past any one bug. Scaling out multiplies whatever efficiency a single process has, so it is worth doing after the per-process arithmetic checks out and expensive before it. A team that measures effective concurrency before every capacity decision spends its infrastructure budget on real constraints; one that does not ends up with a fleet whose size encodes the history of its bugs.

Everything so far has assumed the work has a place of its own to run in — a batch process whose only job is to make these calls. The next failure appears when the same fan-out is written inside an HTTP request handler, where the concurrency bound and the pool are shared with every other endpoint in the service and the deadline belongs to a load balancer.