Concurrency limits and queue time

A semaphore is a queue with N servers, so the number in Semaphore(20) sets a service rate, and once arrivals exceed it the excess shows up as p99 latency that has nothing to do with the provider. This chapter covers how to split semaphore wait from provider latency in one metric, why retrying with backoff while still holding a slot removes servers rather than adding load, and how to derive the limit from a provider's own rate-limit headers.

Async and concurrency in Python — read the current version at https://daviddominguez.dev/learn/en/async-and-concurrency-in-python/rate-limit

Back to top