Eight concurrent branches, 1.4x faster: the quota is queueing

The contract reviewer used to be one call: a 6,000-token rubric with four worked examples, a 30,000-token agreement, and a 4,400-token report covering eight review dimensions — indemnity, intellectual-property assignment, termination, payment terms, data protection, liability caps, warranties, governing law. It took 92 seconds, which was long enough that the product team had built a progress spinner with encouraging messages. The rewrite gave each dimension its own branch, each branch the whole agreement and only its own slice of the rubric, and launched all eight with asyncio.gather. Per-dimension quality went up. Wall-clock time went from 92 seconds to 66.

That is 1.4×. Eight branches, each generating about 700 output tokens instead of 4,400, should land somewhere near 13 seconds — a 7× improvement — and the team's first explanation was the one everybody reaches for: the slowest branch dominates, parallel time is the maximum rather than the mean, and the indemnity section is much longer than governing law. It is a real effect and it is the wrong explanation here, which is worth establishing before anything else, because accepting it sends people off to rebalance section sizes for a week.

Tail domination cannot produce 1.4x#

The way to rule an explanation out is to work out what it would have to imply and check whether the implication is true.

If eight branches truly run at the same time, wall-clock time is the duration of the slowest one plus a small amount of coordination. Getting 66 seconds out of that requires one branch to take 66 seconds, so the question becomes what a branch would have to be doing for that long.

A branch's duration has two parts. Each branch reads the same 36,000 input tokens, so prefill — the pass over the input tokens that a model makes before any output appears — is identical across them and cannot be the variable. What remains is generation, and generation time is the number of output tokens divided by the rate at which they are produced. The single-pass version supplies that rate: it wrote 4,400 tokens in 92 seconds, of which roughly two seconds was prefill, so the generation rate is about 49 tokens per second.

Now invert it. A branch occupying 66 seconds spends all but the prefill generating, which is sixty-four seconds, and sixty-four seconds at 49 tokens per second is around 3,100 output tokens. That is 70% of everything the original report contained, produced in one dimension, while the other seven branches split the remaining 30% between them.

That is not what the outputs looked like. The longest branch, indemnity, produced 1,190 tokens and the shortest, governing law, produced 310. Feed the real distribution back through the same arithmetic — the longest branch sets the wall clock, so 1,190 tokens at 49 tokens per second plus prefill — and the predicted wall clock is about 26 seconds, which is 3.5× rather than 1.4×.

The prediction can be pushed further before it stops helping. Suppose the split had been deliberately pathological, with one branch carrying three times the mean share of the original report's output. That branch's duration still comes out around 34 seconds, and a 2.7× improvement. Tail domination has a floor, and the floor is well above what was measured. Whenever the observed speedup is worse than the worst uneven split you could plausibly have produced, the branches were not all running.

Measure the concurrency you achieved#

The framework's tracing showed all eight tasks created within four milliseconds of each other, which is what convinced the team the fan-out was working. Task creation is not request admission, and the difference is not a subtlety — it is two different systems. An asyncio.Task exists the moment you schedule the coroutine, which is a fact about your own process. Whether the provider has started generating is a fact about the provider, and the cheapest proxy for it is time to first token, because the first content byte cannot arrive before generation has begun.

@dataclass
class BranchTrace:
    name: str
    submitted_at: float   # perf_counter() before the SDK call
    first_token_at: float # perf_counter() on the first content delta
    finished_at: float
    retries: int          # from the SDK's retry hook

async def run_branch(name: str, prompt: str, traces: list[BranchTrace]) -> str:
    submitted = time.perf_counter()
    first = None
    chunks: list[str] = []
    async with client.messages.stream(
        model=MODEL, max_tokens=1500,
        messages=[{"role": "user", "content": prompt}],
    ) as stream:
        async for text in stream.text_stream:
            if first is None:
                first = time.perf_counter()
            chunks.append(text)
    traces.append(BranchTrace(name, submitted, first, time.perf_counter(), retries_for(name)))
    return "".join(chunks)

The eight time-to-first-token values from one traced request, in seconds after the fan-out started, were 0.9, 1.0, 14.2, 14.4, 27.6, 27.9, 41.1 and 41.3. The single-pass version's time to first token was 0.9 seconds.

Read the shape rather than the values. They arrive in pairs, and the pairs are separated by long, roughly equal gaps. Contention inside your own process does not look like this: an event loop that is oversubscribed or a client that is serialising produces a staircase with one step per branch, because each branch is released as the previous one finishes. Steps of two, with nothing at all in between, means two branches were let through and the rest were turned away — which is what an admission control system looks like. The SDK's built-in exponential backoff, which waits a doubling interval before each fresh attempt, absorbed the refusals so silently that nothing reached the application log.

Two properties of that backoff explain why the failure is invisible rather than merely subtle. It retries inside the client, so the refusal is consumed before any code you wrote sees a response object, and an application log records what the application was told rather than what crossed the wire. And it waits longer on each successive attempt, so a branch turned away early keeps sleeping through windows in which budget was briefly free — which is why the gaps between waves are long and roughly equal rather than tightening as the queue drains. Neither of those properties belongs to contention you can watch inside your own process.

There is a single number that summarises this, and it belongs on the dashboard next to latency. Achieved concurrency is the total time branches spent generating, divided by the wall clock of the fan-out. The reason that ratio means what it does is worth one line of reasoning: if two branches generate for the whole of a window, the window accumulates two seconds of generation per second of wall clock, so the ratio is the average number of branches live at any instant. Sum the eight finished_at - first_token_at intervals — about 104 seconds of generation — and divide by the 66-second wall clock, and you get 1.6. You asked for eight and you got between one and two.

The choice of start point is what makes the number honest, and getting it wrong inverts the answer. You must use time to first token as the start, not submission time, because a request sitting in the SDK's backoff loop still has an open span. Counting from submission credits a branch with every second it spent waiting to be admitted as though it had been generating, and doing that here would report a 41-second duration for a branch that generated for 13, making achieved concurrency look like 7.9 — which is the number that would have confirmed the team's original belief.

Why a quota bites here and never bit before#

The account's limit was 400,000 input tokens per minute, shared across every service on the same key, and two other features were drawing roughly 135,000 of it during business hours.

Work out what the single-pass reviewer drew against that. It consumed 36,000 input tokens spread across a 92-second request, and a rate limit is measured per minute, so the comparable figure is 36,000 scaled to a minute, which is an average draw of about 23,500 tokens per minute. That is comfortably inside the headroom, and low enough that in eighteen months nobody had seen a 429, which is the status code a provider returns when it refuses a request for exceeding a limit rather than because anything about the request was wrong.

The fan-out did not change the total by much in the terms anybody was watching. Eight branches at 36,000 input tokens each is 288,000 tokens, against 36,000 before, so total volume went up eightfold — which is a lot, and still inside a per-minute budget of 400,000 if it were spread over a minute. What changed catastrophically was the peak. Those 288,000 tokens were submitted inside a 40-millisecond window rather than trickled over 92 seconds. Token-per-minute limits are enforced against an instantaneous budget rather than a trailing average, so a burst that would be entirely affordable spread over a minute gets refused when it arrives at once.

That is the step worth holding onto, because it is the one that makes the failure counter-intuitive. The quantity in the limit's name is a rate, and a rate computed over a minute would have been satisfied. The quantity actually enforced is a level: how much budget is free at the instant a request arrives. A design can therefore move from comfortably compliant to systematically refused without changing its per-minute consumption at all, purely by changing when within the minute the tokens are presented. The limiter admitted what fitted in the available headroom, refused the rest, and the retry logic converted the refusal into latency instead of an error.

The two 429 rows are the pair that matters, and the fact that they differ is the whole diagnostic. The application log shows zero refusals whether there were none or eleven, because the SDK retried before the application ever saw a response — so the quantity a team watches stays reassuringly constant while the property it is meant to indicate varies from fine to catastrophic. The 11 absorbed refusals are consistent with the staircase: six branches turned away on the first attempt, fewer on each subsequent wave as the earlier ones finished and freed budget.

They are also usually one configuration change away from being visible. Most SDKs expose the raw response or a retry callback, and every provider returns the remaining budget in response headers — a rate-limit remaining count for requests and one for tokens, plus a reset timestamp. Logging those three values on every response turns this entire investigation into a glance at a graph. Until you do, the failure mode is precisely that nothing appears to be wrong.

Sizing the admission window arithmetically#

Fixing the queueing is a capacity conversation rather than a code change. Requesting a higher limit, moving the fan-out to a dedicated key, staggering submissions so the burst is spread across two or three seconds, or admitting branches through a semaphore, a counter that caps how many requests are allowed in flight at once, sized to the measured headroom will all recover most of the missing speedup, and staggering has a second benefit that the next chapter puts a price on.

Sizing that semaphore is worth doing arithmetically rather than by trial. The headroom is the account limit minus what everything else on the key draws, so 400,000 minus 135,000 leaves 265,000 input tokens per minute. Each branch consumes 36,000 of them at admission — at admission rather than over its lifetime, because the limiter charges the whole prompt when the request is accepted. Dividing the headroom by the per-branch draw gives the sustainable number of branches in flight, which is around seven, and five once you leave margin for the other two services' own bursts, which are not smooth either.

A semaphore of five turns the eleven silent retries into zero and lands the wall clock near 30 seconds, and the reason is a change in the shape of the schedule rather than in the speed of anything: the work now runs as one wave of five and one of three rather than four waves of two, so the wall clock is two branch durations instead of four. The number is only valid for the traffic mix it was computed against, so it has to be recomputed whenever another service on the key changes volume — which is a stronger argument for a dedicated key than it is an argument against semaphores.

The other things that erase concurrency#

A rate limit is the most common cause, and it is not the only one, so it is worth knowing what the alternatives look like on the same instrumentation.

A blocking client inside an async fan-out serialises perfectly rather than partially. If the branches call a synchronous SDK method from inside a coroutine without handing it to a thread pool, each call blocks the event loop until it returns, and the eight branches execute one after another for a speedup very close to 1.0×, or slightly worse than 1.0× once you add overhead. The time-to-first-token trace shows a staircase with a step per branch rather than per pair, and the retry count is zero. That distinction — steps of one with no retries, against steps of two with retries — separates the two causes without any further work.

A connection pool cap produces a pattern that looks almost exactly like a rate limit, and it is worth ruling out early because it is trivially fixable. HTTP clients ship with a default maximum number of connections, and if the SDK was constructed with a shared client configured for a modest pool, branches queue on the client rather than at the provider. The give-away is that the retry count is zero here too: nothing was refused, the requests simply never left.

Shared mutable state is the fourth possibility, and it damages correctness rather than speed. This is the version that shipped and was caught in review:

# One list, eight branches. Branch 8 sends all eight questions.
messages = [{"role": "system", "content": RUBRIC}]

async def branch(dimension: str) -> str:
    messages.append({"role": "user", "content": prompt_for(dimension)})
    resp = await client.messages.create(
        model=MODEL, max_tokens=1500, messages=messages,
    )
    return resp.content[0].text

The defect is in the first line rather than in the call. messages is defined once, outside the coroutine, so all eight branches append to the same list object. Because the branches interleave at every await, the list each branch sends depends on how many other branches happened to have appended by the time it serialised its request. The output is not an error; it is eight plausible reports, some of which answer several questions at once, and the per-dimension evaluation scores them as slightly degraded rather than broken.

The fix is that every branch builds its own message list from a shared immutable prefix, which is one line, and the reason to raise it in a chapter about throughput is that the same review that measures achieved concurrency should be the review that checks whether anything is shared and writable across branches. A shared retrieval client with a stateful cursor, a shared token accumulator, and a shared memoisation dict keyed on something that is not branch-specific all belong on the same list.

None of it touches money. The same eight-way fan-out that failed to get faster also produced a bill 8.6 times the single-pass version, and the team assumed one cause was behind both numbers, since both showed up in the same week and both are about the fan-out.

They are separate, and the arithmetic above is enough to see why. Queueing costs latency and costs nothing extra, because a refused request is never processed and a request that is not processed is not billed — the eleven 429s in the table have no price attached to them at all. The bill went up for reasons that have to be enumerated line by line, and the enumeration includes at least one item that a straightforward multiplication of the per-call price cannot contain.