Splitting the work and repeating the work are different bets

Parallelisation is answering one incoming request with several model calls that run at the same time, rather than with a single call that does all of the work in sequence. It exists because one call is a bottleneck twice over, and both halves of that are worth walking through, because they are the reason every measurement later in this topic is taken.

The first is latency. A model writes its answer one token at a time, and each token is chosen using the tokens already written, so the second cannot be produced before the first exists. That ordering is a property of how the output is generated rather than of how much hardware you point at it, which means a single call's duration has a floor set by the length of its own answer. The only way under that floor is to have fewer tokens in each answer, which means more than one answer, which means more than one call.

The second is attention. A prompt carrying every instruction at once leaves the model with every set of rules in context simultaneously, competing for the same attention, and a model given forty sets of extraction rules has to decide, for every field, which of the forty apply. Concurrent branches address both, since they generate at the same time and each one carries a narrower brief.

You meet the design wherever code launches several calls and waits for all of them — an extractor fanning out over groups of fields, a review that runs one call per section of a document, a service that asks the same question five times and keeps the most common answer. Those last two examples are the two shapes the design comes in, and they are the subject of this chapter, because a ticket that says "parallelise the model calls" does not say which one it means and the two are not interchangeable.

The property that decides which pattern applies#

Sectioning splits a task into subtasks whose outputs never need to read each other. Branch three produces its answer without knowing what branch five found, and the merge step assembles disjoint pieces. Voting runs one whole task N times and aggregates the N candidate answers into one, which requires that the candidates be comparable — that it means something to ask whether sample two and sample four gave the same answer.

The question that decides which is even applicable is whether independent subtasks exist. Work through what independence has to mean here, because the word does a lot of load-bearing work. Two subtasks are independent when neither one's output is needed to produce the other's. That is a property of the task, not of the code: you can always launch two calls concurrently, and if the second one needed the first one's answer it will simply produce a worse answer without it. So the test is not "can these run at the same time" but "would either of them be different if it had seen the other".

If independent subtasks exist, sectioning is available and voting is also available, and you choose on measured quality, latency and cost. If they do not, sectioning is not an option at all, no matter how large the input is, and the only way to spend more than one call on the request is to spend it on repetition.

Input size is the wrong criterion, in both directions#

Input size correlates with the answer often enough to be misleading, which is what makes it worth taking apart with a case on each side.

A 220-word product description is small by any measure, and it still sections cleanly into six branches: one assigns a taxonomy category, one flags regulated health claims, one checks age-rating triggers, one normalises the stated dimensions and weight, one extracts material composition, one screens for prohibited-terms violations. Every branch reads the same 220 words. None of them needs any other branch's output — the age-rating verdict does not depend on the material composition, and the taxonomy category does not depend on either — and each one gets a shorter, sharper prompt than the combined version had.

A 60-page master services agreement handed the question "would we sign this as written?" is the other side. There is no way to cut it into six verdicts that assemble into the answer, and the reason is in the question rather than in the document. The output is one judgement, so any partition of the work produces branches whose outputs have to be combined by something that makes the judgement — which is the original task, undone.

That second example is worth separating from the first for another reason, because sectioning comes in two flavours that people conflate. In one, the input splits and each branch sees a slice — six sections of a contract, one branch each. In the other, the input is shared in full and the subtask splits — the product description read six times for six different judgements. Both are sectioning, because both satisfy the independence property, and they behave very differently on cost. The first keeps total input tokens roughly flat, since the six slices together are the document. The second multiplies them by the number of branches, since each branch is sent the whole thing. The chapter on cost accounting comes back to that, and the difference turns out to be the largest single term in a fan-out's bill.

Diagram

The two diagrams have the same shape and almost nothing else in common, which is exactly why the code for the two is hard to tell apart. In the sectioning arm the merge step is mechanical, because the parts do not compete — you are concatenating findings, filling different fields of one record, or stitching sections of a document. In the voting arm the aggregation step has to adjudicate, because the branches produce rival answers to the same question and at most one of them can be kept.

What each one buys, and what it cannot buy#

Laying the two side by side against the dimensions that actually differ makes the choice mechanical in most cases.

Read down the "what increasing N buys" row, because it is the one that gets misapplied. Sectioning increases coverage: with six branches you examine six things you were previously examining in one crowded pass, and each gets a prompt that is not competing with five other sets of instructions for the model's attention. It does not make any individual judgement more reliable, and the reason is that nothing in the design samples any judgement twice. If branch four misreads its section, running the whole design again produces the same misreading, because branch four is a single call with a fixed prompt over a fixed slice, which is precisely the arrangement whose reliability you started with.

Voting does the opposite. It makes one judgement more reliable and adds no coverage whatsoever. Five samples of "which of these three contracts has the shortest termination notice?" all read the same three contracts and all answer the same question. If the answer depends on a clause none of them thought to look at, five samples miss it exactly as thoroughly as one does, since the thing that would have to change is the question or the input and the design changes neither. Self-consistency and best-of-N with a judge, covered in 4.7, are both mechanisms of this kind: they buy you a better estimate of what the model believes about a question it has already been asked, and they cannot buy you a question it was not asked.

This is why a fixed budget of five model calls, spent on a single hard question with one verifiable answer over a short input, goes to repeated sampling and a vote. There is nothing to section. The task has one output, so any attempt to split it produces branches that either duplicate the work or answer fragments that a further call has to reassemble into the original question, as the contract-comparison feature in the scenario below discovered. Whether five is the right N is a separate matter that has to be measured, and the chapter on the accuracy curve deals with it, but the shape of the spend is settled by the task's structure before any measurement happens.

The row worth pausing on is the one about input tokens, because it closes an identity that later chapters keep using. Total input across a fan-out is the per-branch input times the number of branches, and the per-branch input is the whole prompt unless the input was genuinely split. So sectioning by slice is the only one of the three arrangements — slice-sectioning, subtask-sectioning, voting — where the bill does not scale with N. That licenses a quick estimate before any code is written: if every branch has to read everything, expect the input line to multiply. The residual is that caching can flatten the multiplication when the shared part is identical and warm, and that qualification is large enough to need its own chapter.

The neighbouring ideas this gets confused with#

Splitting a long input into chunks because it does not fit the context window, which is the maximum amount of text a model can be given in one call, is not sectioning, even though the code looks identical. Chunking is a mechanical response to a limit, the chunks are usually not independent — chunk seven refers to a definition introduced in chunk two — and the reason you did it has nothing to do with quality or throughput. You can run chunks concurrently, and you often should, but the design question of whether the parts are independent is exactly the question chunking dodges. When the parts are not independent and you pretend they are, you get the failure that the sectioned-review chapter takes apart in detail.

Routing is not parallelisation either. Routing picks one branch and runs it, so the extra calls are a classification cost rather than a redundancy cost, and the architecture chapter (11.1) placed it. The distinction that matters here is that a router spends one call to avoid N, while sectioning and voting both spend N deliberately.

A batch or asynchronous API is an execution mode, not a pattern. Submitting five samples to a batch endpoint at a discount changes what the five samples cost and when they come back; it does not change the fact that you are voting. The same is true of provider-side parallel tool calls, which schedule several tool invocations from one model turn — useful, and orthogonal.

The case where neither pattern is right is more common than either. If a single call already gets the answer correct 97% of the time and the remaining 3% are cases where the required information was never in the prompt, both patterns spend money to reproduce the same failure. Sectioning gives you six branches that each lack the information. Voting gives you five samples that each lack it and, more damagingly, agree with each other about what to do without it, which turns a gap in the input into a confident consensus. The reason those samples agree is that they share the thing that is broken: given the same missing information and the same prompt, five draws have no more reason to differ than one, and whatever the model does in the absence of the fact is what all five will do. The chapter on decorrelation returns to this, because the temptation to reach for a larger N when the real problem is upstream is the most expensive habit in this area.

There is also a hybrid, and it is legitimate: section into three parts and sample each part twice, or section into six and vote only on the two subtasks that the evaluation shows are unreliable. The budget chapter at the end of this topic works through what six calls can actually buy under each allocation, and the answer is less symmetric than it looks, because a vote needs at least three samples before it can break a tie.

Everything above treats the branches as though they run at the same time, and that assumption is doing more work than it appears to. The invoice extractor's drop from 14.1 seconds to 4.2 is a 3.4× improvement on a six-way split, which is already well short of six, and nobody on that team asked why — a win is a win, and 3.4× looked like the ordinary cost of coordination.

It is not. The next chapter takes a fan-out that returned 1.4× on eight branches, shows why the usual explanation for a disappointing speedup cannot produce that number, and gives you the one measurement that identifies what actually happened. The distinction you now have between splitting the work and repeating the work determines which pattern you reach for; whether the pattern pays for itself depends on whether the branches you launched were ever concurrent.