Non-determinism shows up in the spread, not in the average

Two designs for one task will each come back with a percentage, and the two percentages will differ by a few points. A workflow, in this topic, is a program whose steps and the order they run in are written in code before anything executes. An agent is given a set of tools and picks at each turn which one to call, so the sequence of steps for a particular run exists only once the run is over. Point both at the same task, grade them with the same function, and each reports one number. Only one of those two numbers is a property of the thing it describes.

The reason is that a model samples rather than computes. Each token it emits is drawn from a distribution over possible next tokens, so two runs of the same prompt can differ. In a workflow that difference is confined to what the model writes inside a step, because the step after it was chosen by the author. In an agent the difference reaches the choice of the next step, so it changes the path. A workflow's percentage is therefore an average over one fixed path through the program, and re-running it returns the same value; an agent's is an average over one draw from a distribution of paths, and re-running it returns something else. Every measurement in this chapter is built to expose that asymmetry, and each of them costs one extra loop in the test harness.

Two numbers from one run of a system that samples#

The task is contract-clause extraction: given a supplier agreement of 8 to 60 pages, pull out the termination notice period, the liability cap, the governing law and the auto-renewal terms. The workflow version chunks the document, retrieves candidate sections per field, makes one model call per field, and validates. The agent version gets a search tool, a page-reader tool and a calculator, and iterates until it produces all four fields. On 200 held-out contracts the workflow scores 88% and the agent scores 91%. Held-out means those 200 contracts were kept out of every prompt, template and retrieval index built for the task, and each carries a hand-checked set of the four correct values — the gold answers — which a grader compares each run's output against.

Three points is not nothing, and both numbers are averages over a single run of a stochastic system, which means neither of them is a property of the prototype in the way a slide with two percentages on it implies. Running the same 200 contracts through the agent a second time produces a different 91% — different in composition, and slightly different in value. Running them through the workflow a second time produces the identical 88%, the same 176 contracts passing and the same 24 failing.

That asymmetry is the measurement, and it does not appear anywhere on a slide carrying two percentages. It is also available before any extra tooling is written. Re-run each system once and compare the two result sets item by item. If the failing set is identical, the system has one path per input and its percentage is a fact about the system. If the failing set moves while its size barely does, the percentage is a fact about the run, and every decision taken on it inherits that.

Repetition, and the two columns it pays for#

Matched prototypes means the same inputs, the same underlying model, the same tools where both have them, and the same grader. What has to be added is repetition:

def bench(system, contracts, repeats=30):
    rows = []
    for c in contracts:
        for r in range(repeats):
            t0 = time.perf_counter()
            out = system.run(c.text)
            rows.append({
                "contract": c.id,
                "repeat": r,
                "passed": grade(out, c.gold),        # all four fields exact-match
                "tool_calls": out.tool_call_count,   # 0 for the workflow's fixed path
                "tokens": out.usage.total,
                "seconds": time.perf_counter() - t0,
                "answer_hash": sha256(canonical(out.fields)),
            })
    return pd.DataFrame(rows)

Two columns there do the work that a success percentage cannot. answer_hash lets you count, per contract, how many distinct outputs the system produced across 30 identical runs, which is a direct measurement of reproducibility. And keeping the per-repeat rows rather than averaging them lets you separate which contracts pass from how often they pass.

Those two are separable only because nothing is aggregated inside the loop. An average computed as the rows arrive discards the item identity, and item identity is what every reading below is made of: a mean pass rate can be recovered from per-run rows at any time, and per-run rows cannot be recovered from a mean. That is the whole argument for the extra column of storage.

Six thousand runs per system at these sizes cost about $310 for the agent and $74 for the workflow, and took eleven hours wall-clock on eight workers. That is the price of the number. It is worth flagging that this bill is smaller than the per-contract costs reported later in this chapter predict — six thousand runs at those means would come to more than three times each figure — and nothing measured here says where the difference went. Repeating one document thirty times is exactly the workload a provider-side cache is built to discount, which would explain it and is not evidence. The bill is reported as observed and no later argument is built on it.

How many repeats, and what "matched" has to mean#

Thirty repeats is not arbitrary and it is not a lot. The quantity being estimated per contract is a per-item pass probability, and with 30 samples the 95% confidence interval (the range in which the item's true pass rate plausibly sits, given the sample) on an item that passes 20 times out of 30 runs from roughly 47% to 83%, which is wide.

That width is worth trusting rather than arguing with. It is the exact interval for that many successes in that many trials, computed from the binomial distribution itself rather than from the bell-curve approximation, and the approximation would report a slightly narrower band at the bottom end. Either way the message is the same: two-thirds of thirty pins the true rate only to a range about a third of the scale wide.

That is adequate for the question being asked, because the question is categorical: is this item's pass probability near 1, near 0, or somewhere in between. Distinguishing 0.67 from 0.80 would need several hundred repeats and would not change any decision. Five repeats, by contrast, is genuinely insufficient — an item with a true rate of 0.8 shows up as unanimous in about a third of five-run samples, so a third of the unstable items would be misclassified as stable, and the stability column would understate the problem.

The step under that last figure is worth doing rather than accepting. An item passes every one of five independent runs when it passes the first and the second and so on, so the chance is its pass rate multiplied by itself five times. Raise eight-tenths to the fifth power and the answer is a third. The same reasoning at thirty runs gives a number too small to matter, which is the argument for thirty stated as arithmetic rather than as habit.

Matched has a stricter meaning than teams usually apply, and three mismatches are common enough to name. The two prototypes must run against the same model version, because comparing a workflow built in February against an agent built in April with a newer model measures the model. They must have the same information available: if the agent's page-reader can see the full document and the workflow's retriever only surfaces three chunks, the comparison is between retrieval strategies and the control-flow question has been confounded. And they must be graded by the same function on the same gold data — a workflow scored on exact string match against an agent scored by a model-based grader will differ by several points from the grading alone.

The last of those is the one that survives review most easily, because both graders look reasonable in isolation. The check is to run both systems' outputs through both graders and confirm the gap is stable; here the exact-match grader gave 88.0 and 91.2, and a model-based grader gave 89.4 and 93.1, preserving the ordering and widening the gap by half a point. Had the ordering flipped, the grader would have been the finding.

What 91% turns out to consist of#

Read that table as gaps rather than as levels. The mean pass rates differ by 3.2 points and the stable pass rates differ by −3.5 in the other direction, and the sign flip between those two rows is the finding. On the row everybody quotes the agent is ahead; on the row that asks how many contracts it handles every time, it is behind by about the same margin.

The second gap is inside the agent's own column. Take the share of contracts it passed at least once and subtract the share it passed every time: the first is nineteen contracts in twenty of the two hundred, the second is a little under seventeen in twenty, and the difference is twenty-one contracts. Those twenty-one pass sometimes and fail sometimes, and they are the whole of the agent's apparent advantage. Of that advantage, none of it is a class of contract it reliably handles and the workflow reliably does not; it is 21 contracts that sometimes work, weighted against 7 contracts the workflow gets right every time and the agent gets right most of the time.

The distinct-output row measures something adjacent and not identical, and the two are worth keeping apart in the reading. A contract counted there produced more than one answer across its repeats. A contract counted in the twenty-one above produced more than one verdict. Ship the agent and 41 of your 200 contracts have an output that depends on which run happened, while twenty-one of them have an outcome — a pass or a fail — that does.

That reframing changes what the two numbers mean to the people downstream. The workflow's 88% is a fixed, inspectable coverage set: the 24 failures are the same 24 contracts, they can be read, and they turn out to be 19 scanned documents with poor OCR and 5 agreements where the liability cap is expressed as a formula rather than a figure. Those two counts sum to the failure count exactly, which is what makes the set readable rather than merely counted — there is no residual bucket of failures nobody has looked at, so a list of two work items covers everything the workflow gets wrong. Both are addressable, and addressing the first with a better OCR pass is a known piece of work. The agent's failures are a moving population, so "what does it get wrong" has no stable answer, and the natural next question — what should we fix — has nowhere to start.

The tails, and the budget they have to be read against#

The tail numbers matter separately from the averages. A p95 of 51 seconds against 8 seconds is not a 6× inconvenience if this runs as an overnight batch over 4,000 contracts; at 20 seconds median and eight workers that is under three hours, comfortably inside the window, and the latency column can be ignored entirely.

That last claim is a division worth showing, because it is the step that decides whether a whole column of the table is relevant. Multiply the number of contracts by the median seconds each takes to get the total seconds of work, then divide by the number of workers running in parallel to get elapsed time. Four thousand documents at twenty seconds is eighty thousand seconds of work, which spread over eight workers is ten thousand seconds, and ten thousand seconds is under three hours. Nothing in that calculation used the tail, which is the point of running it: in a batch the tail is absorbed by the workers that are not stuck on it.

If the same extraction runs inside a procurement UI where a buyer is waiting, a p95 of 51 seconds is a product failure regardless of accuracy, and the workflow wins before anyone opens the pass-rate column. Latency budgets are properties of the use, so a latency measurement with no stated budget beside it cannot decide anything.

The p99 for tokens is where the surprises live. The agent's token p95 of 214,000 against a median of 68,300 is a factor of three, and the runs producing it were contracts where a search tool returned near-duplicate sections and the loop re-read overlapping pages. Four of the 6,000 runs hit the step cap and returned partial results, which the grader scored as failures and which would appear in production as an incident with no reproduction.

Those two figures are the same phenomenon measured twice. A search returning near-duplicates gives the model a reason to read more pages; reading more pages puts more text in the message list; a longer message list is re-sent on the next iteration, so tokens grow faster than steps do. The runs at the top of the token distribution and the runs that exhaust the step budget are therefore drawn from the same set, which is why a token cap and a step cap tend to fire on the same documents.

The recommendation, and what would reverse it#

On these numbers the workflow ships. The reasoning is that the agent's headline advantage does not survive being asked what it consists of, its failures cannot be enumerated, and it costs four times as much to produce a result that is less stable per contract. The 12% of contracts the workflow cannot do are a known, readable list, and a fallback branch sending them to a human — 480 contracts a month at four minutes each — is 32 hours, which is a number the business can price.

Both halves of that sentence are products with nothing left over, which is what makes them safe to quote. The share the workflow cannot handle is the complement of its pass rate, and that share of four thousand contracts a month is the four hundred and eighty in that sentence; four hundred and eighty at four minutes each is nineteen hundred and twenty minutes, which is the thirty-two hours. No unlisted category hides inside either figure, so a reader with a different monthly volume can redo both in their head rather than asking for a new study.

What would reverse it is worth stating precisely, because a recommendation with no reversal condition reads as a preference:

The cost row is the one a reader can carry to another problem, so it is worth unpacking. Subtract the workflow's mean cost per contract from the agent's and the difference is about thirteen cents; that difference over four thousand contracts a month is the five hundred and twenty dollars in the table. The threshold it defines is a single division: an error the agent avoids has to be worth more than that monthly figure divided by the number of errors it avoids. Quoted that way the row settles the question at any volume, rather than at the one volume somebody happened to measure.

The last row is the one people expect to rescue the agent, and it repays understanding why it usually does not. Temperature is the sampling parameter that controls how randomly the model picks each next token, and setting it to zero makes that choice always take the highest-probability option. The belief that follows is that a system with no randomness in it produces no variation, and it is tempting because it is true of the layer it describes.

What is actually true is that a quantity has been held fixed while the property that produces the variation has not been touched. It affects token sampling. The variance that matters here is in the path, and a loop whose second step depends on what a search tool returned will diverge on the first result that ties — two candidate sections with equal scores, ordered by whichever the index happened to return first, and the run follows the one it was handed. Determinism at the decoding layer does not produce determinism at the trajectory layer, and the measurement says so: of the forty-one unstable contracts, thirty-three remained unstable with temperature pinned and caching on.

Reproducibility is not correctness#

Low variance is not correctness. A workflow that returns the same wrong answer 30 times out of 30 scores perfectly on reproducibility and is still wrong, and the 24 failing contracts here are exactly that. Reproducibility is valuable because it makes failures addressable, and it is not evidence of quality. A system can be pinned to one path and be pinned to the wrong one.

High variance is also not automatically disqualifying. On tasks where any of several answers is acceptable — summarising, drafting, generating candidate clauses for a human to pick from — measuring output-hash stability produces an alarming number that means nothing, because the grader, and not the hash, defines acceptability. Two summaries differing in word order hash differently and are equally good. The variance measurements to take there are the ones with a cost attached: tool calls, tokens, latency, and the rate at which runs hit their budget caps.

The distinction between those two cases is whether the task has a single correct answer that a grader can recognise. Extraction does: a liability cap is a figure, and there is one right one. Drafting does not. Applying an output-stability measure to work of the second kind produces a number that falls as quality rises, which is worse than having no number at all, because somebody will optimise it.

There is one more limit on everything above, and it is about what was measured rather than how. Both prototypes were graded on whether the extracted liability cap matched the gold value, and both were read-only: the output went onto a screen for a human to accept. Change the task so that the same extraction auto-updates a contracts database, or triggers a renewal notice to a supplier, and the six thousand runs above still tell you the failure rate while telling you nothing about what the failures cost.

A pass rate is a claim about how often a system is right. Repetition converts it into two claims — how often, and how reliably the same items are the ones it is right about — and the second is what makes a failure list actionable, a cost predictable and a tail visible. None of the three requires a new statistic; all of them require keeping the rows.

What none of them touches is what a wrong answer does once it leaves the harness. Two properties decide that, and neither of them appears in a benchmark harness at all.