Three extra round trips need a reason the score cannot show

The product description generator you inherited has five stages: pull attributes from the supplier feed, normalise units and sizes, select the marketing angle, draft the copy, then a compliance pass that strips unsupported claims. It scores 84 on a 400-item evaluation set. On a slow afternoon you wrote a single prompt that does the whole thing, ran it on the same set with the same model and the same judge, and it scored 82. The chain uses 4.6 times the tokens and its p95 is 9 seconds against 2. Ninety thousand descriptions a month go through it, which puts the chain at roughly $3,350 a month against $730.

Two points looks like a reason and it is not one yet, because you do not know what this suite does when nothing changes at all. Sampling temperature moves individual generations, the model judge scores borderline items differently on different runs, and 400 items is a small enough sample that a handful of coin flips moves the aggregate. Until the run-to-run spread is measured, 84 and 82 are two draws from distributions of unknown width, and every argument built on the gap is an argument about noise.

Measure the spread before you compare anything#

The reassurance that makes teams skip this step is worth naming, because it is the same suite-is-controlled belief that decides architecture arguments elsewhere in this book. Both configurations ran on the same 400 items with the same judge, so the inputs and the grader are genuinely held constant, and that removes a real source of error. What it does not hold constant is the draw. The same configuration, run twice on that same suite with nothing edited in between, returns two different numbers, and until you know how far apart those two numbers usually sit, a two-point gap between different configurations has nothing to be compared against.

The measurement is unglamorous: run each configuration five times on the same set, unchanged, and look at what comes back.

chain          84  81  85  83  86     mean 83.8   sd 1.9   range 81-86
single prompt  82  84  80  83  81     mean 82.0   sd 1.5   range 80-84

The means differ by 1.8 points, the standard deviation beside each one — the typical distance of a single run from its own configuration's mean — is around two points in both, and the ranges overlap across four of the five runs each. Nothing here establishes that the chain is better, and if you had happened to run the chain once on the day it scored 81 and the single prompt on the day it scored 84, the whole design review would have gone the other way on the same underlying systems.

That last sentence is the argument in its strongest form, and it is worth checking against the numbers rather than accepting. The chain's five runs include an 81 and the single prompt's include an 84, so both of those are outcomes the identical systems really did produce. A comparison built from one run of each is therefore capable of reporting a three-point win for either side, which means the sign of the result, and not merely its size, is inside the noise.

Comparing aggregates is also the weakest test available, because it throws away the pairing. Both configurations ran on the same 400 items, so score them item by item and count:

wins = sum(1 for i in items if chain[i] > single[i])     #  63
losses = sum(1 for i in items if chain[i] < single[i])   #  51
ties = len(items) - wins - losses                        # 286

# Standard error of the paired win-rate difference over discordant pairs.
d = wins + losses                                        # 114
diff = (wins - losses) / len(items)                      # 0.030
se = math.sqrt(d) / len(items)                           # 0.027

Two hundred and eighty-six of the four hundred items came out the same either way, and those carry no information about which design is better — they would have looked identical whichever one you shipped. What is left is a hundred and fourteen items where the two disagree, and that count, rather than four hundred, is the size of the instrument.

The last two lines are worth reading as arithmetic. The difference is the net wins, twelve of them, spread over the four hundred items, which is the three percentage points. The standard error takes the square root of the discordant count and divides by the same four hundred, which lands at two point seven percentage points. So the edge and its own error bar are almost the same size.

The chain wins on 63 items and loses on 51, which is a three-percentage-point edge with a standard error of 2.7 points, the standard error being roughly how far that edge would move if you drew a different 400 items from the same population. That is a coin landing heads 63 times in 114 throws. The paired view is worth running even when the aggregates look decisive, because it costs nothing extra and it answers a different question — not "is the mean higher" but "on how many individual items does this actually change the output for the better", which is the thing a product decision rests on.

The three-way comparison#

With spread in hand, the comparison worth putting in front of a team lead covers all three architectures on fixed inputs, measured on the same day, on the same model. The loop column here is a tool-using agent given an attribute-lookup tool and a claims-checking tool over the same feed.

Three of these rows are quality and cost, and they say the three designs are within noise of each other at a 7.4x spread in monthly spend. The rest are the rows the score cannot see, and they are where an honest case for the chain has to come from.

The latency row rewards a second look, because the loop is faster than the chain at p50 and nearly twice as slow at p95, which sounds like a contradiction and is not. The chain makes five calls on every item, so its typical item and its slow item differ only in how long those five calls took. The loop makes between three and nine, so a typical item is a short run and a slow item is a long one, and the number of round trips is itself part of what varies. A distribution whose call count varies will always spread more between its middle and its tail than one whose call count is fixed, which is why the design with the lower median can be the one that misses a timeout.

Reading the table as gaps makes the asymmetry plain. Across the first row the three configurations sit inside each other's error bars, so the gap is zero for practical purposes; across the cost row the gap is a factor of more than seven. When one column of a comparison has no measurable spread and another has an order of magnitude, the decision cannot be made from the first column, whatever the arrows on the numbers suggest.

Two adjustments make the cost row more honest than it looks. Retries are missing from it: the measured token counts are for clean runs, and with a 4% contract-failure rate at two of the five stages and a budget of two attempts each, the chain's real multiplier is closer to 4.9x than 4.6x. Prompt caching pushes the other way, since the five stages share a long system prefix that the provider can serve from cache at a fraction of the input rate, which on this pipeline took the chain's monthly figure from $3,350 to about $2,900 once cache-aware billing was accounted for. Neither correction changes the verdict here, and both are worth applying before quoting a multiple to anyone with a budget, because a number that turns out to be wrong in your favour costs you the argument.

The reasons that survive scrutiny#

When the scores tie, the chain has to justify itself on something the aggregate is structurally blind to, and the list of things that qualify is short.

A stage you must audit on its own is the strongest. The compliance pass in this pipeline strips claims the supplier data does not support, and the legal team's requirement is not that descriptions are compliant on average — it is that every description passed a compliance step whose input, output and prompt version are recorded. Folding that into one prompt makes compliance an emergent property of a generation you cannot inspect. As a separate stage it has its own log line, its own fixtures, and an answer to "show me that this ran" that does not require trusting the model.

A boundary where something bad has to be stopped before it reaches a customer is the second. If normalisation produces a size in the wrong unit, the chain fails at a gate and the item is dead-lettered; the single prompt produces a fluent description containing the wrong size, and nothing in the system knows. That matters in proportion to how expensive a wrong output is, so it is a strong argument for a medical device catalogue and a weak one for socks.

A stage with genuinely different operating requirements is the third. Attribute extraction runs fine on a small model at a tenth of the price, the drafting stage needs the large one, and the compliance pass runs at temperature 0 with a pinned version because legal signed off on that specific configuration. Separate stages let those settings differ; one prompt makes them uniform by construction. When this is the reason, it usually also shrinks the cost gap, because the cheap stages stop being billed at frontier rates.

The three have a property in common that is worth extracting, since it is what makes them durable while other arguments are not. Each of them names something that must be true of the process rather than of the output: that a step ran and was recorded, that a value was checked before it travelled, that a component ran under settings somebody signed off. No inspection of a finished description can establish any of the three, however good the description is, which is exactly why a score cannot express them.

What does not survive is modularity as a value in itself. "It is easier to reason about" and "we can improve stages independently" describe capabilities, and a capability nobody has used in eighteen months is not worth $2,600 a month. The test is whether you can name a specific thing you did, or will do within the quarter, that the separation made possible.

Latency deserves the same treatment rather than an automatic penalty. Nine seconds at p95 is disqualifying for a synchronous request behind a spinner and irrelevant for a nightly batch that runs against a feed at 02:00, where the only constraint is finishing before the store opens. What makes it a real cost is that a p95 becomes a product property the moment anything downstream starts depending on it — a preview button, a bulk re-generate, an integration partner's timeout — and by then it is expensive to give back.

The measurement has a shelf life#

Whichever way it goes, the output of this exercise is a record, because the comparison you just ran describes today's models and today's prices. Model prices fall, model quality rises, and the single prompt that scored 82 today may score 87 on the next release while the chain gains nothing, since its per-stage prompts were tuned against the old model's quirks.

That direction of travel is worth expecting rather than being surprised by. A large share of the chains written in 2023 and 2024 existed because a single prompt could not hold a long document, could not reliably produce structured output, and lost the plot across a multi-part instruction — all three of which have moved substantially since, and all three of which were the reason for a stage boundary rather than an argument for one. Every model release quietly re-runs this comparison on your behalf and does not tell you the result. The stages that survive that erosion are the ones whose justification was never about capability: a step that has to be logged and evidenced separately, a gate that has to stop something before it reaches a customer, a component that runs on a different model at a different temperature under a different sign-off. Those reasons do not get cheaper as models get better, which is exactly why they are the ones worth writing in the record.

decision: keep the five-stage chain for product descriptions
date: 2026-03-24
owner: catalogue-platform
measured:
  suite: product-desc-400 v3, judge v2, temperature 0.7
  chain:         {mean: 83.8, sd: 1.9, runs: 5}
  single_prompt: {mean: 82.0, sd: 1.5, runs: 5}
  paired: {wins: 63, losses: 51, ties: 286, se: 2.7}
  verdict: not distinguishable on quality
  cost_per_month_90k: {chain: 3350, single_prompt: 730}
  p95_seconds: {chain: 9.0, single_prompt: 2.0}
reason_kept: >
  Compliance stage must be independently logged, versioned and evidenced
  for legal; normalisation gate stops wrong-unit sizes before publication.
  Quality is not the reason.
flips_if:
  - a single prompt plus a standalone compliance call reaches parity on the
    suite, since that keeps the auditable stage and drops three round trips
  - legal drops the per-item evidence requirement
  - monthly volume passes 250k items, where the gap exceeds $7k/month
re_measure_when:
  - the drafting model changes minor version or is deprecated
  - the suite is revised, or the judge is upgraded
  - 2026-09-24 at the latest

The flips_if block is the part that makes this a decision rather than a rationalisation. It names, in advance and while nobody is defending anything, the conditions under which the answer changes, so the next engineer inherits a hypothesis with a test attached rather than an architecture with a story about it. Each of its three lines is checkable by someone who was not in the room: two of them are measurements, and the third is a question with an owner. The volume line is worth verifying rather than trusting, since it is the kind of threshold that gets written from memory. At 90,000 items the gap between the two designs is $2,620 a month; 250k items is nearly three times that volume, and a gap that scales with volume is therefore something over seven thousand a month at the higher figure, which is what the line says. The re_measure_when block does the same job for time, since the erosion above happens whether or not anybody schedules a review, and a dated backstop is what keeps a stale comparison from being quoted for a year.

That record is also the thing to write first when you build the next chain. Contracts at the joints, a failure policy per stage, fixtures per link, a replay you can run on a single record, and a measured comparison against the simpler design — the pipeline you are about to build should carry all of them before it carries its second stage, because each one is cheap on day one and expensive to retrofit onto something already in production.