An end-to-end score says the chain failed, not which link did

The support-reply pipeline is four stages of prompt chaining — extract the customer's facts, normalise them against the account record, classify the request, draft the reply — and it had scored 91 on a 400-conversation evaluation set for two months. One release touched two prompts: someone tightened the classifier's category definitions, and someone else shortened the drafting instructions. The next run scored 84. Nobody could say which change did it, so the team reverted both, got 91 back, and lost whatever the classifier improvement had been worth along with the regression. Working out which one was responsible would have meant re-running the whole 400-item set twice more, at forty minutes a run, which is what they did on the Thursday, and it turned out the drafting change was harmless and the classifier change had cost seven points on its own.

That is a cheap version of the problem. The expensive version is a chain that scores 88 while quoting the wrong product tier to a customer, where the score tells you something is broken and stops. A field dropped at a boundary and a prompt that reasons badly are the same number in an aggregate and different repairs on a Monday morning: one is a schema and a plumbing fix that takes an hour, the other is prompt work and a new fixture set that takes a week. An evaluation that cannot distinguish them is measuring the product and not the system.

What an aggregate score can and cannot see#

The argument against per-stage evaluation is worth stating in its strongest form, because it is not silly. If the final output is right, every stage did its job; if the final output is wrong, the end-to-end set catches it; and per-stage fixtures are therefore effort spent testing internals that the composed test already covers. Anyone who has argued against over-mocked unit tests has said something close to this and been right.

It fails on chains for a reason specific to their shape. An end-to-end score is a single number over a composition of four probabilistic functions, and it is not decomposable. Three things hide inside it.

The first is attribution, which is the point above: 88 is a property of the chain, and every repair you might make is a property of one link.

The second is compensation. Stage two drops tier on 6% of records; stage four, drafting the reply, infers the tier from context and gets it right most of the time. End-to-end looks fine — perhaps 88 rather than 90 — and the pipeline has a live defect that surfaces as a wrong tier only when stage four's guess is wrong, which correlates with unusual accounts, which is to say with the customers most likely to complain. No aggregate score will ever separate "stage four is guessing" from "stage four is reading", because both produce the same output most of the time.

That is the sentence to take away from this chapter, because it is a defence that people believe in and it does not hold. The end-to-end score is genuinely measuring something real, and what it holds constant is the quality of the finished reply on the cases in the set. What varies underneath it is whether the reply was assembled from data or from inference, and no measurement taken at the end of the chain can carry that quantity, since the two produce identical outputs whenever the guess is right.

The third is dilution. If each stage is independently good, end-to-end quality is roughly the product of the stage qualities, so four stages at 97% land near 88% composed. That multiplication is worth doing rather than accepting: a record survives the chain only if it survives every stage, so with independent stages the chance of surviving all four is the chance of surviving one, multiplied by itself four times. Ninety-seven per cent taken to the fourth power is a little under eighty-nine, which is where the composed figure comes from. A stage sitting at 82% and a stage sitting at 99% both disappear into that number, and the one worth your week is invisible.

Two residuals sit on that identity and they pull in opposite directions, which is why the composed score is hard to reason about from the outside. Independence is an assumption: a hard conversation tends to be hard for every stage, so failures cluster on the same records and the true composed figure lands above the product. And compensation, the point above, also lifts it, because a downstream stage repairing an upstream omission means a record can fail a stage and still survive the chain. The product is therefore a rough floor rather than a prediction, and the useful part of it is the shape — that adding a stage multiplies rather than adds, so a fifth stage at 97% costs about three per cent of everything that reached it.

end-to-end 88  ─┬─ stage 1 extract    conformance 100%   correctness 97%
                ├─ stage 2 normalise  conformance 100%   correctness 94%   ← tier dropped 6%
                ├─ stage 3 classify   conformance  99%   correctness 96%
                └─ stage 4 draft      conformance 100%   correctness 91%

The 94 on stage two is the one to chase, and nothing in the 88 points at it.

That diagram is also the answer to the compensation problem, which is worth making explicit because it is the reason per-stage correctness is not merely a convenience. Stage four covering for a dropped tier keeps the end-to-end number high, and it cannot keep stage two's number high, since stage two is scored against what stage two should have emitted for that record. So the defect that is invisible at the end of the chain is directly visible one measurement earlier, and it shows up as a gap between two numbers that ought to track each other: a stage whose correctness sits well below the composed score is a stage somebody downstream is quietly repairing.

The two evaluation sets a prompt chaining pipeline needs#

Per-link evaluation means each stage gets its own fixtures, and the design decision that makes it useful is that a stage's fixtures pin the stage's input. Stage three is evaluated against a frozen set of stage-two outputs held in a file, not against whatever stage two produces today. That is what isolates a prompt regression from contamination arriving from upstream: when stage three's number moves and its inputs did not, the cause is inside stage three.

The reasoning there is worth making explicit because the whole arrangement rests on it. A stage's output is a function of two things, its input and its own configuration. Hold the input fixed in a file and the input cannot be what changed, so any movement in the score has one remaining candidate. That is the same manoeuvre as the ablation in the topology chapters — pin everything but the factor you want to price — applied at the granularity of a stage.

The frozen inputs then create a second gap, which the other set covers. An integration set runs the real chain on raw inputs and scores each stage's output where it lands, so it sees what the isolated sets cannot: stage two changing what it emits in a way stage three's frozen fixtures never contained.

The row that decides how many of these you need is the last one, and it is the only row where the three columns do not overlap. Each set is blind to something the others see, and no two of them are blind to the same thing, which is why the honest answer is not to choose. The row above it follows from cost and is the practical consequence: the cheapest and narrowest set runs most often, the most expensive and broadest runs least often.

Each stage carries two measurements rather than one. Conformance is the contract pass rate from the gates of chapter 1, and it is free — the gate already runs, so counting its failures costs a log line. Correctness is a task metric and it needs labels, so it is where the work is. The metric has to fit what the stage does, and the ones below are ordinary supervised measures — F1 is the harmonic mean of precision and recall, so a stage that invents fields and a stage that misses them are both penalised, and recall on its own is the fraction of the items that should have been found which were:

The reason F1 rather than either half alone is the extraction stage's failure modes. A stage that emits every field it can think of scores well on recall, since it found everything that was there, and badly on precision, since most of what it emitted was invented. A stage that emits only the two fields it is sure about scores the reverse. Taking a harmonic mean rather than an ordinary average is what makes a low score on either half drag the result down, so a stage cannot buy a good number by being extreme in one direction.

The per-category floor on the classifier is doing real work. Aggregate accuracy of 0.96 across nine categories is entirely compatible with one low-volume category at 0.4, and low-volume categories in support are things like billing disputes, where being wrong is expensive out of proportion to frequency.

Fixtures are cheapest to build from the ledger that chapter 2 introduced, because it already records what each stage received and emitted per record. Sampling production runs gives you inputs that look like production, which hand-written fixtures famously do not:

{"id":"f-0012","input":{"account_id":"AC-88213","facts":{"product":"Team","seats":14,"renewal":"2026-05-01"}},"expect":{"category":"billing_dispute","confidence_min":0.6}}
{"id":"f-0013","input":{"account_id":"AC-19004","facts":{"product":"Enterprise","seats":410,"renewal":null}},"expect":{"category":"renewal_question"}}
{"id":"f-0014","input":{"account_id":"AC-77510","facts":{"product":null,"seats":3,"renewal":"2026-02-14"}},"expect":{"category":"account_access"}}

Fixture f-0014 carries a null product on purpose. Half the value of a per-stage set is the degraded inputs, because a stage's behaviour on a missing upstream field is precisely the behaviour that an end-to-end set built from clean conversations will never exercise.

The runner is small, and its output is what makes the Thursday bisect unnecessary:

def eval_stage(stage: Stage, fixtures: list[Fixture]) -> StageScore:
    conform, correct, failures = 0, 0.0, []
    for fx in fixtures:
        out = stage.invoke_raw(fx.input)
        try:
            value = stage.contract.validate(out)
            conform += 1
        except ValidationError as e:
            failures.append((fx.id, "conformance", str(e)))
            continue
        s = stage.metric(value, fx.expect)      # 1.0 / 0.0, or a rubric score
        correct += s
        if s < 1.0:
            failures.append((fx.id, "correctness", stage.metric.explain(value, fx.expect)))
    n = len(fixtures)
    return StageScore(stage.name, conform / n, correct / n, failures)

The continue after a conformance failure is the line that keeps the two measurements honest. A payload that did not validate cannot be scored for correctness, since there is no value to compare against the expectation, so it counts against conformance and is left out of the correctness numerator. Reading the two numbers together is what tells you which kind of repair you are facing: conformance down means the stage stopped producing the declared structure, correctness down at full conformance means it is producing well-formed answers that are wrong.

The drafting stage's row in the metric table hides the most fragile piece of this arrangement, which is the model judge. A rubric graded by a model is the only practical way to score free text at volume, and it is a dependency with the same properties as any other stage: it has a prompt, it has a version, and it drifts. Measured against 120 human-labelled drafts, the judge in this pipeline agreed with the human 88% of the time, which sounds fine until you notice that a 3-point movement in stage four's score is inside the judge's own disagreement band. The consequence is that stage four's score is a measurement of the judge's opinion of the drafts, and the judge's opinion differs from a person's on roughly one item in eight. A handful of borderline drafts landing on the other side of that boundary moves the stage's score by a few points with nothing about the drafting prompt having changed, which is what "inside the disagreement band" means.

Two habits keep it usable. The judge's prompt and model version are pinned and recorded alongside every score, so a suite result is never comparable across a judge change without saying so. And a rotating sample — 40 items a month — goes to a human, whose agreement rate with the judge is tracked as its own number; when that rate falls, the judge is recalibrated before anyone interprets a stage-four regression. Tracking the agreement rate separately is what lets you tell a drifting judge from a drifting stage, since both produce the same symptom — a score that moved — and only one of them is a problem with the pipeline.

Not every stage will have labels, and holding out for them is how per-link evaluation gets postponed forever. A stage with no labelled correctness metric can still carry conformance, which is free, plus invariants that need no ground truth: every entity in the output appears verbatim in the input, the item count does not exceed the number of line items in the source, no field is longer than the field it was derived from. What those three have in common is that each compares the output against the input rather than against a correct answer, which is why they need no labels — the input is already in hand. These are weak measures and they catch a surprising share of real regressions, because the failures that matter early are usually structural rather than subtle. Adding a labelled set of 60 items later upgrades the stage; starting with invariants means the stage is not invisible in the meantime.

Where per-link evaluation misleads#

Frozen fixtures rot. Stage three's set was sampled from stage two's output in January; stage two's prompt has been edited four times since, and its output now carries a field the fixtures do not contain and omits one they all have. Stage three keeps scoring 0.96 against a distribution that no longer exists, and the number is worse than no number because people trust it. That is the price of the isolation: the same freezing that makes a moved score attributable also guarantees the fixtures diverge from production, and the divergence is silent because a frozen file cannot fail. The countermeasure is scheduled rather than clever: resample each stage's fixtures from the ledger quarterly, and diff the new sample's key set against the old one before replacing it, since a changed key set is itself the finding.

Per-stage sets also cannot see interaction. Every stage can pass its own fixtures while the composed system produces something worse than any individual score suggests, because quality is not additive across boundaries — a slightly clipped extraction and a slightly literal classifier compose into a reply that is technically correct and useless. This is why the end-to-end set stays, and why the argument at the top of the chapter is half right: neither set replaces the other, and a team that has only one of them is guessing about a different thing depending on which one it kept.

The last honest cost is that labels are work. Four stages at 100 fixtures each is 400 labelled items to create and then to maintain, and the drafting stage's rubric needs a human pass often enough to keep the model judge calibrated. On the support pipeline that came to about three engineer-days to build and half a day a month to maintain, against the two days the single bisect above consumed. The payback is real but it is not immediate, and pretending otherwise is how these sets get built in a hurry after an incident and abandoned two quarters later.

There is one more thing per-stage sets will do that looks like a contradiction until you have seen it. You will edit a stage, its own fixtures will pass at 100%, and the end-to-end score will fall by seventeen points.