95% and 97% steps producing a 79% pipeline: the handoff

Two steps in a chain each have a pass rate, and multiplying them is the natural prediction for a pipeline built out of them: a step at 95% feeding a step at 97% should land near 92%, because 0.95 × 0.97 = 0.9215 and losses compound. That multiplication is worth doing, and it is worth understanding as a hypothesis rather than an estimate. It assumes step two's behaviour on step one's output does not depend on whether step one got it right — that its 97% applies uniformly across everything step one hands it.

When a pipeline lands persistently below the prediction, the hypothesis has been falsified and the quantity to go and measure has a name: the conditional pass rate, the rate at which step two is correct given that step one was correct. It is almost always lower than the number on step two's dashboard, and the reason is almost always that step two was evaluated on inputs somebody wrote by hand.

What independence predicts, and what a gap below it means#

Write out what end-to-end success actually requires and the assumption becomes visible:

P(end ok) = P(s1 ok) · P(s2 ok | s1 ok)  +  P(s1 bad) · P(s2 recovers | s1 bad)
0.79      = 0.95    · x                  +  0.05      · y

That first line is the law of total probability written out, and reading it as such is what makes the rest of the chapter mechanical. Every run of the pipeline falls into exactly one of two cases: step one was right, or step one was wrong. The chance of ending up correct is therefore the chance of landing in the first case times the chance of succeeding from there, plus the chance of landing in the second case times the chance of succeeding from there. Nothing about models is assumed in that sentence — it is arithmetic on a partition, and it holds for any two-stage process whatsoever. What the naive multiplication does is quietly set the second term to zero and assume the first case's conditional rate equals step two's headline number, which are two assumptions rather than one.

Set the recovery term y to zero, which is generous to nobody and close to true for a classifier that has no route back to the source, and x comes out at 0.832. Step two's pass rate on the inputs step one actually produces when step one is correct is 83%, not 97%. Fourteen points of the gap live there, before any question of what step two does with step one's mistakes.

The step from the equation to the 0.832 is one division and it is worth doing rather than accepting. With y at zero the second term vanishes entirely, so 0.79 equals 0.95 times x, and dividing 0.79 by 0.95 gives 0.832. Two different gaps then need keeping apart, because they get conflated and they measure different things. The gap between what independence predicted and what the pipeline delivered is 92% against 79%, thirteen points, and that is the size of the surprise. The gap between step two's dashboard and step two's real conditional rate is 97% against 83%, fourteen points, and that is the size of the measurement error. The first is the symptom and the second is the cause, and they are close in magnitude here because the recovery term really is near zero.

A gap of one or two points is worth checking against the sampling error, which at these rates and n=1,000 is about 1.3 percentage points. Thirteen points is ten times that, and there is nothing to attribute to noise.

That 1.3 is the usual expression evaluated at the pipeline's own rate rather than at either step's: the rate times one minus the rate, divided by the number of emails, square-rooted. At roughly eight in ten over a thousand emails the multiplication gives about seventeen in a hundred, the division gives about seventeen in a hundred thousand, and the square root is the 1.3 points. The reason to evaluate it there and not on step two's 97% is that the measured quantity in dispute is the end-to-end number, and a proportion's noise depends on where the proportion sits — a rate near a half is noisier on the same sample than a rate near one. Two mechanisms produce a conditional collapse of that size, and a pipeline can have both.

Correct upstream output does not look like hand-written gold#

The first mechanism is a distribution mismatch, and it is entirely mundane. A downstream eval set written before the upstream step existed — or written by the engineer who owns the downstream step, working from the schema — is complete, normalised and canonical: every optional field populated, dates written as 2026-03-14, names spelled correctly, currency as a decimal number. Real upstream output that a human grades as correct looks nothing like it.

Every one of those rows is an extraction that faithfully represents what the customer asked for, and a human grading step one marks it right. Step two has simply never seen anything like it. Where its prompt carries two canonical examples and a null shows up in a field those examples always populated, it reads the absence as evidence rather than as an absence.

The fix is mechanical. Build the downstream eval set by running the upstream step over the corpus — the fixed collection of real inputs the pipeline is measured against — and freezing what it produces, then labelling the correct answer for each of those actual records:

def build_downstream_eval(corpus, step_one, n=1000):
    rows = []
    for email in corpus[:n]:
        extraction = step_one(email)                 # the real thing, warts included
        rows.append({
            "input": extraction,
            "upstream_ok": human_label_extraction(email, extraction),
            "gold_class": human_label_class(email),  # from the EMAIL, not the extraction
        })
    return rows

Two details carry the weight. The eval input is step one's output rather than a hand-written record, so the distribution is right. And gold_class is labelled from the original source rather than from the intermediate, so that when step one drops a reference number, the correct answer is still what the source says and step two's failure to produce it is recorded as a failure rather than defined away.

The diagram is the whole argument in one picture: eval A never touches the corpus, so nothing it measures is downstream of anything the pipeline does, while eval C is the only one whose input distribution is the pipeline's own.

Keep the first one. It is a useful ceiling and it tells you whether a downstream regression is the step's own fault when the third number moves. What it cannot be is the number anyone quotes in a planning meeting, because it is the only one of the three with no relationship to what the pipeline delivers.

Confident processing of a plausible wrong value#

The second mechanism is what happens on the 5% of inputs where the upstream step is wrong, and that half of the story behaves worse than the arithmetic allows for. When step one extracts a wrong-but-plausible value, step two has no way to know it is wrong, so it processes with full confidence and produces an output that looks exactly like a good one. Nothing downstream flags it, because there is nothing to flag: the JSON is valid, the class is one of the permitted ones, and the confidence is high.

The version where step two is a validator is sharper. Extraction passes its own eval at 96%, validation at 98%, end to end 84% — a ten-point gap below the 94% independence predicts. A validator's job is to catch bad values, so a naive reading says it should raise the end-to-end number above what the extraction alone achieves, and instead it is destroying ten points. What is happening is that a total extracted as £5,400 instead of £540 is inside a legitimate approval band, satisfies every business rule the validator checks, and gets approved. The validator was scored on gold inputs where the wrong values were obviously wrong — negative totals, missing VAT numbers, dates in the future — and the wrong values it actually receives are the ones that look fine.

So on upstream failures a validator is not neutral and not random; it is systematically wrong, because the same plausibility that fooled the extractor fools the check. That is why the gap exceeds what compounding predicts, and it is why an amplified fault label is worth keeping separate from an inherited one.

The general form is worth extracting, because it applies to every check placed downstream of the thing it checks. A check catches a defect only where the defect leaves a trace of the kind the check looks for. An extractor produces a wrong value by reading the document in a way that seemed plausible to it, so the values it gets wrong are, close to by definition, the ones that look plausible — and looking plausible is precisely the property a range check, a schema and a business rule are testing for. The population of errors a validator meets in production is therefore the population its criteria are worst at. Score it on obviously wrong inputs and you have measured it on a distribution its own upstream step almost never produces, which is the first mechanism again, arriving somewhere it does more damage.

The full-distribution eval needs one addition the conditional one does not: a defined correct behaviour for bad inputs. Where the upstream step is wrong, the right answer for step two is usually not an answer at all — it is to flag, abstain, or route to review. Scoring against that expectation converts a category of silent amplification into a measurable abstention rate.

An eval set can expire without anything failing#

The operational cost of a distribution-matched eval is that it has to be rebuilt whenever the upstream step changes materially, because the upstream output distribution is the input distribution. On this pipeline that cost arrived as an incident three weeks after the fix. Step one's owner added a line_items array to the extraction schema to support a new report, which is a change with nothing wrong with it: step one's own eval went from 95% to 96.5%, review passed, and it shipped on a Tuesday. End-to-end accuracy fell to 76% over the following four days, and step two's eval kept reporting 94% throughout, because it was still being run against extractions frozen before the schema change. Step two had never seen a line_items field, and on multi-item orders it read the array as evidence of an amendment to an existing order.

Nothing failed. No test went red, no schema validation rejected anything, no alert fired, and the two owners each had a green dashboard for their own step. What makes this class of incident hard is that the invalidated artefact is an eval set rather than code, and eval sets have no compiler and no type system telling you they are out of date. The check that catches it is small and belongs in CI rather than in a runbook:

STEP_ONE_FINGERPRINT = "sha256:9f2c1b7e"   # prompt text + model id + decoding params

def test_downstream_eval_matches_upstream():
    assert fingerprint(step_one) == STEP_ONE_FINGERPRINT, (
        "step one changed; regenerate step two's eval set before merging"
    )

Regenerating the set is cheaper than the fear of it suggests, which is the argument that gets this accepted rather than waived. Re-running step one over the 1,000-email corpus cost about $4 and eleven minutes. Of the 1,000 records produced, 118 differed from the frozen set in any field, so only those 118 needed a human to re-confirm the gold class, and the other 882 — the 1,000 less the 118 that moved — carried their labels forward on a hash match. That is roughly forty minutes of annotation per upstream change, which is a real tax and is two orders of magnitude cheaper than four days of a pipeline running at 76% while every dashboard reports health. Most of the work of regenerating the set is done by the hash match rather than by a person, which is why the tax scales with how much the upstream step's behaviour moved rather than with the size of the corpus.

The same tax applies in the other direction and gets noticed less. Improving step two changes nothing about step one's eval, so a downstream fix never invalidates an upstream set — but improving step one changes the input distribution even when the schema is untouched. A prompt edit that raises extraction accuracy from 95% to 96% moves the mix of defects the classifier sees, retires some failure patterns and can surface new ones, and the downstream conditional pass rate measured before the edit no longer describes anything. Teams read that as a regression in step two and go looking for it in step two's prompt. It is not there.

The shape of that hazard is worth stating generally, because it recurs wherever one artefact is derived from another. A downstream eval set is a derived artefact: its inputs were produced by its predecessor, so it is valid only for as long as that predecessor is unchanged. Code has tooling for exactly this situation — a compiler that refuses to link against a changed signature, a lockfile recording what a build was made from — and an eval set has none of it, because it is data and data does not get compiled. The fingerprint is the smallest thing that gives a derived data artefact the property code gets for free, which is that it refuses to be used against the wrong version of the thing it came from.

The pinning is the part to get right, because it is what makes the tax bounded. Recording the exact prompt text, model identifier and decoding settings the stored inputs were generated with turns "did anything upstream change" from a judgment call into a hash comparison, and it is the only mechanism in this chapter that runs without anybody remembering to run it.

Independence is the prediction, and the gap below it is the measurement request. When a pipeline lands well under the product of its steps' pass rates, the number to go and get is the downstream step's conditional pass rate on real upstream output — not a better prompt, not a bigger model, and not another round of arguing about which owner is right. That number is usually the whole gap, it is available in the time it takes to re-run one step over a corpus you already have, and it converts a disagreement into a sorted list of failure patterns.

Fixing the evaluation tells you how much you are losing at the boundary and where. It does not tell you what the boundary should carry, and there is a class of loss that no amount of re-scoring recovers — the case where step one has already thrown away the information step two needs, and step two has nowhere to go and get it.