Cascades and conditional accuracy

A cascade is a chain of stages ordered by price, in which each input stops at the first stage willing to answer it and only the leftovers reach the expensive one. For a text extraction step that usually means a pattern extractor first and a model second, and the arithmetic that justifies it is a single line: patterns cover 70% of traffic and score 91% standalone, the model scores 94% standalone, so the cascade lands at 0.70 × 0.91 + 0.30 × 0.94 = 91.9%, two points below the model alone, at 30% of the token bill.

The multiplication in that line is worth doing slowly, because the formula itself is correct and it will be used again in a moment with different inputs. Seven-tenths of the traffic is handled at 91% right, contributing a bit under sixty-four in a hundred of the total. The remaining three-tenths is handled at 94% right, contributing a bit over twenty-eight. Add the two contributions and the answer is 91.9%. That is exactly the right way to combine two accuracies measured on two disjoint slices of traffic, and nobody should be talked out of it.

Every number in that line can have been measured on the same evaluation set under the same label policy, which is more rigour than most proposals carry, and the line is still wrong. Both standalone figures describe a system's behaviour over the full input distribution, and in a cascade neither system faces that distribution any more.

The defence available at this point is the strongest one in the topic, and it is worth naming because it holds up against every check a reviewer normally runs. Both accuracies came off the same evaluation set. The set has independent labels and a written policy. Neither arm was scored on data it was tuned on. All of that is true, and none of it is the problem — the inputs to the formula were measured rigorously, they are simply not the quantities the formula requires. A weighted average of accuracies over disjoint slices needs each arm's accuracy on its own slice. A standalone figure is an accuracy over the whole distribution, which is a different quantity that happens to be reported with the same word and the same units. Rigour in obtaining a number is no defence against using it to answer a question it does not answer.

The residual is not a random sample#

The patterns answer only where they match, which is where they are strongest, so their accuracy on the traffic they actually handle is 96.0% rather than 91% — the 91% included the 30% of inputs where they had to guess or default. The model receives only what the patterns failed on, which is documents with unusual phrasing, negation, two intents in one message, misspellings and layouts nobody anticipated, and on that residual it scores 74% rather than 94%.

The first of those two corrections can be recovered from the standalone figure, and doing so makes the direction of the error concrete. The 91% is a blend over the whole distribution: it is the 96.0% on the seven-tenths the patterns actually match, blended with whatever they scored on the three-tenths where nothing matched and something had to be returned anyway. Subtract the matched part's contribution from 91 and divide by the unmatched share, and the defaults turn out to have been right roughly four times in five. A default can do that on a skewed distribution — always answering with the commonest label is right as often as that label is common. So the standalone 91% is partly a measurement of how well a fallback guesses, and in a cascade those inputs never reach the fallback at all. The figure was never describing the thing the cascade would do with them.

The router selects for difficulty on purpose. A pattern fails to match precisely when the input departs from the forms somebody wrote down, and departure from expected form is also the main driver of model error, so the two failure modes are correlated rather than independent. That correlation is why the arithmetic breaks, and it is why the effect is not a small correction.

The measured end-to-end figure comes out of the same formula as the projection, which is the detail that makes the table worth staring at. Seven-tenths at 96.0% right contributes a bit over sixty-seven in a hundred. Three-tenths at 74% contributes a bit over twenty-two. The sum is 89.4%. Nothing about the arithmetic changed between the two columns — the shares are identical, the operation is identical, and the answer moved by two and a half points because two inputs were replaced by the numbers those inputs should have been all along. The lesson generalises past cascades: when a projection and a measurement disagree, the formula is the last place to look.

Two readings matter. The projection was 2.5 points optimistic, and the direction of that error is not random — the decomposition is optimistic every time, because the router always hands the second stage the harder half. And the shipped cascade is 4.6 points below simply sending everything to the model, which is the option the proposal was implicitly arguing against.

The only honest measurement is end to end, on the full evaluation set, with the router in the loop.

def evaluate_cascade(dataset):
    rows = []
    for row in dataset:
        out = patterns.extract(row["text"])
        arm = "patterns"
        if out.status is Status.UNDETERMINED:
            out, arm = llm.extract(row["text"]), "llm"
        rows.append({"true": row["label"], "pred": out.value, "arm": arm})

    overall = accuracy(rows)
    by_arm = {a: accuracy([r for r in rows if r["arm"] == a])
              for a in ("patterns", "llm")}
    routed_share = {a: sum(r["arm"] == a for r in rows) / len(rows)
                    for a in ("patterns", "llm")}
    return overall, by_arm, routed_share

The by_arm figures are conditional accuracies and they are not comparable to anything measured standalone, so they belong in the report labelled as such. A cascade report that lists "LLM: 94%" next to an end-to-end number invites the next reader to redo the wrong arithmetic.

Measurement gets harder in three ways at once#

Sample size is the first. The residual is 30% of the set, so a 500-row evaluation set gives the model arm 150 rows, and a minority class appears in it perhaps a dozen times. Every conditional number reported about the model arm carries an interval wide enough to hide a real regression, and fixing that means enlarging the evaluation set by roughly the inverse of the residual share — a 1,700-row set to get the model arm back to 500 rows of support.

That inverse is the whole calculation and it is worth seeing as a division rather than a rule of thumb. The model arm sees three-tenths of whatever the set contains, so the rows it sees are the set size multiplied by three-tenths. Turn that round: to get some target number of rows in front of the model, divide the target by three-tenths, which is the same as multiplying by a bit over three. Five hundred divided by three-tenths is a little under seventeen hundred. The unwelcome part is what the multiplier is made of. It is the inverse of the residual share, so a better cascade — one whose patterns cover more traffic — needs a larger evaluation set, not a smaller one, because it starves the arm whose behaviour is hardest to predict. Push the hit rate to nine-tenths and the same five hundred rows of model support costs an evaluation set of five thousand.

Coupling is the second and it is worse. A change to the patterns changes which inputs reach the model, which changes the model's conditional accuracy, which means the model arm's number expires whenever anyone edits a regex. Adding a pattern that lifts the hit rate from 70% to 78% moves eight points of the easiest residual traffic out of the model's share, and the model's conditional accuracy falls even though the model is unchanged and the end-to-end number improved. A dashboard tracking the model arm in isolation will show a regression that is not one, and the team will spend a day looking for it.

Attribution is the third. When end-to-end accuracy drops two points, the cause is a pattern that started matching things it should not, a shift in the input distribution, a model version change, or the router; the four are entangled, and the log has to record which arm answered each decision, with the rule that fired, or none of them can be separated after the fact. That is the same field an extractor carries for review purposes, doing a second job.

The hit rate that makes it worth doing#

Cost has a break-even and most proposals never compute it, because the token saving is visible and the rest is not. Take a volume of 1.8 million decisions a month at $0.0004 a call, and price the operational rows monthly at the same engineer-day rate the ledger used.

Sending everything to the model costs $720 in tokens plus prompt maintenance and the retry, rate-limit and dead-letter path, which together run about $600 a month, for $1,320. The cascade keeps every one of those rows — it still calls the model, so it still needs all of the failure handling — and adds pattern maintenance and a second evaluation path with its own harness and its own report, at roughly $400 a month combined. The token bill is the only line that shrinks.

Each token row is the traffic the patterns did not handle, priced at the per-call rate. At a 70% hit rate the model still sees three-tenths of 1.8 million decisions, which is five hundred and forty thousand calls, and that many at $0.0004 is $216. The ops column does not move at all across the table, because it is the model-alone operational rows plus the cascade's own, and neither of those depends on how often the patterns match. That flatness is why the arrangement has a break-even rather than a smooth advantage.

Solving for it takes one line, and the line is more useful than the number it produces. The two arrangements cost the same when the money the patterns save on tokens exactly equals the money they add in operations. What the patterns save on tokens is the hit rate multiplied by the full token bill, because every input they answer is one call not made. What they add in operations is the gap between the two ops figures, which here is the pattern maintenance and the second evaluation path. Set those equal and divide, and the break-even hit rate is the extra operational cost divided by the full token bill — four hundred dollars against seven hundred and twenty, which is a shade under 56%. That form is worth carrying, because it says which quantities move the answer. Anything that shrinks the token bill, a cheaper model or lower volume, raises the hit rate a cascade must reach. Anything that makes the patterns cheaper to maintain lowers it. The accuracy figures appear nowhere in it at all.

A proposal promising a fraction of the cost delivers, at a 70% hit rate, 8% — in exchange for 4.6 accuracy points and a second system. Below a 56% hit rate the cascade costs more than the thing it was meant to replace, which is the specific condition worth naming: whenever the cheap arm covers less than about half of traffic, you are paying for the model on most inputs anyway and paying to maintain a router and a rule set on top.

Hit rates also decay. Patterns match the forms that existed when they were written, and a stream that gains new vendors, new products or a new customer segment pushes traffic into the residual month by month, which is the same silent erosion an unseen vocabulary causes a trained classifier. A cascade launched at 70% and left alone drifts toward its break-even, and the maintenance line in the table is what holds it up rather than an optional extra.

Volume moves this more than anything else. At 20 million decisions a month the token bill dominates every other row, and the same 70% hit rate saves about 60% of total cost, which is a real result. That is the same principle that applies to a single arm: per-call cost is what scales, so the case for spending engineering effort to avoid it gets stronger as volume rises, and at 40,000 decisions a month a cascade is a second system built to save $10.

Both of those follow from the break-even expression rather than from any new principle. At twenty million decisions a month the token bill is more than ten times the figure above, so the fixed operational rows are a small fraction of it, and a hit rate of 70% removes seven-tenths of much the largest line in the ledger — which is where the 60% total saving comes from. At forty thousand a month the entire token bill is a couple of tens of dollars, so seven-tenths of it is a couple of tens of dollars saved, and no arrangement of patterns can save more than the whole bill however good it is. That is the ceiling worth checking before any of this: compute the full token bill first, because it is the most a cascade can possibly return, and if that ceiling is smaller than the cost of maintaining a rule set the answer is already decided.

Latency behaves in a way that surprises people who have only looked at the mean. Median latency collapses, because 70% of requests now finish in under a millisecond. The 95th percentile does not move at all, because at a 70% hit rate the slowest 30% of requests are still full model calls, so p95 is a model call plus the pattern attempt. An SLA written at p95, which most are, sees no benefit from a cascade until the hit rate clears about 95%.

The reason those two percentiles behave so differently is that a percentile is a position in a sorted list rather than an average of anything. Sort every request by how long it took. The fast ones are pattern hits and the slow ones are model calls, so the list is essentially the pattern population followed by the model population. The median is the request halfway along that list; with seven-tenths of requests in the fast block, halfway is inside the fast block, and the median is a pattern latency. The 95th percentile is the request ninety-five hundredths of the way along; with three-tenths of requests in the slow block, that position is inside the slow block, and stays inside it as long as the slow block holds more than five hundredths of the traffic. Only when the model's share falls below that does the boundary cross the 95th percentile and the number drop. Hence the hit rate of about 95%: it is not a performance threshold, it is the point at which the two populations swap sides of the percentile you happen to be measuring.

The safety net that puts the model back on everything#

That suggestion is worth working through rather than dismissing, because it is the natural next thought and its failure is arithmetic rather than aesthetic. The immediate consequence is that the model runs on 100% of traffic, so the token bill returns to $720 and the total becomes $1,720 a month, which is more than sending everything to the model and skipping the patterns entirely. Whatever the cascade was for, it is gone.

The quality case does not rescue it either. Sampling 10,000 documents where the patterns matched confidently and asking the model independently produced 520 disagreements; adjudicating those against the label policy, the model was right in 310 and the patterns in 210. Always deferring to the model therefore fixes 310 and breaks 210, for a net gain of 100 decisions in 10,000 — one point on the 70% of traffic the patterns handle, or 0.7 points end to end.

Two conversions happen in that sentence and each is a place the case gets overstated. The first is the subtraction: a rule that always prefers the model does not collect 310 wins, it collects 310 wins and 210 losses, because the disagreements where the patterns were right are resolved the same way as the ones where they were wrong. Netting them is not pessimism, it is what "always defer" means. The second is the scaling. A hundred net decisions in ten thousand is one in a hundred, but the ten thousand were all pattern-matched documents, so that point is a point on seven-tenths of traffic. Multiply by seven-tenths to express it end to end and it is 0.7. Reporting the raw figure as a one-point gain would be quoting an improvement in the wrong denominator — and against a projection that was itself two and a half points optimistic, 0.7 is inside the noise the chapter opened with. If 0.7 points is worth having, the cheaper way to get it is to delete the patterns and send everything to the model, which costs less than running both and yields the model's full-distribution accuracy on that traffic instead of a policy-mediated blend.

There is also a new measurement problem. The system now emits an output belonging to neither arm, produced by a resolution policy — model wins, patterns win on checksum-verified fields, disagreements go to review — and its accuracy is a property of that policy. Changing "model wins" to "disagreements go to review" changes the end-to-end number, the review queue volume and the cost, so all three have to be re-measured together, and the two conditional accuracies already being tracked are now conditioned on the resolution rule as well.

The version that does work is a review path with a budget: send a sampled 2% of confident matches to the model as an ongoing audit, alert when the disagreement rate on that sample moves, and never let the model's answer overwrite the pattern's in production. That costs $14 a month at this volume, detects the pattern rot described above, and leaves the accuracy accounting intact because nothing in the serving path changed.

Where cascades are the right shape#

A cascade is not a bad idea in general; it is a bad idea when the router keys on failure. Routing on a stable, observable property behaves entirely differently. An exact lookup of a known vendor ID that hits 88% of invoices is a router whose cheap arm is correct by construction, and whose residual — invoices from vendors seen for the first time — is not systematically harder for the model than the average invoice.

The property that distinguishes the two arrangements is worth stating in one line, because it is the thing to check on any router. Ask what the routing condition is a fact about. A match/no-match router keys on whether the cheap arm succeeded, which is a fact about the difficulty of the input, and difficulty is also what makes the expensive arm fail — so the condition and the second arm's error rate share a cause, and the residual is enriched for error. A vendor-ID lookup keys on whether this supplier has been seen before, which is a fact about the vendor's history with your company and not about how the invoice is written. A first-time vendor's invoice is new, not malformed. Because the condition carries no information about difficulty, the residual is close to a random sample of the distribution, the model's standalone accuracy does describe its behaviour on it, and the weighted average that failed at the start of this chapter becomes legitimate. The test is not whether a router is cheap or clever. It is whether the thing it keys on is correlated with the thing that makes the next stage wrong. The comparative literature on regex versus model extraction reports parity on exactly this kind of well-formed structured field, and A comparative performance analysis of regular expressions and a large language model-based approach to extract is worth reading for how narrow the conditions are.

Confidence-thresholded routing on a calibrated classifier is the better-behaved relative of a match/no-match cascade, because the router's output is a measured quantity rather than a binary accident. Sweeping the threshold produces a coverage-versus-accuracy curve, so you can read off what routing 5%, 10% or 20% of traffic to the model buys, and the residual's difficulty is described by the same probability that selected it.

What every working version of this has in common is that the numbers stay attached to their conditions. An evaluation set whose labels came from somewhere other than the system being scored, a baseline that runs on every release rather than once, a ledger with both sides in the same column, and a record naming the thresholds and the traffic assumptions with an owner against each — those are the parts that survive the first product change, and a cascade is simply the arrangement where forgetting them is easiest, because it produces two numbers that look like the ones you measured and are not.

The step in your own system most likely to need all of it is the one where somebody has already put a model and nobody has run the alternative. Pick it, label three hundred rows independently, run the pattern arm and the trained arm beside the model on the same set, and see which of the four conclusions in this topic the numbers hand you.