Rules first with the model on the residual changes the arithmetic

The slide had two columns. On the left, the incumbent: a rules-and-lookup classifier that sorts two million inbound documents a month into twenty-three types — invoice, remittance advice, purchase order, credit note, delivery note, statement, and so on — at 91% accuracy and no marginal cost, because it runs in-process on machines the company already owns. On the right, the proposal: a model classifier at 96% accuracy and $0.0008 per document, which is $1,600 a month. Five accuracy points for the price of one contractor-day. The room agreed within ten minutes.

Both numbers on that slide are real measurements, and putting them side by side compares two systems that nobody should build. The 91% is what the rule engine scores when it is forced to produce a label for every document, and the 96% is what the model scores when it is shown every document, including the 1.5 million that the rule engine already classifies at 99.2%. The interesting design was not on the slide.

Where the 91% actually comes from#

The rule engine does not have a uniform 91% opinion about documents. It fires on documents it recognises — a known sender domain, a form identifier that optical character recognition (OCR) lifts from a fixed position on the scanned page, a regex over the first 200 characters that matches a template the company has been receiving for six years — and on those it is right 99.2% of the time. It fires on 78% of the volume, which is 1,560,000 documents a month. On the remaining 440,000 it has no matching rule, and the production system defaults those to the majority class, invoice, which happens to be correct 62% of the time.

That default is where the 91% comes from:

0.78 × 0.992  +  0.22 × 0.62  =  0.774 + 0.136  =  0.910

Nothing about the rule engine's competence produces 91%. It produces 99.2% on the part it understands and a coin-weighted guess on the part it does not, and the average of those two is a number that describes neither. The moment the engine is allowed to say "I don't know" instead of guessing, the 440,000 abstentions become a routable stream, and the only question left is what to do with a stream that is one fifth the size of the original.

Getting the 78% and the 99.2% out of an existing rule engine is usually a day of work rather than a research project, because the engine already knows internally which rule matched and simply throws that information away when it applies the default. Instrumenting it to emit matched_rule_id alongside the label, then joining a month of those events against the operations team's corrections in the document management system, produces a coverage-and-precision figure per rule. That per-rule breakdown matters more than the aggregate: in this system, eleven rules cover 52% of volume at 99.9%, another forty cover 25% at 98.7%, and a long tail of ninety-odd rules contributes the last 1% at 84%, where deleting them and letting those documents abstain is worth more than any tuning.

The comparison that was missing#

Sending only the residual — the documents the rule engine declined to label, once it is allowed to abstain — to the model changes both terms. Cost falls in proportion to volume: 440,000 × $0.0008 is $352 a month rather than $1,600. Accuracy does not simply carry over, because the residual is the hard part of the distribution — the documents the rules could not recognise are unfamiliar senders, scanned photographs, multi-document PDFs and foreign-language forms. Measured on a 4,000-document labelled sample drawn only from abstentions, the model scores 89%, seven points below its all-documents figure. The blend is still better than either system alone:

0.78 × 0.992  +  0.22 × 0.890  =  0.774 + 0.196  =  0.970

Ninety-seven per cent at $352 a month beats ninety-six at $1,600, and the gap is wider than one accuracy point suggests, because the errors sit in different places. The all-model design puts 4% error uniformly across the whole stream, including 62,400 mistakes a month on documents from senders the company has processed correctly since 2019 — errors that look inexplicable to the operations team and destroy their confidence in the system. The cascade concentrates its errors in the unfamiliar 22%, where operations already expect trouble and where a review queue is a natural place to put it.

Accuracy stated as a single percentage also hides that the twenty-three document types do not cost the same when confused. Mistaking a remittance advice for a statement causes a clerk to re-key a record, which is minutes. Mistaking a credit note for an invoice pushes a document into the payment run, where the error is caught by reconciliation about 80% of the time and paid otherwise, at a mean recoverable exposure of $2,400 per instance. Weighting the confusion matrix — the grid showing how often each true type gets labelled as each of the other twenty-two — by those costs rather than counting errors changes which of two designs at the same headline accuracy is preferable, and it is the calculation that justifies routing a specific document type to human review regardless of what the confidence score says.

Two latency arguments that point opposite ways#

The rule engine answers in about 0.4 ms at the median: a dictionary lookup against a 40,000-row sender table plus a handful of compiled regexes, all in-process. The model path, including network, queueing and the provider's own generation time, measures p50 640 ms, p95 2,000 ms and p99 3,400 ms, where pN is the time that N per cent of calls come in under and the p99 is therefore the experience of one call in a hundred.

The blended median is 0.4 ms, because 78% of documents never leave the process, so the median document is a rules document. Quoted on its own, that number makes the cascade look like a latency improvement, which is a claim worth being careful about, since one document in five now waits somewhere between half a second and three and a half.

The blended p99 requires a step of arithmetic that teams routinely skip. The blended distribution is 78% of a near-zero mass and 22% of the model's distribution, so the 99th percentile of the blend is the point where the model's own cumulative distribution function — the share of its calls that have completed by a given time — reaches (0.99 − 0.78) / 0.22 = 0.955 — roughly the model's p95.5, which is about 2,100 ms. Compare that to the all-model design, where the p99 is the model's p99 at 3,400 ms. The cascade improves the tail as well as the median, and by a smaller factor than the median figure implies.

The review line is the one that changes the decision, and it is absent from most proposals. Routing the 6% of residual documents where the model's confidence falls below 0.85 into a human queue costs $8,580 a month, which is twenty-four times the inference spend and roughly five times the entire cost of the all-model design's inference. Once that number is on the table, the productive engineering question stops being which model to use and becomes how to shrink the review queue — by raising rules coverage, by improving calibration, so that a stated confidence of 0.85 really does correspond to being right about 85% of the time and the threshold admits fewer documents, or by accepting a higher error rate on the cheapest document types where a mistake costs a re-keying rather than a payment.

Retries and provider failures are smaller but they behave in a way worth anticipating. About 1.8% of model calls return something unparseable or a label outside the twenty-three-value enum; retried once, that costs another $6 a month and adds a second full round-trip for those documents, which pushes them into the tail that the p99 calculation above is measuring. Hard failures — 5xx responses and rate-limit rejections that survive retry — run around 0.05%, which is 220 documents a month on the residual path against roughly 1,000 on the all-model path. Both numbers require a dead-letter queue, which is where a document goes when every retry has failed so that nothing is silently dropped, and a replay path, so the work exists either way; the cascade makes it a quarter of the size.

What the cascade costs you#

Two systems now sit in the path, and both need to be kept honest. The rule engine's 99.2% is a measurement on a slice defined by the rules themselves, which makes it exactly the kind of number that drifts silently: when a large supplier changes its invoice template, the rule keeps matching on the sender domain and starts producing confident wrong labels, and nothing in the design notices, because the rules layer has no confidence score and no abstention for that case. Chapter 3's condition applies to the deterministic half of the system as much as to the model half, so per-rule accuracy needs sampling on its own cadence — twenty documents per high-volume rule per month is enough to catch a template change inside four weeks.

The abstention threshold is a tuned parameter with its own maintenance burden, and pushing rules coverage upward stops paying sooner than intuition suggests. Extending coverage from 78% to 88% by adding fuzzier rules leaves a residual that is 12% and considerably nastier, where the model might manage 82%; the blend comes out at 0.88 × 0.992 + 0.12 × 0.82 = 0.971, which is a tenth of a point for a quarter's work. Worse, fuzzy rules do not abstain honestly. A rule that fires on "the word INVOICE appears anywhere in the document" will match a remittance advice that references an invoice number, and it will do so with the same silent confidence as the well-founded rules, dragging the 99.2% down where nobody is looking.

None of this arithmetic can rescue a problem that lacks a verifier. Every figure above — the 99.2%, the 89%, the 62% majority-class rate, the confidence threshold — comes from labelled documents, and the labels came from the operations team's existing corrections in the document management system, which is chapter 1's fourth row again. A proposal with no such source produces a cascade whose coverage and accuracy are both unmeasurable, and a cascade you cannot measure is worse than a single system you cannot measure, because it has two failure surfaces instead of one. Eugene Yan's The First Rule of Machine Learning: Start without Machine Learning makes the general case for reaching for the deterministic baseline first; the specific case here is that the baseline is also what tells you how much problem is actually left.

The arithmetic settles which of two designs to build for a problem already worth building. It has nothing to say about proposals that are not yet a problem at all — the ones phrased as an ambition rather than a task, where there is no volume to divide, no residual to route, and no accuracy figure because there is nothing to be accurate about. Those need a different move before any of this applies, and it is the move that saves more proposals than it kills.