Classification and extraction with LLMs

Read these first

• Schema-constrained generation

What you will know

• Do a classification task with an LLM and measure its accuracy

  Use a fixed, representative test set with class-level precision and recall; aggregate accuracy can hide the minority errors that matter most.

• Compare it against a regex or a small classifier on cost and accuracy

  Keep the evaluation set and label policy constant, and include inference, maintenance, and failure-handling costs rather than token price alone.

• Decide which one ships, with numbers

  Record the quality threshold, traffic assumptions, latency budget, and cost per decision so the choice remains defensible when those constraints change.

• Explain when the LLM's flexibility is worth 100× the cost

  That premium is justified only when semantic variation, an open-ended label space, or rapid policy change outweighs deterministic behavior and predictable operations.

• Replace an LLM with a regex, lookup table, or small classifier and verify the result

  A successful replacement should preserve the required recall and reject ambiguous inputs explicitly instead of silently converting model errors into false confidence.

Chapters

1. 1

   Independently labelled evaluation sets

   Scoring 97% on 200 tickets whose labels came from marking the model's own output reasonable is a self-consistency figure, not accuracy. Building a set the model can actually lose to means fixing labels independently, before any prediction exists, and reporting precision and recall per class: refund requests at 4% of traffic can be missed almost entirely while aggregate accuracy still reads 91%.

   Read chapter
2. 2

   Majority-class and keyword baselines

   Before an LLM classifier ships, a constant predictor and an hour of regex run on the same evaluation set under the same label policy. The usual objection — that there is no labelled data for an alternative — contradicts the accuracy figure in the same document, and the twelve few-shot examples in the prompt are hand-labelled data. On one complaint router the arms came out at 58.0%, 84.5% and 93.0%, and the 19-point recall gap on billing disputes is what kept the model.

   Read chapter
3. 3

   TF-IDF and logistic regression

   The arm that competes with an LLM classifier on quality is a concrete artifact you have to be able to build: text turned into TF-IDF features, a logistic regression fitted on a few thousand labels, scored with the same per-class precision and recall. The learning curve plateaus around 0.88 macro F1 at a thousand labels per class, a fit takes 2.1 seconds, and it fails in three predictable ways — unseen wording, a label set that moves, and anything needing knowledge outside the corpus.

   Read chapter
4. 4

   Total cost per decision

   A token price compared against a classifier described as essentially free puts a fully loaded cost on one side and a marginal cost on the other. Over twelve months at 1.8 million decisions a month the honest totals are $26,260 and $17,290 — a 1.5x gap rather than the 400x the slide showed — and the ranking is stable across volume because the classifier's fixed costs sit $330 a year below the model's.

   Read chapter
5. 5

   Decision records and reopen triggers

   Write down the quality threshold, who set it, the traffic assumptions, the latency budget and the cost per decision, and a later product change becomes a documented trigger instead of a ninety-minute argument. The worked example is a vendor-normalisation step at 4 million calls a month where both options clear the stated 5% error bar, so the extra 2.4 accuracy points buy nothing while costing $28,800 a year and 900ms at p95 inside a 400ms budget.

   Read chapter
6. 6

   Conditions that justify an LLM premium

   Volume is a reason to avoid per-call cost, not a reason to spend it, because an LLM call has a small fixed cost and a marginal cost that scales. The premium is earned by a label space that keeps moving, semantic variation no pattern can track, or an output space with no class list at all — which is why a 40,000-a-month cancellation-reason job at $16 is a better home for a model than a 2-million-a-month fixed template at $1,800.

   Read chapter
7. 7

   Explicit no-match outcomes

   A regex plus a lookup table can match a model field for field on a 1,200-row evaluation set and still be unsafe to ship, because an unmatched IBAN arrives downstream as an empty field and reads as a vendor that legitimately has none. The replacement needs three outcomes rather than two, with the undetermined one routed to review — 1,044 documents a month at first, which is more analyst time than the model cost until the patterns catch up.

   Read chapter
8. 8

   Cascades and conditional accuracy

   Running a pattern extractor first and falling back to a model changes three things at once. The traffic-weighted average of two standalone accuracies is clean arithmetic and fiction, because the router hands the model an adversarially selected residual — a projection of 91.9% shipped at 89.4%. Below about a 56% hit rate the cascade costs more than the model alone, and having the model also review confident matches puts it back on 100% of traffic.

   Read chapter