Decision records and reopen triggers

Classifier-versus-model choices are made against facts that expire. Traffic volume moves, the taxonomy is revised, a product change alters the shape of the input, and the price of a model call falls. Each of those can invalidate a decision that was correct when it was taken, and none of them announces itself as such — the decision simply becomes wrong while continuing to look settled.

That last phrase is doing more work than it looks and the mechanism under it is worth spelling out, because it is the reason a record is an engineering artefact rather than paperwork. A decision is a conclusion drawn from premises. The premises here are measurements: this arm scores this much, the volume is this, the budget is this, the price is this. A conclusion inherits nothing from its premises once it is stated — the sentence "we chose the classifier" contains no reference to the traffic figure it depended on, so when the traffic figure changes there is nothing in the system that points from the changed number back to the conclusion resting on it. Nobody is being careless. The link genuinely does not exist unless somebody builds it. A decision record is that link, written down: the premise list, and a monitor on each premise, so that a number moving in a dashboard can reach the conclusion it invalidated.

What survives that is not the conclusion but the reasons, recorded in a form somebody else can test. That means the measured numbers, the thresholds they were compared against, who set those thresholds, and — the part most teams omit — the specific conditions under which the answer would change, each phrased tightly enough that a dashboard can decide whether it has happened.

The form is the architecture decision record, a convention that predates any of this: a numbered file in the repository, written once, superseded rather than edited, stating a context, a decision and its consequences. What a classifier choice adds to that template is measurement. An ordinary architecture record can leave its context as prose, because "we run on Postgres" does not have a value that drifts; a record about which system labels your inputs has a quality bar, a traffic volume, a latency budget and a price in it, and every one of those is a number that will be different in a year.

A decision where the accuracy ranking is not the decision#

Take a vendor-normalisation step: incoming invoice line items carry a vendor string as the supplier typed it, and the step maps that string to a canonical vendor ID. It runs 4 million times a month. Two options have been measured on the same evaluation set, under the same label policy.

The immediate temptation is to read down the accuracy column, note that 96.5% beats 94.1%, and ship A. That treats the ranking as the decision, and the ranking is not what anyone downstream consumes. The consuming team stated a bar: the reconciliation process tolerates 5% error as long as the errors are reviewable, because a mis-normalised vendor surfaces in a weekly exception report that a finance analyst works through anyway. Both options clear 95%. The 2.4-point difference is therefore invisible to every human and system downstream of this step, and buying it costs $28,800 a year and 900ms at p95 in a path that has a 400ms budget and could otherwise run in a millisecond.

The cost figure is one multiplication carried through two units, and both steps are worth seeing because either one is easy to skip. Four million calls a month at $0.0006 each is $2,400 a month, which is the figure in the table. Twelve of those months is $28,800, which is the figure in the paragraph. The monthly number is the one a provider's console shows and the annual number is the one a budget conversation happens in, and quoting a monthly rate into an annual comparison is the single most common arithmetic error in this whole exercise. Note also what the difference is being spent on: not accuracy the process can use, since both arms already clear the bar, but 2.4 points measured on an evaluation set and consumed by nobody.

Option A also fails a stated requirement outright, which the accuracy comparison obscures. The latency bar is not a preference; the normalisation runs synchronously inside invoice ingestion, which has a two-second end-to-end SLA covering four steps, and 900ms in one of them consumes nearly half the budget before the other three have run. An option that misses a hard constraint does not get to compete on the soft ones.

Where the 400ms comes from is worth reconstructing, because a budget somebody can derive is a budget they will respect. Four synchronous steps have to fit inside two seconds. Four steps at 400ms each is 1.6 seconds, which leaves the remainder of the second as slack for the network, the queueing between steps and the tail that a p95 does not cover. The bar is not somebody's preference for fast software; it is the two-second commitment divided among its consumers with a margin left over. That is also why 900ms cannot be traded against the accuracy column. It is not slightly over a target — it is more than twice the share this step was allocated, so shipping it makes some other step's budget unpayable, and none of those other steps are party to this decision.

The requirement that actually gates Option B is the reviewability clause rather than the accuracy number. "5% error as long as errors are reviewable" means the step must be able to say which of its outputs it is unsure about, and a calibrated confidence score is what supplies that. Thresholding at 0.75 sends roughly 6% of traffic to the exception queue and lifts precision on the auto-accepted remainder to about 98%, which is materially better than the headline 94.1% for the analyst who only ever sees auto-accepted rows. Getting that right is the engineering work of shipping B, and it is a completely different activity from squeezing out accuracy points.

Where the missing accuracy went is the part that makes the manoeuvre honest, and it can be recovered from the two figures already given. Overall accuracy is the auto-accepted slice's accuracy weighted by its share of traffic, plus the queued slice's accuracy weighted by its. The auto-accepted slice is the traffic left after roughly six in a hundred are diverted, and it runs at about 98%, which accounts for most of the 94.1% on its own. Subtract that contribution and divide by the diverted share, and the classifier is right on something like a third of the rows it flagged. Thresholding did not raise anyone's accuracy. It separated a small pile in which the classifier is barely better than guessing from a large pile in which it is nearly always right, and it told the finance analyst which pile they are looking at. That is the property the consuming team asked for when it wrote "as long as the errors are reviewable", and it is why the reviewability clause and not the accuracy column is what Option B had to satisfy.

What goes in the record#

decision: vendor-normalisation-classifier
date: 2026-08-08
owner: payments-platform
status: active
chosen: gradient-boosted classifier over TF-IDF features
rejected: LLM call (frontier tier, 6-shot prompt)

quality:
  bar: ">= 95% top-1 accuracy on canonical vendor ID"
  bar_set_by: finance-reconciliation (Priya N., 2026-07-30)
  bar_rationale: "errors surface in the weekly exception report and are
    corrected there; the process is built for a few percent of misses"
  measured:
    eval_set: vendor-norm-eval-v3 (2,400 rows, sha 7c1f9ab)
    label_policy: docs/vendor-label-policy.md@v2
    llm: 0.965
    classifier: 0.941
  reviewability: "required — chosen option thresholds at p<0.75,
    routing ~6% of rows to the exception queue"

traffic:
  volume: 4.0M calls/month (Jul 2026 actual)
  growth: +3%/month over trailing 6 months
  input_distribution:
    source: "supplier-typed vendor string on the invoice line item"
    observed: "11,400 distinct strings; 92% exact-match a canonical name"
    assumption: "strings are names, not free-form descriptions"

latency:
  budget: "<= 400ms p95 for this step"
  budget_rationale: "invoice ingestion SLA 2s p95 across 4 synchronous steps"
  measured: { llm: 900ms, classifier: 1ms }

cost:
  per_decision: { llm: 0.0006, classifier: ~0 }
  annual_all_in: { llm: 28800 + 9600, classifier: 17300 }

reopen_triggers:
  - id: input-distribution
    condition: "exact-match rate to canonical names drops below 85%"
    owner: payments-platform
    metric: dashboards/vendor-norm/exact-match-rate (daily)
  - id: quality-bar
    condition: "reconciliation raises the bar above 97% or removes review"
    owner: finance-reconciliation
  - id: volume
    condition: "sustained below 300k calls/month"
    metric: dashboards/vendor-norm/volume (weekly)
  - id: taxonomy
    condition: "canonical vendor list restructured, or >500 IDs added
      in a quarter"
  - id: price
    condition: "per-call price for the required tier falls below $0.00008"

Every field answers a question a future reader will have. The bar and who set it prevents the accuracy ranking from being relitigated by somebody who does not know a threshold exists. The evaluation set identifier and hash mean a later comparison can be run on the same set rather than a new one, which is the only way two measurements taken eight months apart are comparable. The input-distribution assumption is written as a claim about the world — vendor strings are names — which makes it something that can be observed to be false.

The step from that assumption to its trigger is the move worth learning, because most assumptions do not obviously have a number attached and this one did not either. "Vendor strings are names, not free-form descriptions" is a sentence about intent, and no dashboard can measure intent. The chain that turns it into a metric runs: if the strings are names, most of them will be names somebody has already put in the canonical list; a name in the canonical list matches it exactly; therefore the share of incoming strings that exact-match a canonical name is high, and observed at 92% it is. Now run the chain backwards. If the strings stop being names, they stop matching, so the exact-match rate falls. The rate is not the assumption — a supplier renaming itself would also move it — but it cannot stay high while the assumption is false, which is all a trigger needs. Look for that property when writing one: not a number that means the assumption holds, but a number that cannot look normal while it is broken.

The triggers are the part most records omit and the part that pays. Each names a condition specific enough to be checked, an owner, and where the number lives. A trigger phrased as "if things change significantly" fires never, because nobody can tell whether it has fired. A trigger phrased as "exact-match rate drops below 85%, measured on this dashboard, owned by this team" fires by itself.

The volume trigger is the one that catches people out, and it is worth sitting with because the direction looks backwards. It fires when traffic drops below three hundred thousand calls a month, not when it grows. The instinct is the other way round — more traffic sounds like more risk — and the instinct is wrong here for a reason the previous chapter's ledger makes plain. The LLM arm's disadvantage in this decision is almost entirely a per-call cost, and a per-call cost shrinks with volume. Growth therefore makes the recorded choice more right, and there is no point monitoring for it. A collapse in volume is what makes the rejected option's price stop mattering, at which point the accuracy column becomes the only live difference and the decision genuinely is open again. The general rule is that a trigger watches the number that carried the argument, in the direction that would weaken it. Recording growth: +3%/month alongside is what makes the trigger legible: compounded across a year that is growth of roughly forty percent, so a thirteenfold fall to three hundred thousand is not something drift will produce. If it fires, something discrete happened — a product retired, a customer segment lost — which is exactly the class of event worth reopening a decision for.

The price trigger encodes the same idea on the cost axis, and computing one for a record of your own is a division rather than a forecast. Take the annual cost gap between the two arms, divide it by the number of calls the year contains, and that is how far the per-call price would have to fall for cost to stop separating them. Record a figure below that, not at it: a trigger that fires exactly at parity fires when the decision has become a coin toss, which is the least useful moment to convene a meeting.

A record is also not the evaluation report, and the two want to live separately. The report is long, it contains per-class tables, confusion matrices and the errors somebody read, and it is regenerated every time the comparison is run. The record is short, it names the report by identifier, and it is written once. Teams that merge the two end up with a document too long to re-read in a meeting, which defeats the only purpose it had.

Some things do not belong. Any argument that the chosen approach is more modern, more principled or more future-proof is untestable, and untestable claims are what long meetings are made of. Vendor benchmark numbers for either option are irrelevant, because the only measurement that counts was taken on vendor-norm-eval-v3. Predictions about where model prices are going do not belong either — the price trigger encodes the same information in a checkable form, so the record says what price would change the answer rather than guessing when that price will arrive. And the deliberation itself, who preferred what and which meeting settled it, is history rather than input.

When a trigger fires#

Suppose a product change moves the input from a dropdown selection to a free-text box, which drops the exact-match rate against canonical names from 92% to something like 40%. That is the input-distribution trigger, it has a dashboard, and it fires before anyone files a quality complaint. Re-opening the decision then consists of three known steps: build a fresh evaluation sample from the new input distribution and label it under the existing policy, re-measure both arms on it, and check the results against the recorded bar.

That first step is where the discipline shows, because the tempting shortcut is to re-measure on the existing set. The existing set is a sample of dropdown selections, and dropdown selections have stopped arriving. Scoring either arm on it now measures how well it handles inputs the system no longer receives, which is a number that will look reassuring and mean nothing. What must not change alongside it is the label policy: the new sample gets labelled under the policy already written, so that the old and new figures differ because the inputs differ and not because somebody quietly redefined a correct answer in between. One thing moves, and it is the one thing that moved in the world.

Reading the new numbers against the record produces something more interesting than a straight flip. The classifier scores 71% on free-text inputs and the LLM scores 93%, which is 93.2% against a bar of 95%, so neither option qualifies on accuracy alone and the decision falls back to the reviewability clause that gated Option B in the first place. Thresholded at 0.70, the LLM's disagreement queue takes 11% of rows and leaves auto-accepted precision at 97.6%, which is what finance is actually consuming. The bar did not move and neither did the reasoning; the input did, and the same criteria produced a different winner with an extra condition attached.

The same subtraction as before says what is inside that queue, and it is worth doing because it is what makes the qualifier in the superseded record honest rather than a way of dressing up a failing score. Auto-accepted rows are the roughly eight-ninths of traffic left after the queue takes its share, running at 97.6%, and the same arm, before anything is thresholded away, runs at 93.2% across the whole population. Subtract the first contribution from the second and divide by the queue's share: the model is right on a bit under three in five of the rows it flagged. That is a coin-flip-grade pile, and it is eleven rows in a hundred rather than all of them. Notice that the arm chosen here is the one whose overall accuracy is below the recorded bar — the record permits that only because the bar was written together with a reviewability clause, and the clause is satisfied. A record without that clause would have forced a different and worse answer, which is the argument for recording what a bar was for alongside what it was.

decision: vendor-normalisation-classifier
date: 2027-02-11
status: active
supersedes: vendor-normalisation-classifier@2026-08-08
trigger_fired:
  id: input-distribution
  observed: "exact-match rate 92% -> 38% over 3 weeks, 2027-01-19"
chosen: LLM call (frontier tier, 6-shot prompt)
quality:
  bar: ">= 95% top-1"            # unchanged, re-confirmed with Priya N.
  measured:
    eval_set: vendor-norm-eval-v4 (1,800 free-text rows, sha 21ad3c9)
    llm: 0.932
    classifier: 0.714
  qualifier: "neither option clears the bar unaided; chosen option meets it
    on auto-accepted rows (97.6% at p>=0.70, 11% to the exception queue)"
latency:
  budget: "<= 400ms p95"
  exception: "normalisation moved out of the synchronous path,
    payments-ingestion ADR 2027-02-04"

Keeping the superseded record rather than editing it is what makes the pair readable a year later: the two files together say what changed in the world, and a single edited file says only what somebody currently believes. The record also constrains the re-opening usefully. The latency budget has not changed, so if the LLM still runs at 900ms p95, switching to it requires either a latency exception from the ingestion owners or an architectural change that moves normalisation out of the synchronous path — a question the record surfaces immediately rather than three weeks into the migration.

The record decays in a way the meeting will not notice#

The failure mode that follows a well-written record is quieter than the one that follows no record, and it takes about three quarters to arrive. Nothing in a file like this is self-maintaining. Dashboards get migrated and the vendor-norm/exact-match-rate path stops resolving; the evaluation set is regenerated under the same name and the recorded hash no longer matches anything; finance-reconciliation reorganises and the person who set the bar is not there to be asked; the owning team hands the service over and the new owners have never opened the file. Each of these leaves the record looking correct — status still active, triggers still listed — while none of the triggers can fire.

The Orphaned state is the one worth instrumenting, because it is entered silently. A quarterly job that parses every record, resolves each metric path against the dashboard API, checks that the named evaluation set still exists at the recorded hash, and confirms each owner maps to a live team turns a decayed record into a ticket. Fifteen records take a few seconds to check, the failures are almost always a moved dashboard, and the alternative is discovering at the next incident that the trigger everyone relied on has been pointing at a deleted panel since March. The check also gives the file a legitimate reason to be read on a schedule, which is the only mechanism that reliably keeps a new owner aware it exists.

Decisions not worth recording#

This is not a governance ritual to attach to every choice. Records earn their keep on choices with a recurring cost and a plausible reason to change, which describes classifier-versus-model decisions and does not describe most of what a team settles in a week. Writing one for a library upgrade produces a directory of documents nobody reads, which is worse than none, because it teaches everyone that these files are not load-bearing.

A record is also not a commitment. Nothing in the file above says the team will keep the classifier; it says what would have to be true for the classifier to stop being right. A record that reads as a promise invites people to defend it, and one that reads as a set of testable conditions invites people to check it. And it does not work without the measurement it points at: a trigger whose metric does not exist is a sentence, so the exact-match-rate dashboard has to be built at the same time as the classifier. The honest weakness is that a record distributes a small ongoing obligation — somebody has to keep those numbers alive — and teams that treat the document as the deliverable end up with triggers that were true for one quarter.

Two of the triggers in that file describe the world changing in a way that would justify paying substantially more per decision: the vendor list restructuring, and the input becoming free text. Those are not arbitrary. There is a short, specific set of conditions under which an LLM's flexibility is worth a hundred times the per-decision cost of a trained model, and volume is conspicuously not one of them, which the next chapter works through on two extraction jobs that arrive in the same week.