Promotion is a list of preconditions, and you will get only some of them

The meeting lasted nine minutes. The CTO had already decided that invoice extraction goes live for the top forty suppliers on 1 September, because the AP backlog is real and the PoC number was good, and what he wanted from the engineer was a list of what it needs. Then, before the list was finished, he said there were four weeks and one engineer, so pick the two that matter and he would take the risk on the rest.

That is the ordinary shape of this conversation, and the outcome depends almost entirely on whether the engineer walks in with a ranked list or assembles one under pressure. An unranked list gets triaged by whoever is holding the budget, which means it gets triaged by cost, and the two cheapest items win regardless of what they protect.

The candidates#

The last row is on the list to be ranked last, deliberately. The previous chapter covered why a rewrite is the demand that sounds strongest and changes least; putting it on the table with a price and a benefit is more useful than leaving it off, because it lets you concede something visibly early, which improves the odds on the items that matter.

The item this topic contributes that the others do not is the second-from-last, and it is the one most likely to be missing from a generic production-readiness checklist. Chapter 4 established that a PoC's result is inseparable from the conditions that produced it and produced a ledger of those conditions. That ledger is a work plan: each row is a condition that has to be retired, and retiring them one at a time is what keeps the production drop diagnosable.

Staged retirement, and why the order matters#

Stage 1 is the PoC's own population, running for real. That sounds like it learns nothing, and it learns the two things the PoC could not: what happens when the operator is a clerk following a procedure rather than the engineer who wrote the prompt, and what the integration does under a daily rhythm. Widening the input distribution before those are stable means any drop has two candidate causes and no way to separate them.

The order of the gates follows the sizes from chapter 4's decomposition. Document types outside the schema cost 5.8 points and channel mix cost 4.8, so those get their own stages; the operator and the load are retired last because they are the conditions whose removal is hardest to reverse once AP has reorganised around it. Each dotted arrow is a rollback to the previous stage rather than to keying, which matters because a rollback that costs the business its whole benefit is a rollback nobody executes.

The gates are measured on live data, which requires labels, which is what the weekly canary is for. What it watches for is drift, meaning a change in what the arriving documents look like rather than any change in the code, which the frozen set cannot show because the frozen set never changes. Fifty invoices a week reviewed by a clerk is about two hours of somebody's time and produces, over a four-week stage, 200 labelled live cases — the same sample size the frozen set uses, for the same reason, resolving roughly the same interval.

Wiring the frozen set into CI#

Of the technical items, the eval-in-CI one is the least familiar to most backend teams, so it is worth being concrete about what it looks like. The job runs the frozen 200 on any change to the prompt, the model identifier or the extraction code, fails the build below the bar, and reports per-stratum movement so that a change which trades scanned accuracy for PDF accuracy cannot hide inside a flat headline.

# .github/workflows/eval.yml
name: extraction-eval
on:
  pull_request:
    paths: ["prompts/**", "src/extract/**", "config/model.yaml"]

jobs:
  frozen-200:
    runs-on: ubuntu-latest
    timeout-minutes: 60
    steps:
      - uses: actions/checkout@v4
        with: { lfs: true }          # the 200 PDFs live in LFS

      - name: Verify the set has not been edited
        run: sha256sum -c evals/invoices-200/manifest.txt

      - name: Run
        env:
          ANTHROPIC_API_KEY: ${{ secrets.EVAL_KEY }}   # eval-only key, own budget
        run: |
          python -m extract.eval \
            --set evals/invoices-200 \
            --bar 0.85 \
            --by-stratum channel,supplier_tier \
            --max-spend-usd 15 \
            --baseline main \
            --out eval.json

      - name: Comment the diff
        run: python -m extract.eval_report eval.json >> $GITHUB_STEP_SUMMARY

Two details carry most of the value. --baseline main makes the report a paired comparison against the current production configuration rather than a bare percentage, so the summary reads "9 invoices fixed, 4 broken, net +2.5 pp, scanned stratum −6 pp" and the reviewer can open the four regressions. --max-spend-usd 15 stops a prompt change that triples token use from being discovered on the monthly bill; the eval is also the cheapest place to measure cost per invoice, since it runs the same documents every time.

Running the full set on every pull request costs about $7.40 and forty minutes. Teams that find that too slow usually settle on a 60-case smoke subset per commit and the full 200 nightly and on release, which is a reasonable trade as long as everyone remembers that the 60-case version resolves nothing near the bar.

Which two, and what the concessions buy back#

Given two, take the named owner with a kill switch under their control, and the frozen eval set in CI.

The owner and the switch are one item because either alone is inert. An owner without a way to stop the system is a person who gets paged and can only escalate, and a switch with no owner is a control nobody is watching for a reason to use. Together they make every other missing precondition survivable: an unbounded cost, an unhandled document type, a bad model upgrade all end the same way, with someone noticing and turning it off within an hour instead of fourteen. This is the item that would have changed the June outcome, and it costs one day of engineering plus a conversation with a manager about capacity.

The eval set in CI earns the second slot because it is the only precondition that keeps working after everyone stops paying attention. Attention decays predictably: the first month after launch has a dashboard on a wall and someone reading it daily, and by month four the dashboard is stale and the prompt has been edited twice for reasons nobody wrote down. A gate in CI is the only control on the list that continues to assert the PoC's claim against the current code, and it is what prevents the specific and very common failure where a system that launched at 89% is at 74% eight months later without a single incident having been raised.

Cost caps lose to those two because the exposure is bounded elsewhere. At the volumes involved, a runaway costs hundreds of pounds before the kill switch is pulled, whereas silent quality decay costs wrong payments that surface as supplier disputes months later. Retention and logging lose on the same reasoning for an internal AP tool, and would win on a different system.

The concession worth making explicitly is the staged retirement, and it can be converted rather than surrendered. Instead of four weeks of engineering for stages and gates, ask for stage 1's scope as a written constraint: top forty suppliers, PDF-native attachments only, clerk reviews everything, revisited on 1 November. That is a paragraph in a decision record rather than an engineering task, so it costs the CTO nothing to grant, and it restricts production to the population where the 89.5% figure actually applies. A concession that can be re-expressed as a narrower scope is not really a concession, and it is worth checking each item on the list for that property before giving it up.

The ranking above is not universal, and copying it into a different context is a way to be confidently wrong. For a system that moves money without a human in the path, or that faces customers, retention and an audit trail move above CI, because the first serious incident will be reconstructed by someone who is not an engineer and the absence of records is itself the finding. For a system with a low-cost reversal — an internal summarisation tool where a bad output wastes a minute — the honest list is short and the answer to "which two" might be the owner and nothing else. The ranking follows from what a failure costs and how fast it can be reversed, which is a judgement about the system rather than a property of the checklist.

What a list cannot do#

None of these preconditions makes the system good. They make it observable and reversible, which is a smaller and more achievable claim, and they leave the quality question exactly where the PoC left it — at 89.5% under conditions that are now being retired on a schedule. A team that gets all eight items and still cannot read handwritten annotations has a well-instrumented system that does not solve the problem.

Preconditions can also be granted on paper. A named owner who has no capacity is a name, which is why the owner item includes the manager's written agreement rather than just the individual's; a kill switch that has never been tested is a config flag with a hopeful comment, so it gets exercised once in the first week while everyone is still paying attention. The version of this list that gets checked off in a spreadsheet without either of those steps is the same failure the whole topic is about, arriving one level up.

Which is the point at which this becomes something you do rather than something you read. The artefacts are four short files, and they are worth carrying from one project to the next: the criteria file written before any prototype code, with the sponsor named and the bar, the population and the date in it; the assumption ranking that decides where the prototype days go; the conditions ledger written the week the result lands, while the choices are still visible; and the decision record that says what happened and what would change the answer. On the next PoC, write the first one on day zero — before the repository exists — and see how much of the argument you were expecting never happens.