A PoC proves capability under the conditions you chose

The invoice PoC passed. Invoice-level exact match on the frozen 200 came in at 89.5%, three runs within a point of each other, against an 85% bar signed off on 2 March, and the review took eleven minutes. Six weeks later the same prompt, the same model snapshot — the dated, pinned build of the model, so that the provider's later updates cannot silently change the answer — and substantially the same code were processing the live accounts-payable mailbox, and the weekly report showed 71.2%.

Nothing had broken. No deployment was misconfigured, no dependency had shifted, and the model version was pinned. The 18-point drop decomposes cleanly into four things that were true during the PoC and quietly stopped being true afterwards, and every one of them was a decision somebody made in the first week of March without thinking of it as a decision.

What the PoC actually held constant#

A clinical trial reports two different things depending on how it was run. Efficacy is what a treatment achieves in a trial: selected patients, excluded comorbidities, monitored adherence, specialists administering it. Effectiveness is what it achieves in a clinic, where the patients are whoever walks in and half of them stop taking the tablets. The gap between the two is not fraud or bad science; it is the price of the controls that made the trial interpretable in the first place. A PoC is an efficacy study, and the standard mistake is reporting an efficacy number in a conversation about effectiveness.

The controls in this case were four. The frozen 200 was stratified by document channel to a mix chosen in early March. The set contained only standard supplier invoices, because credit notes and multi-page consolidated statements had been set aside as out of scope. An engineer sat with every run and, on eleven documents that came back truncated or empty, re-fetched the attachment by hand and re-ran it. And the whole evaluation moved 200 documents through the API over forty minutes at concurrency 8, which is well inside every rate limit the account had — the caps a model provider puts on requests and tokens per minute, above which calls are refused rather than queued.

The first row is arithmetic anyone can do the day the PoC ends. The PoC's own per-stratum results were 96% on PDF-native, 82% on scanned and 61% on photographed documents; reweighting those three numbers to the December channel mix gives 84.4% before anything else changes. That calculation needs no production data and no new labels, only the discipline of reporting per-stratum rates alongside the headline and asking someone in AP what the mix looks like in a month that is not March.

The third row is the one that is hardest to see from inside, because it does not feel like an intervention. Re-fetching an attachment that arrived truncated is obviously the right thing to do while you are debugging, it takes ninety seconds, and it does not get logged anywhere. Eleven documents out of 200 is 5.5% of the sample, handled by a human who is not in the production design and whose work was never costed. Every PoC has some version of this, and the way to find it is to ask what the engineer did with their hands during the evaluation runs.

Writing the conditions down while they are still visible#

The conditions are obvious in the week the PoC ends and unrecoverable three months later, because the people who set them will have moved on and nobody wrote down that they were choices. The remedy is a ledger produced as part of the PoC's output, alongside the number, listing each favourable condition, what production is expected to look like, and what it would take to retire it.

## Conditions under which 89.5% was achieved
Result: 89.5% invoice-level exact match (3 runs: 89.0 / 90.0 / 89.5)
Set: evals/invoices-200/ @ sha256:4f1c… frozen 2026-03-04

| # | Condition held favourable | Production reality | Retiring it costs |
|---|---------------------------|--------------------|-------------------|
| 1 | Channel mix fixed at 62/31/7 | varies monthly, 44/38/18 in Dec | 120 new labels from a live month: 8 h |
| 2 | Credit notes + statements excluded | 9% of volume | schema + prompt work: 5-8 d |
| 3 | Human re-fetch of 11 truncated docs | no operator in the loop | retry + dead-letter queue: 2 d |
| 4 | No rate limits reached, concurrency 8 | 900-doc spike on the 1st | backpressure + tier upgrade: 1 d |
| 5 | Model snapshot pinned, prompt frozen | snapshot deprecates in ~12 mo | CI eval on version bump: 3 d |
| 6 | Inputs from known suppliers, no adversary | supplier could alter a PDF layer | text-layer/render cross-check: 4 d |

Not measured at all: throughput under sustained load, cost at production
volume with retries, behaviour on non-EUR/GBP currencies, handling of
personally identifiable information (PII)
for invoices containing individual contractors' home addresses.

The "not measured at all" section does as much work as the table. A PoC that ran 200 documents has measured nothing about sustained load, and the absence of a measurement is not a passing grade. Writing it as an explicit list stops the reasonable-sounding inference that anything not flagged as a problem is fine.

Row 6 deserves a sentence because it is the condition teams almost never think of during a PoC and the one that produces the worst incidents. Every document in the frozen 200 came from a supplier with no motive to manipulate it, so nothing in the evaluation says anything about what happens when a supplier learns that invoices are read by a model and that a PDF can carry an invisible text layer saying one thing while the rendered page shows another. The PoC did not fail that test; the test was never run, which is a different statement and should be recorded as one.

Finding the human interventions after the fact is nearly impossible, so the practical technique is to capture them while the runs are happening. A one-line log entry for every manual action taken during an evaluation — re-fetched attachment, re-ran with a tweaked prompt, excluded a document as corrupt, fixed a label — costs nothing and turns into a section of the ledger. Teams that do this are usually surprised by the count; on this project the engineer would have said he intervened "once or twice" and the honest number was eleven documents plus four labels he corrected mid-run, which together account for about 7% of the sample. The rule that makes it tractable is that anything you do with your hands during a run is either part of the production design or is a favourable condition, and there is no third category.

What the result licenses you to claim#

With the ledger in hand, three kinds of statement can be separated. The licensed claim is narrow: on invoices resembling the frozen 200, with an operator available and no load, this model extracts all six header fields correctly about 89.5% of the time, with a 95% interval of roughly 84.5% to 93.1%. That is a real and useful finding, and it is what the two weeks bought.

The unlicensed claims are the ones that get made anyway. "The system extracts invoices at 90% accuracy" drops the population. "We can automate AP" adds an operating model, a review workflow and an integration that were never built. "It works" collapses all of it. The gap between the licensed and unlicensed versions is exactly the ledger, which is why the ledger's most valuable moment is not the review meeting but the third conversation afterwards, where somebody who was not in the room describes the result to somebody else who was not in the room.

There is a third category worth naming: claims the PoC positively refutes, which are cheap and often overlooked. This one refutes "photographed delivery documents are as readable as PDFs" with a 35-point per-stratum gap, and refutes "line-item extraction is the hard part" because header fields alone were harder than expected. Negative results have narrow intervals when the effect is large, and they are the part of a PoC that stays true longest.

Where the ledger changes behaviour most is in the estimate that follows the PoC. The question after a successful prototype is always how long until this is live, and the answer people give is anchored on how long the prototype took — two weeks, so perhaps six or eight to productionise. Adding up the retirement costs in the ledger gives a different and better-founded number: roughly 21 to 26 engineer-days of work that has nothing to do with model quality, before anything is deployed and before anyone has looked at monitoring or ownership. That estimate is not precise, and it is derived from a list of specific tasks rather than from a feeling about how nearly finished the demo looked.

A PoC is not a pilot, and passing is not readiness#

The neighbouring thing this gets confused with is a pilot, and the confusion is expensive because a pilot is what most sponsors actually want when they ask for a PoC. A pilot deliberately keeps the unfavourable conditions: real users, real inputs, real load, real failure handling, with the scope narrowed instead — one depot, one supplier segment, one month — and a working fallback when it goes wrong. Its purpose is to measure effectiveness rather than efficacy, and it costs several times what a PoC costs because everything the PoC waived has to be built.

Both are legitimate, and the sequencing matters: a pilot for a capability that has not been shown to exist wastes the operational work, and a PoC for a capability nobody doubts wastes the two weeks. What causes damage is running a PoC and reporting it as though it were a pilot, because the operational learning a pilot produces is precisely the learning that the PoC's favourable conditions removed.

It follows that a passing PoC establishes feasibility and nothing about readiness. The system in this chapter was feasible, which the 71.2% figure does not contradict — every one of the four causes is addressable, and a team that shipped the retry queue, extended the schema to credit notes and re-labelled against a live month would plausibly recover most of the 18 points. That work is a project with a cost, and the point of the ledger is that the cost is visible before somebody commits to a date, rather than discovered as a series of unpleasant weeks after the launch announcement.

All of this assumed a PoC that passed. The harder situation is the one where the number lands below the bar and there is a genuinely good argument for continuing anyway, which is where a precommitted criterion is either binding or decorative.