Pricing a model migration

The headline number on a migration paper is a price per thousand calls, and it is correct. It is also one line item in a system that has several, and the others move in the opposite direction: a model that needs repairs costs repair calls, a model that fails more often costs human review, a model that fails quietly costs whatever a wrong value does after it lands, and a second model in production costs somebody's time keeping a second prompt variant alive.

Effective cost is the name for the sum of those on your own traffic. It is an arithmetic exercise with eight or so inputs, most of which you can measure and two of which you have to estimate, and its purpose is not precision. Its purpose is to establish which of the inputs the decision actually turns on, before three weeks of porting work makes everyone's judgement worse.

The failures you keep are priced by how they fail#

Two systems with the same accuracy are not equally expensive, because the cost of an error depends on where it lands. A failure that a validator rejects at the process boundary costs a human review, and a human review has a known price. A failure that validates and is wrong costs whatever the wrong value does downstream, discounted by the probability that anybody notices.

Working that through for an invoice extractor at 1.2 million documents a month, where the model call moves from $2.40 to $0.55 per thousand:

The line that decides it is the fourth. The patch in the previous chapter turned eleven loud failures into three silent ones on the frozen 400, and 3 in 400 is 0.75%, against 0.10% on the incumbent. A wrong-but-valid discount goes into the ledger, is caught only when someone reconciles a supplier statement, and costs roughly 40 minutes of a finance analyst's time at $56 an hour to trace and correct. Most are never caught at all, which is worse for the business and, perversely, cheaper on this table — the $23,088 is only the 8% that surface.

Every cell in that table is the same shape of multiplication, and doing two of them by hand makes the rest checkable rather than something to be taken on trust. The base-call line divides the volume by a thousand, because the price is quoted per thousand, and multiplies by that price: 1.2 million documents is twelve hundred thousands, and twelve hundred at $2.40 is $2,880. The human-review line is volume times a rate times a unit price, with no division, because a rate is already a fraction: 1.2 million at 0.15% is eighteen hundred documents, and eighteen hundred reviews at $0.62 apiece is $1,116.

The silent-error line has three factors instead of two, and the extra one is where the arithmetic gets its teeth. Volume times the rate gives the number of wrong values written. Multiplying by the detection rate gives the number anybody ever finds out about. Multiplying by the correction cost turns that into money. For the incumbent: 1.2 million at 0.10% is twelve hundred wrong values, of which 8% surface, which is fewer than a hundred, at $37 each. Read the row labels before reusing those two cells, because they are not stated on the same basis — the incumbent's rate is absolute and model B's is written as a rate above A, which is where 0.75% minus 0.10% comes from. The multiplication follows the label in each case, so the pair reports a difference and not two totals.

That gives a way to re-do the arithmetic on the other half of that patch decision. Dropping the default-to-zero sentence puts the eleven missing-field failures back and removes the three silent ones, which moves the rates to roughly 2.8% unrecoverable and 0.10% silent on model B:

Neither variant beats the incumbent's $7,548. The saving on the model call is real and it is 8% of the size of the effects it sets off, and saying so requires no argument about which model is better — it requires the same frozen set, the per-error-type breakdown, and a price on each failure class.

It also points at where a viable version of the migration would come from: the human-review price. At $0.62 per review the loud-failure variant is dominated by that line, so a change that makes review cheaper — batching, a UI that shows the model's output pre-filled against the document image, review only of the specific field that failed rather than the whole record — changes the answer materially. If review dropped to $0.15, model B's loud variant lands at $9,883 a month, still above the incumbent but close enough that a further reduction in the 2.8% failure rate would decide it.

That figure is the same five lines with one price changed. Reviews are 1.2 million documents at 2.8%, which is a little under thirty-four thousand of them; at $0.62 each that is the $20,832 on the table, and at $0.15 each it is around five thousand dollars. Substituting the second for the first and leaving the base calls, the repairs, the silent errors and the maintenance exactly where they are is what takes $25,675 down to $9,883. The reason one substitution moves a total that far is readable off the table before any arithmetic: the review line was more than four-fifths of it, and a term that large decides the sum almost on its own.

def effective_monthly_cost(volume, *, price_per_1k, repair_rate, repair_price_per_1k,
                           hard_fail_rate, review_cost, silent_rate, detect_rate,
                           silent_cost, maintenance):
    base    = volume / 1000 * price_per_1k
    repairs = volume * repair_rate / 1000 * repair_price_per_1k
    review  = volume * hard_fail_rate * review_cost
    silent  = volume * silent_rate * detect_rate * silent_cost
    return base + repairs + review + silent + maintenance

Every argument in that signature is a number somebody has to supply, and the ones people resist are silent_cost and detect_rate. They are estimates, they will be wrong by a factor of two, and a wrong-by-two estimate of a $23,000 line still dominates an exact $2,220 saving. Refusing to estimate them is a decision to treat them as zero.

Latency is a separate budget and it fails separately#

Cost and latency do not trade against each other cleanly, because latency usually has a hard budget rather than a price. Where the extraction sits inside a document-upload request with a 3.0-second p95 target, the repair round trip lands entirely in the tail.

Model B is faster on the median and blows the budget at p95, because 3.1% of documents take a second call and the second call's input includes the first response.

The mechanism there is worth separating from the figures. A repair is not a slower call. It is a second call, so a repairing request pays for the first response and then for the second, and the second is the dearer of the two because its input carries the first response along with the original document. That makes repairing requests a distinct population sitting above the rest, rather than a heavier tail on one distribution. Whether any given percentile moves is then a question about size: a percentile is a rank, and it shifts only if the repairing population reaches down as far as that rank. This is why a change that barely registers on a mean can move a tail figure by more than the entire budget, and why a repair rate and the percentile you are held to have to be quoted together. Either one on its own is unreadable.

A repair rate of 3.1% and a p95 target both being satisfied would require the repair rate to fall below about 2%, or the repair to move off the request path into an asynchronous queue, which is a product change rather than a prompt change since the user no longer sees extracted fields immediately.

Which input the answer is actually sensitive to#

An arithmetic that turns on eight estimates invites the objection that the estimates are soft, and the productive response is to find out which of them the decision depends on. Varying each input across a plausible range while holding the others fixed, and recording where the effective cost crosses the incumbent's $7,548, takes about twenty lines around the function above.

The price that motivated the migration is the input the answer is least sensitive to, which is the general shape of this exercise rather than a quirk of invoices: per-call prices differ by factors of two to five across models, and error-handling costs differ by factors of ten to a hundred depending on where the errors land. Knowing which rows matter also tells you which estimates deserve a day of work. Spending that day narrowing the silent-error rate from "somewhere between 0.2% and 1.0%" to a measured 0.4% ± 0.1 changes the decision; spending it negotiating the per-call price does not.

The first row is that whole argument in one line, and converting it is what makes it stick. The price range runs from $0.30 to $0.80 per thousand, and 1.2 million documents is twelve hundred thousands, so the base-call line can land anywhere between $360 and $960. What it is being asked to close is the gap between the patched variant's effective cost and the incumbent's, a little over $19,000 a month. Moving a line item by six hundred dollars against a gap of nineteen thousand is not a decision variable. It is a rounding adjustment, and it is the number every migration paper leads with.

The detection-rate row repays reading twice, because it moves the arithmetic in the direction nobody expects. Raising detection raises the priced line, since that line prices what surfaces and a better detector surfaces more. Nothing about the underlying harm has changed. The undetected wrong values are still in the ledger; they are free on this table only because nobody has spent anything on them yet. So an improvement in detection makes the migration look worse in an arithmetic that measures money rather than damage, and the version of this system with the lowest visible silent-error cost is the version that never finds out. Anyone using the table has to carry that distinction deliberately, because the arithmetic will not carry it for them.

The estimate worth buying rather than arguing about is the silent-error rate, and shadow traffic buys it. Running both models on every document for a week, serving the incumbent's output and logging model B's beside it, gives 280,000 paired outputs at a marginal cost of about $154 in model calls plus the storage. Fields where the two disagree are a superset of the errors, so sampling 300 disagreements and adjudicating them by hand — half a day — produces a silent-error rate measured on real traffic rather than assumed from a 400-case set.

Both of those numbers come off figures already on the page. A week is a bit under a quarter of a month, so a week of 1.2 million monthly documents is the 280,000. Dividing that by a thousand gives the number of thousands, and at model B's $0.55 per thousand the extra calls come to about $154. That is the cheapest measurement in this chapter by a wide margin, and what it buys is the softest input in the sensitivity table above.

The claim about disagreements needs its condition attached, because it is sound in one direction only. Two models that are both right agree, so every case where model B is wrong and the incumbent is right does show up as a disagreement. What the disagreement set never contains is the cases where both models are wrong the same way — an ambiguous layout, an OCR failure both inherit, a supplier convention neither was shown — which is precisely the class a shared weakness produces. So what the sample measures is B's excess error rate over A. That happens to be the quantity the silent-error row is asking for, and it is not the absolute silent-error rate of either model. Quoted as the latter, it will be quoted low.

Write the abandon criteria before you run the port#

The reason these arguments happen at review time is that nobody wrote down in advance what result would stop the migration, so every measured number arrives as something to be interpreted by people who have already spent three weeks on the work. Fixing the thresholds first converts that into a check.

# migrations/2024-11-invoice-extractor-model-b.yaml
motivation: "Reduce model spend on invoice extraction, currently $2,880/mo at 1.2M docs."
headline_saving_per_month: 2220

abandon_if:
  - metric: p95_latency_ms
    threshold: "> 3000"
    rationale: "Upload request budget. Async extraction would be a product change."
  - metric: schema_violation_rate_after_repair
    threshold: "> 0.5%"
    rationale: "Above this the review queue costs more than the model saving."
  - metric: silent_error_rate
    threshold: "> 0.20%"
    rationale: "Wrong values reach the ledger; detection is ~8% and correction is ~$37."
  - metric: effective_saving_fraction
    threshold: "< 35% of headline_saving_per_month"
    rationale: "Below this the migration does not pay for its own maintenance."
  - metric: refusal_rate_delta_on_policy_set
    threshold: "> +2 points"
    rationale: "Policy set from the first chapter; a refusal shift is customer-visible."

measured_on:
  set: evalsets/invoices-400.json@c41d09ab7712
  held_out_slice: 120
  repeats: 5
decision_owner: "AP platform lead"

Three properties make that file useful rather than ceremonial. Every threshold has a rationale that ties it to a cost or a commitment, so a request to relax one is a request to accept a named consequence. The metrics are the ones the harness already produces, so nothing has to be measured specially at decision time. And there is a named owner, because the failure mode of a criteria document with no owner is that a marginal result gets discussed until the team's investment in the work resolves it.

One of those thresholds is written as a fraction rather than as an amount, and that choice is deliberate. The headline saving moves: traffic grows, a provider reprices, the incumbent gets cheaper. A threshold denominated in dollars goes stale the first time any of that happens, and goes stale silently. A threshold expressed as a share of the headline saving asks the question that does not go stale — of the money this migration was proposed to save, how much survives contact with the failures it leaves behind. Where the share is set is a judgement about how much overhead a second production path is worth carrying, and it is a judgement that is easy to make in advance and close to impossible to make in a room with three weeks of work sitting on the table.

Committing in advance also protects the migration in the other direction. A port that clears every threshold ships without a fresh round of doubt, and the engineer who did the work does not have to relitigate it against whoever is most risk-averse in the room.

Workloads whose errors are cheap#

These numbers are specific to one workload's error economics, and the same comparison on a different product inverts cleanly. A model that drafts marketing copy has no ledger, no silent-error cost worth the name, and a review step that was already there; on that workload the headline saving survives nearly intact and the migration is straightforwardly worth doing. The method transfers; the conclusion does not.

Nor does this price the reason many migrations happen, which is not cost at all. Removing a single-vendor dependency, meeting a data-residency requirement, or being able to fail over when a provider has an outage are all worth paying for, and against those the arithmetic changes role — it tells you what the second model is costing you, not whether to have one.

That is the situation most teams end up in: two models in production, either as a migration in progress or as a permanent second path. It carries the $600-a-month line from the table above, which is the least examined number in this chapter and the one that grows. A second variant of one prompt is manageable. What happens next is a third model, a policy change applied to two variants and missed on the third, and three weeks before anyone notices.