Routing requests into a critique pass

A pass applied to every request pays for itself on the requests that needed it and pays nothing back on the rest. The share that needed it is usually small: a single-pass generation is wrong some percentage of the time, and the subset of those errors a critique round can actually repair is smaller still, so a pass running on all traffic is buying a few thousand corrections with a few hundred thousand extra calls.

The worked example here is a pricing-explanation feature that answers "why is my bill $412 this month" for a subscription product, serving about 420,000 requests a month. Single-pass, each request runs roughly 900 input and 380 output tokens, which at $3 and $15 per million tokens is $0.0084 a request and $3,530 a month. The critique pass, gated and capped, was a genuine quality win on a properly run paired eval, and it took the bill to $7,410 and p95 from 1.9 to 3.8 seconds.

That per-request figure is worth assembling rather than accepting, because its shape decides where a saving can come from at all. Input is billed at $3 per million tokens, so 900 input tokens cost nine hundred millionths of three dollars, which is a little over a quarter of a cent. Output is billed at $15 per million — five times higher — so 380 output tokens cost a little under six-tenths of a cent, despite there being fewer than half as many of them. The two add to the $0.0084, and multiplying that by 420,000 gives the $3,530 a month. The reading that matters is that most of what a request costs is the text the model writes rather than the text it is given, which is why a second pass that rewrites the whole answer roughly doubles the bill while a check that merely reads the answer costs nothing at all.

The trigger has to predict fixable errors, not errors#

A paired eval already contains the label needed to build the router. For every case, you know whether the single pass was right and whether the critique version was right, and the cell that matters is the one where the single pass was wrong and the critique version was right. Those are the critique-fixable cases, and they are the only requests worth routing into the pass.

That distinction is what stops the trigger from being designed against the wrong target. On this feature the single pass was wrong 9.1% of the time, and only 3.6 percentage points of that was critique-fixable — the rest were numerical errors the critique pass left wrong, plus a handful the pass converted from wrong to differently wrong. A trigger tuned to detect "errors" would spend most of its routing budget sending the numerical failures into a pass that cannot repair them.

Building the trigger against errors rather than against fixable errors is tempting for a concrete reason. Errors are the thing that has labels, the thing that generates tickets, and the thing anybody would name if asked what the router is for. It is wrong because the router is not choosing whether an error exists. It is choosing whether to spend a critique pass, and a critique pass has a different repair rate on every error class — routing a numerical error into it buys nothing at all, since the pass hands back the same wrong figure at twice the cost. The quantity being traded against spend is repairs, and repairs and errors are different populations that only partly overlap.

Where in the request the routing decision sits is the other thing to be clear about, and it is visible in the diagram below. Four of the five triggers read the generated answer, which means the single pass has already run by the time any routing happens. The router is therefore not saving a generation; it is saving the critique pass and the revision that follows it, which is the expensive half. Only complex_billing_event can be evaluated before generation, because it reads the invoice rather than the answer — and that is exactly why it catches cases where nothing about the output looks wrong.

The triggers themselves are unremarkable code, which is the point — each one runs in well under a millisecond against the generated answer and the request context.

import re
from decimal import Decimal

MONEY = re.compile(r"\$\s?([\d,]+\.\d{2})")

def triggers(answer: str, invoice: "Invoice") -> set[str]:
    flags: set[str] = set()

    quoted = {Decimal(m.replace(",", "")) for m in MONEY.findall(answer)}
    known = invoice.all_figures()          # every number the pricing engine produced
    if not quoted <= known:
        flags.add("unreconciled_figure")

    if len(answer.split()) > 220 or answer.count("\n\n") > 6:
        flags.add("over_length")
    if "|" in answer and not re.search(r"^\|.*\|$", answer, re.M):
        flags.add("broken_table")
    if invoice.has_proration or invoice.plan_changed_mid_cycle:
        flags.add("complex_billing_event")
    if not re.match(r"^(Your|This month'?s)\b", answer):
        flags.add("opening_convention")

    return flags

unreconciled_figure is the strongest single line in that function. Every dollar amount in the explanation must be a figure the pricing engine actually computed for this invoice, and set containment — asking whether every quoted figure is a member of the set the engine produced — answers that exactly. It fires on 1.2% of traffic, and those requests do not go to the critique pass at all — they go to a repair step that substitutes the engine's figures, because a number the model invented is not a thing a critique round can be expected to correct.

Set containment is the right operator there because of what the two sides are. The engine's set is every figure the engine computed for this invoice, which makes it the complete list of numbers an honest explanation could quote. Asking whether the quoted set sits inside it therefore asks exactly one question — did the answer quote anything the engine never produced — and a subset test answers that in a single pass with no threshold to tune and no way to be partly right.

The same precision states what the test does not do, and each limit is readable in the code rather than a matter of opinion. Containment runs in one direction only, so an explanation that omits a figure it should have mentioned passes. Containment is also blind to roles, so an explanation quoting a genuine engine figure in the wrong place — the tax presented as the total — passes, because both sides of the comparison are then real engine output. And MONEY recognises only dollar amounts written with cents, so a figure written without them is invisible to the trigger entirely. None of the three is a bug, and all three belong written down beside the trigger's flag rate, because an unstated boundary is how a check ends up credited with work it never did.

Why a flagged request goes to a repair step rather than into the critique pass follows in three steps, and each is checkable. The model produced a figure absent from the engine's set, which means it did not read that figure from anything it was given. A critique round runs the same model over the same context, so it has no source for the correct figure that the first pass did not already have. The engine holds the figure and took no part in the error. Substituting the engine's value is therefore right by construction, while asking a critique round for it is asking the same process to supply information it does not possess.

complex_billing_event is a context trigger rather than an output trigger, and it fires before the answer is even inspected: prorations — the partial-period charges that appear when a subscription changes partway through a billing cycle — and mid-cycle plan changes are where the explanations go wrong, and routing on the shape of the input catches the cases where the output looks fine and is not.

What the routing buys, measured#

Four configurations against the same frozen eval and the same month of traffic volume:

The last row keeps 91% of the available gain for 14% of the incremental spend, and p95 across all traffic barely moves because 86% of requests never enter the pass. The requests that do enter it are slower — their own p95 is 4.1 seconds — and they are the requests where a slow correct answer is worth more than a fast wrong one.

Both of those fractions are readable straight off the table, and reading them out is what makes them portable. The available gain is the distance between the first two rows, 90.9% to 94.2% — a little over three points, and that is the whole of what the critique pass has to sell. The last row reaches 93.9%, which is three of those points, and three parts of a little over three is the 91%. The row above it reaches 93.6%, a smaller share by the same subtraction, and that share is the 82% sitting in its own fixable-errors column. The two columns agreeing is worth noticing rather than passing over: accuracy gain and fixable errors caught are the same quantity measured from opposite ends, and a table where they disagreed would have something wrong in it.

The spend fraction is the same operation on the money column. The routed configuration costs $4,080 against a single-pass $3,530, a difference of a little over five hundred dollars a month, while critique-on-everything costs $7,410 against the same baseline, a difference of just under four thousand. The first difference is about a seventh of the second, which is the 14%. Both fractions are ratios of differences, not ratios of totals, and that is the step most easily skipped: setting $4,080 beside $7,410 directly suggests the routed configuration costs a little over half as much, when what it actually does is spend about a seventh as much extra.

The p95 column needs a different reading, because a percentile over a mixed population is a position in a sorted list rather than an average. With 86% of requests fast and the rest slow, the ninety-fifth position from the bottom falls just inside the slow group rather than deep in its tail, which is why the all-traffic p95 lands near the fast group's neighbourhood and not near the routed group's own 4.1 seconds. The consequence is that the all-traffic p95 is close to useless as an alarm on the routed path, and the routed path's own tail is the number that has to be watched.

Choosing where to sit on the cost-against-gain curve those four rows trace out is a calculation rather than a judgement call, once two numbers exist. The incremental cost of routing one more percent of traffic into the pass is $37 a month at this volume and token count, and the value of a caught error is whatever a support contact costs — $9 here, from the support team's own figures, against an error rate that generates a ticket about 30% of the time. A trigger that routes one more percent of traffic and catches one more percentage point of the 3.6% fixable population is worth about $45 a month against $37, which is marginal; one that routes three percent to catch the same point is not worth building. Plotting each candidate trigger as a point on that plane sorts them in a way that survives a conversation with finance, and it also shows when the whole exercise has run out of headroom, which on this feature happens at around 20% of traffic routed.

The structure of that comparison transfers to another feature more reliably than any of its figures do. Both sides have to end up in the same unit, dollars a month, and each side is a product. The cost side is the number of requests routed multiplied by the incremental cost of routing one, and that incremental cost is the gap between the two bills spread over a month of traffic. The value side is a chain of three: the number of fixable errors a trigger newly catches, times the share of errors that become a support contact, times what a contact costs. Five measurable quantities and no preferences, which is why the exercise sorts candidate triggers instead of ranking opinions about them. It is also why the exercise terminates: as the remaining fixable errors thin out, each new trigger catches fewer of them for the same traffic, so the value side stops growing while the cost side carries on.

Two thresholds in that trigger function were set this way. The length flag at 220 words was chosen because the fixable-error rate above that length is 14% against 2.4% below it, and moving it to 180 words would route another 4.1% of traffic to catch another 0.3 points. The proration flag has no threshold to tune, since it is a property of the invoice, and it is the highest-yield trigger in the set precisely because it is not a heuristic about the output at all.

The length threshold shows what actually makes a trigger worth its traffic, which is enrichment rather than accuracy. Above 220 words the fixable-error rate is 14%; below it, 2.4%. The flagged population is therefore something close to six times denser in repairable errors than the unflagged one, and that density is the entire product being bought, because a trigger pays a fixed price per request routed and collects a return only on the repairable errors among them. Moving the threshold down routes more traffic at a lower density, which is why 180 words costs 4.1% of traffic to gain 0.3 points. The limiting case makes the principle plain: a trigger firing on a population no denser in fixable errors than the average is worth nothing whatever, however many errors it flags, because it is buying the same errors at the same price as routing at random.

The 9% of fixable errors the router misses are a real cost and they should be stated as such rather than buried. They are cases where the explanation was subtly incomplete, with no formatting problem, no unreconciled figure and an ordinary billing event, and no cheap signal distinguished them from the 86% that were fine. Buying them back means either widening a trigger, which trades cost linearly for a few points of recall, or finding a new cheap signal. The trade is explicit and it is a product decision rather than a technical one.

Shadow sampling is what keeps the number honest#

A router creates a population you can no longer see. Once 86% of traffic bypasses the pass, nothing in production tells you whether the trigger's recall has drifted, and it will drift: the model version changes, the pricing engine adds a plan type, the prompt is edited, and the error distribution moves underneath a trigger set that was fitted to last quarter.

The cheap fix is to route a small random sample of unflagged traffic through the pass anyway and compare.

import random

def handle(request) -> str:
    answer = generate(request)
    flags = triggers(answer, request.invoice)

    if flags:
        return repair_or_critique(answer, flags, request)

    if random.random() < 0.02:              # shadow sample
        shadow = critique_and_gate(answer, request)
        if shadow != answer:
            log_shadow_disagreement(request.id, answer, shadow, flags=flags)

    return answer                            # user always gets the unflagged answer

Two percent of 420,000 requests is 8,400 shadow runs a month, costing about $70, and it produces a stream of cases where the critique pass wanted to change an answer the triggers cleared. Those cases are the raw material for the next trigger. On this feature the first month of shadow logs surfaced a cluster around annual-to-monthly downgrades, which became a sixth trigger and lifted fixable-error recall from 91% to 95% at 0.6 points of additional traffic.

The arithmetic behind that cost is worth doing once, so that it stops sounding like a concession. Two percent of 420,000 is 8,400 requests, each costing what a critique pass costs, and that is where the $70 comes from — well under two per cent of the routed configuration's monthly bill. The reason to run it permanently is not that first month's yield, though. It is that a router with no shadow sample generates no record of what it stopped catching, and unlike almost everything else on a dashboard, that record cannot be reconstructed later from data somebody kept for another reason.

The shadow path returns the original answer to the user regardless of what the critique produced, which keeps the sample unbiased and keeps the latency budget intact.

Both halves of that sentence rest on a property worth stating separately. Unbiasedness comes from the sample being drawn at random from the unflagged population: a sample selected on any signal at all — the longest unflagged answers, the ones from a particular plan type — would estimate the miss rate on that slice rather than on the population the router has stopped looking at, and the population nobody is watching is the whole thing being measured. Discarding the shadow answer is what keeps the sampled requests members of that population at all. If the revision were returned, those requests would have been treated, and the disagreement rate would then describe a path users were on rather than the path being estimated.

Routing also introduces failure modes that critique-on-everything does not have, and they are worth handling before they are discovered. The flagged 14% of requests are the slow ones, so any timeout tuned against the old p95 will now fire on exactly the requests that most needed the extra work, and the timeout budget has to be set per path rather than globally. The repair and critique branches need a fallback that returns the unrevised answer when the second call fails or times out, since a flagged answer with a formatting problem is still a better outcome than an error page. And traffic that is bursty in billing-cycle shape will concentrate the complex_billing_event flag around the first few days of the month, so the capacity headroom for the critique path is not 14% of average load, it is closer to 14% of peak, which on this feature was a factor of three higher.

One feature, two error classes, two different answers#

The framing product brought to this feature was that a self-critique pass lets the model catch its own mistakes before the customer sees them. That framing should be declined, and this feature's own eval shows why: at 2.1x cost, the pass repaired 40% of the formatting violations and 4% of the numerical errors, and those two results are different kinds of evidence rather than two sizes of the same result.

Formatting violations are rule application against determinate rules that live in the prompt — the opening convention, the table structure, the length ceiling. A model checking an answer against a stated rule is doing a lookup, which is the regime where model critique performs. Even here the pass is not the best available tool, since a deterministic formatter repairs every broken table and every opening-convention violation at zero marginal cost, and the residue worth sending to a model is the prose problems that no rule expresses.

Numerical errors have a source of truth sitting in the same request. The pricing engine computed every figure on the bill, so an explanation quoting a number the engine did not produce is wrong by a test that takes microseconds and catches every instance. Four percent is within the noise of most eval sets, and it is 4% of the errors that generate the support tickets. Accepting the "catches its own mistakes" framing is what stops a team building the containment check, because the framing implies a general capability and therefore implies that no specific check is needed.

The framing is worth dwelling on because of the sentence a reader would confidently write after accepting it. "The model catches its own mistakes" is a claim about a capability, and a capability is the sort of thing that either exists or does not — so a team holding it stops asking which mistakes, and stops building checks for the classes the pass never covered. What this eval actually supports is narrower and considerably more useful: this model, on this prompt, repairs a determinate rule violation much of the time and repairs an arithmetic error almost never. That sentence names a class and a rate, which is something a design can rest on, and the general sentence names neither. The same discipline applies to the containment check that replaces it. It catches every figure the engine did not produce, and it says nothing at all about a genuine engine figure quoted in the wrong place, which is a smaller claim than "catches every instance" and the only one the code supports.

Trigger thresholds are configuration, not architecture#

Once a router is live, three numbers belong on the dashboard next to the usual latency and error-rate panels, because each one decays quietly. The share of traffic each trigger flags tells you when a trigger has started firing on everything, which is how a router turns back into a critique-on-everything pipeline without anyone deciding to. The gate's rejection rate on revisions tells you when the critique round has stopped earning its tokens for the requests that reach it. And the shadow-sample disagreement rate is the standing estimate of what the router is missing, which is the only one of the three that cannot be reconstructed after the fact if you did not collect it.

Re-run the paired eval on a model upgrade before the upgrade ships, not after. Everything in this topic — the recall of a critic prompt, the round where a loop peaks, the fraction of errors a trigger catches — is a measurement of a specific model against a specific prompt and a specific corpus, and a version bump invalidates all of it at once. Teams that treat the routing thresholds as configuration and re-fit them each quarter keep the shape of the win; teams that treat them as architecture find that the pass they carefully restricted to 14% of traffic is now running on 40% of it and fixing something else entirely.

So the last question to ask about a critique pass is not how good it is but how many requests should see it. A measured win applied uniformly is a cost multiplier applied uniformly, and the fixable subset is where the value lives: a few lines of deterministic code in front of the pass moved this feature from $7,410 to $4,080 a month while keeping 91% of the gain, and the same reasoning decides which error classes belong in the pass at all. Send only the flagged requests through it, and the bill tracks the flags rather than the traffic. What makes the arrangement hold is the 2% that keeps going through anyway, because a router is a claim about the traffic it is not looking at, and that claim expires.