Verifiers and retry loops

The pattern rests on an asymmetry: for some tasks, checking a candidate answer is far cheaper than producing one. A SQL query either parses or it does not; the warehouse knows its own schema; a query planner will say whether a join is a cartesian product. That is the same asymmetry which makes a Sudoku solution easy to check and hard to find, and where it holds, an expensive model is doing work you can buy more cheaply. Where it does not hold, the pattern fails quietly while looking as though it works.

What makes the trade negotiable is that the asymmetry has a cause you can inspect rather than a name you have to take on trust. Producing a query means searching a space of possible queries for one that answers the question, and there is no enumeration of that space and no oracle over it — which is exactly what the expensive model is being paid for. Checking a query means evaluating one candidate against rules that already exist and are already authoritative somewhere else. The warehouse holds its own catalogue. The planner computes a plan for any query handed to it. Neither of those was built for this purpose and neither has to be inferred, so the check is a lookup against systems that were going to answer anyway, which is why it costs milliseconds and no tokens.

The recurring mistake is to count only the generator's cost, so the verifier's own inference bill, its false rejections and the retries it triggers all appear in the arithmetic here rather than as footnotes.

The arithmetic of a retry loop#

A deterministic verifier for this task is a function of the schema rather than of a model. It parses the query, resolves every referenced table and column against the catalogue, runs EXPLAIN to obtain a plan, and asserts a handful of properties: no cross join without an explicit predicate, no reference to a column that does not exist, a LIMIT present on any query without an aggregate, and no scan estimated above a row threshold. That costs about 4 milliseconds of database time and nothing measurable in tokens.

def verify(sql: str, catalogue: Catalogue) -> Verdict:
    try:
        ast = sqlglot.parse_one(sql, dialect="postgres")
    except ParseError as e:
        return Verdict(False, f"parse error: {e}")
    for col in referenced_columns(ast):
        if col not in catalogue:
            return Verdict(False, f"unknown column {col}; available: {catalogue.near(col)}")
    plan = db.explain(sql)
    if plan.has_cross_join():
        return Verdict(False, "cross join between " + plan.cross_join_tables())
    if plan.estimated_rows > 5_000_000 and not ast.find(exp.Limit):
        return Verdict(False, "unbounded scan; add a LIMIT or a filter")
    return Verdict(True, None)

def answer(question: str, max_attempts: int = 3) -> Result:
    feedback = None
    for attempt in range(max_attempts):
        sql = cheap_model(question, feedback)          # $0.0041, ~1.0 s
        verdict = verify(sql, CATALOGUE)               # ~4 ms, ~$0
        if verdict.ok:
            return Result(sql, source="cheap", attempts=attempt + 1)
        feedback = verdict.reason
    return Result(reasoning_model(question), source="escalated", attempts=max_attempts)

Measured against a month of real questions, that verifier catches 68% of the cheap model's errors and falsely rejects 0.4% of its correct answers. A retry carrying the verifier's message back into the prompt succeeds 71% of the time on the first attempt and 40% of what remains on the second. Working it through: of the 18.8% of answers that are wrong, 12.8 points are caught, of which 9.1 are fixed on the first retry and a further 1.5 on the second, leaving about 2.2 points still wrong after two retries and 6.0 points the verifier never saw. Escalating everything still rejected at the end of the loop to the reasoning model recovers most of the first group, and the design lands at 91.6% against the reasoning model's 92.4%.

Cost is where it pays. The mean number of cheap calls per question is about 1.17, because 13.1% of first attempts are rejected and a smaller fraction of retries are rejected again, which is $0.0048. Roughly 3% of questions escalate, adding $0.0014. Total is $0.0062 a call.

The latency column deserves reading as a distribution rather than a single figure, because a cascade — the arrangement in the code above, where the cheap model runs first and the expensive one is reached only by requests that keep failing the check — has a shape the reasoning model does not. Its p50 is 1.3 seconds, identical to the bare cheap model, since 87% of requests pass verification on the first attempt and never enter the loop. Its p95 is 4.2 seconds, three attempts plus verification. Its p99 is around 16 seconds, because that tail is the escalation path, and the escalated request carries every property from the previous chapter: it is a reasoning call with a thinking budget, sitting behind a client deadline that has already had three seconds spent against it. Sizing that budget against the remaining time rather than against the original deadline is the difference between an escalation that works and one that reliably times out.

That is the previous chapter's division applied at a point where the numerator has already shrunk. The deadline an escalated call runs against is not the deadline the request arrived with; it is whatever survives three cheap attempts and three rounds of verification. Multiply the time remaining by the measured decode rate and you have the token allowance the escalation can actually spend, which is smaller than the allowance somebody would configure for a first-attempt reasoning call. Configuring it as though the two were the same is how a cascade acquires a failure mode that shows up only on the requests that needed the escalation most.

Generate-and-check is not a new idea and it is worth knowing where it sits. Sampling several candidates and selecting among them with a scorer is one of the oldest test-time compute techniques, predating trained-in reasoning by several years, and A Survey of Test-Time Compute maps the family: best-of-n, where several answers are sampled and a reward model — a model trained to score candidate answers rather than to write them — picks the winner; self-consistency by majority vote over sampled answers; tree search over partial solutions; and the verifier-guided loop used here. Extended thinking internalises a version of the same thing, which is why the two approaches compete rather than compose neatly — you are choosing between paying a model to check itself in tokens you cannot inspect, and checking it yourself in code you can.

What separates that whole family from extended thinking is where the selection happens. Best-of-n, majority voting and a verifier loop all produce candidate answers and then choose between them using something outside the model that produced them. Extended thinking folds the same idea inward, so the checking happens in tokens, inside one generation, judged by the same weights that are being checked. That is why the two compete rather than stack, and it is also why a deterministic verifier can be so much cheaper than either. It is not a smaller model doing the same job. It is a different kind of thing doing a job no model has to do.

Replacing the deterministic verifier with a model-graded one — a mid-tier model given the question, the schema and the candidate query, asked whether the query answers the question — and re-running the same arithmetic gives four designs to compare:

The model-graded row is worse on every axis than the deterministic one despite catching slightly more real errors — 74% against 68% — for three reasons that compound. Its own inference costs $0.019, four and a half times the generator it is checking, and that charge lands on every call including the 81% that were right the first time. It adds 2.1 seconds of serial latency per attempt, since verification cannot begin until generation ends. And it falsely rejects 6% of correct answers, which drives the rejection rate to 18.8% and sometimes replaces a correct query with an incorrect one after the retry, which is how a design with a better detection rate ends up three and a half points less accurate.

Checking what is convenient rather than what is wrong#

Every number above depends on a detection rate of 68%, and that figure is a property of this error distribution rather than of the verifier. Pull the errors apart the way an extraction service's field errors get pulled apart: 41% of the cheap model's wrong queries reference a column that does not exist, usually a plausible-sounding one from a similar table; 27% join two tables on the wrong key, producing a fan-out the planner reports; 18% misread the question's intent and produce a query that is valid, efficient, and answers something else; 14% get the aggregation grain wrong, summing a pre-aggregated column.

The verifier covers the first two classes almost completely and the fourth partially, which is where 68% comes from. It covers the third class not at all, because a query that answers the wrong question is a perfectly good query. If the workload had a different mix — mostly ad-hoc analyst questions against three well-known tables, where schema errors are rare and intent errors dominate — the same verifier would catch perhaps 12% of errors, the retry loop would fire almost never, and the design would deliver 83.5% accuracy for $0.0045. That is not a bad outcome, it is not remotely a substitute for the reasoning model, and nothing about the verifier's code would look different.

The failure this produces in practice is a verifier built from what is easy to assert. Schema validation, JSON well-formedness, a required-fields check, a regex on a date — these take an afternoon and they check the classes of error a competent model rarely makes. A team ships one, sees a rejection rate of 1.2%, concludes the model is accurate, and never learns that its real error rate is 19% and the verifier is blind to almost all of it.

Two other quantities belong in the comparison and are routinely left out. The verifier's false-reject rate multiplies against the majority of traffic — a 6% rate on a model that is right 81% of the time generates 4.9 points of unnecessary retries — so a verifier that is slightly too strict costs more than one that is slightly too lenient.

That asymmetry is a base-rate effect and is worth working through, because two rates that read as comparable are being applied to populations of very different sizes. A detection rate acts on the wrong answers, and the wrong answers are the minority. A false-reject rate acts on the right answers, and the right answers are the large majority. So a false-reject rate several times smaller than a detection rate can still produce more unnecessary retries than the detection rate produces caught errors, purely because it is multiplied by a much larger number. Any comparison between two verifiers that quotes detection rates and omits false rejections is comparing them on the smaller of the two populations and calling it the answer.

The second of those two quantities is the retry loop's own success rate, which has to be measured rather than assumed: feeding an error message back to the same model that just produced the error works well when the message is specific ("unknown column orders.customer_email, did you mean orders.customer_id?") and barely at all when it is generic ("query is invalid"). The quality of the verifier's message is a lever on the whole design's accuracy, and it costs nothing.

A message can be a lever at all because it is the only thing that changes between one attempt and the next. Same weights, same question, same schema — the second call differs from the first in the text appended to it and in nothing else. A message naming a column that does exist puts that column's name into the context, where it is available to be copied, so the next attempt is scored from a sequence that contains the answer to what went wrong. A message saying the query is invalid adds no token the model did not already have, so the next attempt is scored from a context that is materially the first one and produces materially the first answer. The same reasoning accounts for the non-converging loops described further down: a systematic misunderstanding is one that the feedback never contradicts, so nothing the loop adds changes what gets sampled.

Where the asymmetry is not there#

The gap between generating and checking is real for SQL, for code with tests, for structured extraction against a schema, for arithmetic, and for any output with an executable or decidable property. It is weak or absent for the tasks people most want it for. There is no cheap verifier for whether a summary is faithful, whether a plan is sensible, whether an explanation is correct, or whether a piece of prose is good, because deciding those questions is the same problem as producing the answer, and the only available checker is another model of comparable capability at comparable cost. When someone proposes a verifier for an open-ended task, the question to ask is what the check executes; if the answer is "another model reads it", the economics of the model-graded row apply and the savings usually are not there.

Underneath that question is a sharper one: whether the check has an authority outside the model to appeal to. A parser has a grammar. A catalogue has a list of columns. A test suite has assertions somebody wrote down. Each of those settles the matter with no judgement involved, which is simultaneously what makes it cheap and what makes it trustworthy. Whether a summary is faithful, whether a plan is sensible, whether an explanation is correct — none of those has an authority sitting anywhere to appeal to, so the only available checker is something that forms judgements, and something that forms judgements about an open-ended answer is doing the same work as something that writes one. Cost follows the work rather than the label on the component.

There is also a class of task where verification is possible but the retry loop does not converge, because the model's error is systematic rather than stochastic. If the cheap model consistently misunderstands one warehouse concept — treating a slowly-changing dimension as if it had one row per entity — then every retry produces the same shape of wrong answer with different syntax, and the loop burns three attempts before escalating. Watching the attempt-count distribution catches this: a healthy loop has most rejections resolved on attempt two, and a distribution with a heavy tail at max_attempts means the retries are not doing work and escalating after one would be cheaper.

A verifier is worth exactly its overlap with the errors you actually get, which is why the sequence is to classify failures first and write assertions second, and why the headline number to report is a detection rate against a labelled sample rather than a count of checks. Where the overlap is large and the check executes in milliseconds, the cascade wins on cost by nearly an order of magnitude and on median latency outright, while giving up under a point of accuracy. Where the overlap is small, the same code ships, the rejection rate looks reassuringly low, and the design saves money by not catching anything.

Framing the comparison as one choice for a whole workload is what makes the 91.6% figure slightly unsatisfying: it averages a design that is excellent for schema-heavy questions with one that is mediocre for intent-heavy ones. The verifier is also producing information the framing throws away. When it rejects a query twice, that is a statement about this specific request being harder than usual, and it arrives while the request is still in flight and the expensive model is still an option.