The generate-verify asymmetry

A verifier is an independent check that takes one candidate output and decides, quickly and without redoing the work, whether to accept it or reject it. In practice it is something you already have: a JSON schema wrapped around a structured-output call, a compiler or type checker over generated code, a test suite in continuous integration, a unique index that refuses a bad row, a scratch database carrying the real constraints, or a re-retrieval of the document a model claimed to be quoting. Verifier-first design is the practice of picking that check before committing to the problem, pairing the generator with it, and regenerating whenever it says no.

The word carrying the weight in that definition is independent, and it is worth pinning down before it is used forty times, because the obvious reading is not the useful one. Independent does not mean "a separate piece of software" — a second call to the same model, with the same prompt, asking whether the first answer looks right, is separate software and is not independent in the sense that matters. What makes a check independent is that it decides using something the generator did not have or was not optimising for. The unique index knows which values are already in the table, and the model does not. The foreign key knows which rows exist. The compiler applies a rule set that the generator was never scored against. The re-retrieved document is the source text itself rather than a recollection of it. In every case the checker brings information from outside the generator's own process, which is precisely why it can disagree. A check drawing on the same information as the generator will tend to be wrong in the same places, and a check that is wrong where the generator is wrong catches nothing while reporting that everything passed.

Verifier-first design exists because a language model is a stochastic generator, meaning the same request can produce a different answer each time and some fraction of those answers are wrong in ways the model itself reports no doubt about. Prompt work moves that fraction and does not remove it, so the question that decides whether a feature is buildable is what will catch the remainder, and at what price. The price is the part that gets skipped, and it is the part that decides.

The asymmetry is the component, and it has a price ratio#

The thing that makes a stochastic generator usable as a dependable part of a system is that producing an answer and recognising a bad one are different amounts of work. This is not a new idea and it is not specific to language models. Factoring a 600-digit number is hard and multiplying the two factors back together to check is trivial. Finding a proof is hard and running a proof checker over it is mechanical. Writing the feature takes a week and running the test suite takes ninety seconds. Every one of those pairs is usable precisely because the second half is cheap enough to run on every candidate, including the bad ones, including the ones you throw away.

Take an agent that turns a plain-English description of a schema change into a database migration, and put numbers on it, because the ratio is what matters rather than the absolute cost. A scratch copy of the production schema gets restored from last night's structure-only dump, the candidate migration is applied to it, and then the foreign keys, the check constraints and the repository's own test suite decide. One generation attempt costs about 4,100 input-plus-output tokens; at the rate this team pays, that is around $0.012, and it takes 2.1 seconds of wall-clock. One check costs 3.4 seconds of CPU on a container that is already running and $0.000 in tokens, which after the amortised cost of the scratch database works out to something on the order of $0.0002. The check is roughly sixty times cheaper than the thing it is checking. That ratio is what lets you run the check on every single candidate, and what lets you throw away four candidates in a row without the economics collapsing. When the check fails it fails loudly, with a Postgres error string that names the constraint, and that string goes back into the next generation attempt.

The ratio is one division, and it is worth performing rather than accepting. Twelve thousandths of a dollar to generate, two ten-thousandths to check: divide the first by the second and the check costs about a sixtieth of what it is checking. The phrase "the economics collapsing" then has an arithmetic meaning. Five attempts cost five generations and five checks. The five generations are five times the price of one, which is what anybody expects. The five checks together come to under a fiftieth of a single generation, so they never appear in the bill at all. What a large asymmetry buys is not cheap retries — retries cost whatever generation costs, however good the checker is — but the freedom to check every candidate, including the four that were discarded, without the checking becoming a line item.

Two of those figures are in different units and reading them as one is a mistake worth heading off. The check costs less money and takes more time: 3.4 seconds of CPU against 2.1 seconds of wall-clock for the generation. It is cheaper because CPU on a container that is already running is nearly free and tokens are not, and it is slower because applying a migration and running a test suite is real work. So a sixty-to-one price advantage carries no promise about latency, and a loop of five attempts spends the best part of half a minute regardless of how little it costs. Price and latency have to be budgeted separately, and a design that is comfortable on one can be impossible on the other.

Where the asymmetry is large, engineering has been exploiting it for decades under other names. Compilers and type checkers exist because a machine can reject a large class of wrong programs far faster than a person can write a right one. Property-based testing exists because generating a thousand random inputs and checking an invariant on each is cheaper than reasoning about the input space. Proof-carrying code, in the late nineties, made the asymmetry the interface itself: an untrusted producer ships the artefact together with a certificate, and the consumer runs a small, auditable checker instead of trusting the producer. A language model is an untrusted producer with an unusually good hit rate, and the design question is the same one that literature asked — what certificate can it be made to emit, and what can check that certificate without redoing the work.

Now flip it. Suppose the artefact is a support macro rewritten to match a new tone guide, and the only available check is a senior support lead reading the original and the rewrite side by side, which takes about ninety seconds and, loaded, costs somewhere near $1.50. Generating the rewrite costs $0.004 and takes 1.3 seconds. The check is now roughly four hundred times more expensive than the generation, and there is no arrangement of retries that fixes this, because every retry needs its own check. A loop where checking costs more than generating is a loop that turns your model spend into human review spend at an exchange rate you did not choose.

Run the same two divisions on those figures and the sentence stops being a warning and becomes a measurement. A dollar and a half divided by four thousandths of a dollar is a bit under four hundred, so the check is four hundred times the generation rather than a sixtieth of it. The $1.50 itself is worth unpacking, because it is the number a reader should replace with their own: ninety seconds is a fortieth of an hour, so a loaded cost near a dollar fifty for that ninety seconds implies a fully loaded rate around sixty dollars an hour for a senior support lead. Substitute a different rate and the ratio moves proportionally, and it moves nowhere near far enough to matter.

Now put the two arrangements side by side, because the comparison is the point of the section. Five attempts on the migration agent cost five generations and a rounding error. Five attempts on the macro rewriter cost a bit over seven and a half dollars, of which more than ninety-nine hundredths is the support lead's time and effectively none is the model. The loop has not made the model expensive. It has made a person's afternoon the unit of consumption, and it scales with the number of attempts rather than with the number of accepted outputs — so the worse the generator performs, the more of that person's day it takes. That is the exchange rate nobody chose. The ratio between the two arrangements is sixty on one side and a four-hundredth on the other, which is a gap of something over four orders of magnitude, and it is the reason the closing claim of this chapter about prompt work is arithmetic rather than pessimism.

Choosing the check before choosing the problem#

The repair loops from validation and retry-on-invalid arrive at the end of a project: the feature exists, the JSON comes back malformed some of the time, and a validator plus a retry gets wrapped around it. Verifier-first design moves that decision to the front. Before agreeing to build something, you name the artefact the model will produce, name the thing that will reject a bad one, and price both. If you cannot name the rejecter, the answer is not "build it and figure out validation later" — the answer is that you have a research project rather than a feature, and it should be scoped and staffed as one.

That reordering turns verifiability into a selection criterion. Two proposals that look equally valuable to a product manager can be a week apart in engineering risk, and the distance is almost entirely in what can say no. The following is the shape of the conversation, run over five candidates from a real backlog:

Notice what fills the fourth column. In every row the check covers the mechanical properties of the artefact and stops at the boundary of intent, which is where the interesting failures live. That is normal and it is not a reason to abandon the check; it is a reason to write the column down, because the gap between what the check covers and what the feature promises is the residual risk the team is agreeing to carry. On the migration row that residual is small, since a wrong-but-valid migration is caught in review by a human reading eight lines of SQL. On the analytics row it is large, since a wrong-but-valid query returns a plausible number that lands in a slide with no reviewer between it and a decision. Two features with equally strong checks can carry wildly different residuals, and the residual is what determines whether the check is enough.

The table is doing something a ranked feature list cannot. It is putting the check in the same row as the work, so that the decision to build is made against the check that exists rather than against the one somebody hopes to invent during implementation. Two of these five are declined at the planning stage, which is the outcome the whole exercise is for. The third is reshaped: an agent that writes analytics SQL is a different, smaller, safer product if it is only allowed to answer questions whose results can be cross-checked against a materialised metric.

The loop is the plumbing, not the guarantee#

It is easy to read all of this as a claim about retries, and it is not. The retry is the cheapest thing in the design; you can write it in fifteen lines and you have already written it several times.

from dataclasses import dataclass

@dataclass
class Verdict:
    accepted: bool
    reason: str | None  # fed back verbatim on the next attempt

def generate_verified(task: str, generate, verify, max_attempts: int = 3):
    feedback = None
    for attempt in range(1, max_attempts + 1):
        candidate = generate(task, feedback=feedback)
        verdict = verify(candidate)
        if verdict.accepted:
            return candidate, attempt
        feedback = verdict.reason
    raise NoAcceptedOutput(task, attempts=max_attempts, last_reason=feedback)

Everything that determines whether this is a dependable component or an expensive way to be wrong lives inside the verify argument, and none of it is visible in the loop. What that function admits, what it rejects, how much it costs, and whether its answers are stable across runs are the four properties that the rest of this topic is about. The loop above will run happily with a verify that always returns accepted=True, and the service built on it will report excellent numbers.

Two properties of that fifteen lines are worth reading off explicitly, because they are the only guarantees the plumbing itself provides. The first is in the raise on the last line: the function returns a candidate only when verify accepted it, and otherwise it raises. There is no path through which an unverified output reaches the caller, which is a small thing to get right and the whole reason the loop is worth having as a function rather than as a pattern people reimplement. The second is feedback. It starts as None, and after a rejection it holds the verdict's reason, which is passed into the next generate call — so attempt two is not a re-roll of attempt one, it is a fresh attempt with a Postgres error string in the prompt. The two returned values matter as well: attempt comes back alongside the candidate, and that number is the observable everything downstream is monitored on, because a service whose mean attempt count is drifting upward is degrading long before its acceptance rate does.

The stumble this code invites is the belief that having written it, the output is verified. It is worth saying plainly why that feels true. The loop is the part that looks like engineering — it has a retry, a bound, an exception type and a feedback channel, all of which are real and all of which are correct. But every one of those mechanisms is about what happens after verify returns, and none of them constrains what verify returns. Swap in a function that accepts everything and the code above still runs, still bounds its attempts, still raises the right exception in principle and never in practice, and reports an acceptance rate of one. The guarantee is entirely in an argument, and arguments are the part of a system nobody reviews.

Where a loop amplifies cost instead of accuracy#

Three specific things this design does not give you are worth naming, because each one is a way teams talk themselves into believing they have more than they do.

It does not improve the generator. The accuracy of accepted output is bounded above by what the checker will accept, so a loop wrapped around a weak check produces exactly the same errors it always did, now stamped as verified. That bound is the subject of the next chapter and it is the most common way this pattern is got wrong.

It does not make failure free. Every retry pays for another generation and another check, and the requests that retry are disproportionately the ones that hit the slow path of both. A check that looks like noise at the median can dominate cost at the tail, which is a measurement problem rather than an intuition problem.

It does not help when the failures are correlated across attempts. The loop assumes that a second sample from the generator is a meaningfully different draw. If the model misreads the same ambiguous column name the same way three times running, three attempts cost three times as much and return the same rejected output. Feeding the check's reason back into the next attempt is what breaks that correlation when anything does, which is why Verdict carries a reason rather than a boolean.

The reason correlation is the default rather than the exception follows from what a retry actually does. A second attempt draws another sample from the same distribution, conditioned on the same prompt and the same input. Sampling introduces variation in how the answer is expressed, and it does not introduce new information about the ambiguous column name. If the model's most likely reading of that name is the wrong one, the most likely reading is still the wrong one on the second draw, and the third. Attempts are only independent trials in the sense that matters when something between them changes what the model conditions on — which is exactly what putting the constraint violation into the next prompt does, and exactly what a bare retry does not. The practical consequence is that a rising retry count is not a system working harder. It is a system spending three times the money to be wrong in the same way, and it should be alerted on rather than tolerated.

There is also a limit that no amount of engineering removes. Some outputs are valuable exactly because they are judgements, and a judgement has no cheap decider by construction. That is not a gap in your tooling. It is a property of the work, and the honest response is to say so at planning time rather than to build a rubric that produces a number and calls it verification.

The reordering is the whole move: name the rejecter, price it against the generation, and let the answer decide whether the feature is worth attempting at all. A sixty-to-one ratio makes a stochastic generator into a component you can schedule; a one-to-four-hundred ratio makes it into a way of buying human review you did not budget for, and no amount of prompt work moves a ratio by four orders of magnitude.

Everything after that depends on knowing what your chosen rejecter actually rejects, and that is much less obvious than it looks. A check that runs, returns quickly, and eliminates an entire class of production errors can leave the errors that matter completely untouched — and because it now runs on every output, it attaches its endorsement to them on the way past. The next chapter takes a specific check that a lot of teams have shipped, measures what it admits, and works out what the difference between "accepted" and "correct" is costing.