An answer nobody can check cannot be improved

Two projects were approved in the same planning session, eleven months ago, with roughly the same headcount. One was a natural-language interface over the analytics warehouse: an analyst types a question, a model writes SQL, the query runs, the analyst gets a table. The other was a regional strategy assistant: the model reads a quarter's worth of sales data, competitor news and CRM notes for a region, and writes the two-page memo that the regional director used to write by hand.

Today the SQL project is on its seventh prompt-and-retrieval revision and the team can tell you exactly what each revision bought. Executable-and-correct on the held-out set — 340 questions kept aside so that nothing scored against them has ever been shown to the model in a prompt — went 61% → 68% → 71% (schema summaries in context), then 71% → 79% (a handful of worked question-and-query examples, retrieved by question similarity and pasted into the prompt), then 79% → 84% (a repair loop on execution errors), then 84% → 88% (excluding four deprecated tables from the schema). Each of those numbers came from the same 340-item evaluation set, run in about ninety seconds against a fixture database, and each change that failed to move the number was reverted the same afternoon.

The strategy assistant is still on version one, because after each change the only available test was to read four memos and form an impression. Nobody was neglecting it: three engineers have changed the prompt perhaps forty times between them. Two of the engineers thought the March rewrite improved things; the third thought it made the memos longer and blander. The regional directors, asked directly, said the memos were fine and they had stopped reading them. There is no version two of that project, and there never will be, because there is no procedure by which anyone could establish that version two is better than version one.

Neither team was less capable than the other, and the property that separated them belongs to the problems rather than to the people. Deciding which business problems to hand to a large language model (LLM) and which to send back is a piece of engineering work in its own right, and the name for it is problem selection. It gets done in planning sessions, in the meeting after an executive has seen something at a conference, and in the roadmap review where four proposals compete for one team, usually with nothing to go on except a demo the proposer chose the inputs for. The filter that does most of the work is verifiability, meaning that some cheap and repeatable procedure exists which takes an output and says whether it was right. Running a generated query and comparing its rows against the rows a known-good query returns is such a procedure; reading four memos and forming an impression is not. Getting the choice wrong is expensive in an unusual way, because excellent model work and excellent platform work spent on a badly chosen problem still produce nothing anyone can bank, which is why a well-argued refusal is often the most valuable thing an engineer brings into the room.

What verifiability decides that accuracy does not#

Topic 5.7 established the verifier-first stance at the level of a single system: build the checker before the generator, know that an accept set is wider than the set of correct answers, and know that false accepts are invisible unless you have independent labels. That stance is usually taught as an engineering discipline for a project you have already decided to do. Used as a selection filter, before anything is built, it does something different and considerably more consequential. It tells you whether a proposal admits of engineering at all.

Everything that makes model work tractable descends from the ability to check an output cheaply and repeatedly. An evaluation set is a checker applied to fixed inputs. A regression test is the same checker applied after a change. A claim that the new retrieval strategy beat the old one is a difference between two checker runs with an interval around it. Prompt iteration, model selection, fine-tuning a model on your own examples, a decision about whether to spend six times more per call on a larger model — all of these are comparisons, and a comparison requires a measurement. When the measurement does not exist, work does not stop; it becomes unfalsifiable, which is worse, because the team keeps spending and keeps producing changes, and nobody can say whether the changes helped.

This is why verifiability goes first rather than fourth. A problem can be valuable, well-scoped, cheap to serve and enthusiastically sponsored, and if nobody can check the output, all of that value sits behind a wall you cannot climb with more engineering. The reverse case is much easier to live with: a problem with an excellent verifier and mediocre economics can be re-scoped, priced differently, or deferred, and the verifier survives the re-scoping.

The four things an output can be checked against#

Candidate outputs sort into a small number of families by what the check compares against. The families differ in how much they cost per check, how far they scale, and what class of error they are structurally blind to.

The last row is the strategy memo, and the reason to keep it in the table is that proposals arrive claiming to be in one of the first four rows when they are actually in the last. "We'll have subject-matter experts review the output" is a sentence that sounds like row four and behaves like row five, because a decision a human already makes is a record that already exists in a system, whereas an expert convened to review model output is a new cost per item that will be cut in month three.

Row four deserves more attention than it usually gets, because it is where most business problems actually live and it is the row teams overlook while they hunt for something executable. Any process where a person currently looks at an input and records an outcome has been quietly generating labelled data, often for years. A claims adjuster who marks a claim approved or referred, a moderator who applies one of eleven policy codes, an analyst who dispositions a security alert as true positive or benign — each of those is a row in a table with the input attached to the decision. The label quality is imperfect, and imperfect labels in volume are a functioning verifier, whereas a perfect label set that does not exist is not. The practical question to ask of any proposal is which table already holds the decisions, how far back it goes, and whether the input the human saw is still recoverable alongside it.

The SQL project sits in row one, and the check is worth writing out because its cheapness is the whole argument:

def check(question: str, generated_sql: str, reference_sql: str, conn) -> str:
    try:
        got = conn.execute(generated_sql).fetchall()
    except DatabaseError as e:
        return "execution_error"          # 11.2% of v1 outputs landed here
    want = conn.execute(reference_sql).fetchall()
    if Counter(map(tuple, got)) == Counter(map(tuple, want)):
        return "correct"                  # order-insensitive; the question rarely fixes order
    if not got and want:
        return "empty_result"             # the failure mode analysts complain about most
    return "wrong_result"

Three properties of that function do the work. It needs no human, so the marginal cost of one more evaluation item is the cost of writing the reference query once. It is deterministic on a fixed fixture, so a difference between two runs is a real difference and not sampling noise in a grader. And it produces a categorical failure label rather than a score, so the 12% of remaining failures on the current version decompose into 4.1% execution errors, 5.6% wrong results and 2.3% empty results — three separate engineering problems with three separate fixes, rather than one number that went down.

Compare the memo. There is no reference memo, because the regional director's own memos were not right in any checkable sense either; they were the director's judgement. There is no execution, because a memo does not run. There is no downstream system that accepts or rejects it. And there is no historical record of a decision, because the memo informed a conversation, not a decision with an outcome attached. That is four empty rows, and no amount of model capability fills any of them.

Where this filter is misapplied#

The most common misreading treats verifiability as a claim about how good the model is. It is not; it is a claim about the shape of the output and the environment it lands in. A model can be terrible at generating SQL and the SQL problem is still verifiable — that is precisely the situation the analytics team was in at 61%, and the verifier is what let them climb. A model can be excellent at writing strategy memos, in the sense that a reader would prefer its memo to the director's, and the memo problem is still unverifiable, because preference measured on four documents by three engineers is not a measurement you can run after every change.

A second misreading confuses verifying the output with verifying the business outcome. Executing generated SQL against a reference result tells you the query is right. It tells you nothing about whether the analytics team asks better questions, whether decisions improved, or whether the interface gets used. Those are worth measuring and they move slowly, on quarters rather than seconds, so they cannot serve as the inner loop. What a cheap output-level check buys is the ability to iterate at all; the business-level measurement is a separate and slower instrument, and a project needs both.

A third case is subtler and worth naming because it looks like success. Some outputs are checkable in principle and the check is expensive enough that it never actually gets run — a legal-document comparison that requires a paralegal forty minutes per item, or a code migration where the verifier is the full integration suite at nine minutes and $4 of compute per run. These are not unverifiable problems; they are problems where the evaluation set will be small, the iteration loop will be slow, and the schedule has to be planned around a hundred-item set rather than a ten-thousand-item one. 5.7's cost model applies directly here. Treating a forty-minute-per-item check as though it were the SQL check is how a team ends up with an eval suite of nine examples and a great deal of confidence.

Finally, verifiability is necessary and nowhere near sufficient. Plenty of perfectly checkable problems are bad projects: the check is cheap, the output is correct, and the thing has no users, or it costs more per item than the manual process, or a lookup table already does it better. The verifiability filter defends against none of that. What it does is order the questions, because the checkability question is the one that cannot be fixed later — scope can be narrowed, cost can be engineered down, and demand can be tested, whereas an output with nothing to check against stays that way regardless of what you spend.

What the filter leaves open#

Applying this to the two projects from the planning session gives one keep and one kill, and the kill is the more useful outcome, because eleven months of three engineers is roughly $400k of capacity that produced no measurable change in anything. Had the filter been applied at approval, the strategy assistant would have been rejected in an afternoon, and the version of it that survives — extract the five figures the director actually quotes in every memo, from systems where those figures are authoritative, and put them in a template — would have shipped in a fortnight with a check that is arithmetic.

The filter does not, however, tell you which of several checkable proposals to build, and it has nothing to say about the proposals where a model will simply produce mush. Every proposal that reaches a planning session comes with a demo showing that the model can produce something plausible, and that demonstration is worthless as a discriminator precisely because it succeeds everywhere. The properties that actually separate a good fit from a bad one, once checkability is established, are how ambiguous and language-shaped the input is and how much error the output can absorb, and those are what the next chapter puts to work on four live candidates.