The optimiser searches over your bugs as well as your prompt

Reading failures and running a search are both ways of improving a prompt's score, and they are usually discussed as a matter of taste — whether you trust a machine or an engineer. The useful distinction is different. A search can only move the part of the failure rate that a prompt change can move, and it does not know what that part is; reading thirty failures and clustering them by cause measures it, in about ten minutes, before any budget is committed.

The second difference matters more and is less obvious. Whatever is broken upstream of the model when the search runs is part of what the search is optimising against, so its winner may be compensating for a defect rather than improving on a baseline. That compensation earns real points while the defect is live and becomes dead weight, then a regression, the moment somebody fixes it.

Clustering failures by cause#

The instrument is a function that assigns each failing case a cause, run over the failures of a single holdout scoring.

from collections import Counter

fails = [c for c in run_holdout(baseline_prompt) if c.score < 0.5]

def signature(case) -> str:
    if case.retrieved_doc_version != case.expected_doc_version:
        return "retrieval:wrong-doc-version"
    if not case.retrieved:
        return "retrieval:empty"
    if case.expected_span not in " ".join(case.retrieved):
        return "retrieval:span-missing"
    return "generation:" + case.judge_reason_category

print(Counter(signature(c) for c in fails).most_common())
# [('retrieval:wrong-doc-version', 11), ('generation:incomplete', 7),
#  ('retrieval:span-missing', 6), ('generation:wrong-clause-number', 4),
#  ('generation:hedged-unnecessarily', 2)]

That histogram is also a budget. Seventeen of the thirty failures have a retrieval cause and thirteen have a generation cause, which means that a hypothetically perfect prompt — one that made no generation error at all and changed nothing upstream — could recover at most thirteen of the thirty. Each recovered failure is worth about 0.4 of a point of rubric score across an 86-case holdout, so the entire addressable surface for any prompt change is roughly 6 points. A search that reports 1.2 points against an addressable 6 has taken a fifth of what was there and has room left; the same 1.2 points against an addressable 2 means the room is nearly gone and next month's run will find nothing worth the $47. The number is crude, and it is still the difference between running the search again with confidence and running it out of habit.

Two steps inside that are worth doing rather than accepting. The split into seventeen and thirteen is a sum over the histogram: the signatures beginning retrieval: account for eleven and six of the thirty, with the empty-retrieval case never firing, and the ones beginning generation: account for seven, four and two. Both groups add up and the thirty divide with nothing left over, which is the check that signature is exhaustive rather than quietly tipping cases into a default nobody reads. The 0.4 of a point is the second step. A holdout score is a mean, so a single case moving changes the headline by however far that case moved divided by 86. A failing case does not score zero and a recovered one does not score a perfect 1, so the swing on the case itself is a fraction of the scale rather than the whole of it — and about a third of the scale, spread across 86 cases, is the 0.4 of a point quoted above. Thirteen of those bound everything a prompt change could ever reach.

Setting the two interventions against each other is only meaningful if both are measured the same way — the same holdout, the same grader, the same judge configuration, the same reporting rules — and if the human hours are counted, because those are the scarce input and the machine hours mostly are not.

The two gain-per-hour figures are each a division, and it is worth seeing which numbers go into them. The search returned 1.2 points for 90 minutes of engineer attention; 90 minutes is an hour and a half, and 1.2 divided by that is the 0.8. Error analysis returned 4.0 points for 180 minutes, which is three hours, and 4.0 divided by three is the 1.3. Machine hours appear in neither division because they are not the scarce input: six hours of overnight compute on a machine that would otherwise be idle costs the provider spend in the row above and nothing further.

Gain per engineer-hour is the row that gets quoted, and on its own it is the weaker argument. A ratio of 1.3 to 0.8 is not a large enough difference to build a policy on, and it would flip on a week when the failures happened to be diffuse. The last two rows carry the real weight, and the final one is the whole point.

The winner compensates for whatever is broken#

When a searched winner is re-scored after an upstream fix, a large share of its margin can disappear, and reading the prompt explains where it went. Among the instructions a reflective proposer had added to one winner was this, produced after it read a batch of failures in which the retrieved passages came from the wrong version of a contract:

If the retrieved passages appear to describe a different version or amendment
of the agreement than the one the question asks about, do not answer from them
directly. State which version the passages appear to come from, and give the
general structure of the clause rather than quoting specific terms.

That is a genuinely clever instruction and it earned its points honestly on the tuning cases, because on a system where 13% of date-mentioning queries silently retrieve the wrong document, an answer that notices and says so is better than one that does not. On the fixed system it is worse than useless. It fires occasionally on correct retrievals where an amendment is legitimately referenced alongside the original, and it converts a specific, quotable answer into a vague one.

Laying that out as a chain is what makes the reversal predictable rather than surprising. The retrieval filter is broken, so a share of date-mentioning queries arrive carrying passages from the wrong version of the contract. The grader scores an answer against a reference, so an answer that confidently quotes the wrong version scores badly while one that names the mismatch scores less badly. The proposer reads those failing traces, sees the pattern, and writes the instruction that produces the second behaviour. The instruction therefore earns points, and they are real points, measured correctly, on the system as it stood. Then the filter is fixed. The condition the instruction tests for is now almost never true — but almost never is not never, and on the occasions when an amendment is legitimately cited beside the original it still fires, producing a vaguer answer than the baseline would have given. The same sentence that was worth points is now worth negative ones, and nothing about the sentence has changed.

This generalises past retrieval bugs. Every component that is broken, misconfigured or about to change is part of what the search is compensating for. A prompt searched against a chunker — the component that cuts documents into the passages an index stores — that splits mid-table will contain instructions about reconstructing tables. A prompt searched against a tool whose error messages are unhelpful will contain instructions about retrying. Each of those is a real gain on the current system and a liability on the fixed one, and none of them announce themselves in the score. The nearest familiar analogue is tuning a query plan by hand while a needed index is missing: the tuning is not wrong, it is an elaborate compensation that becomes dead weight and then becomes a regression.

The operational consequence is an ordering. Reading failures comes first, because its findings are permanent and improve every prompt including whichever one the search eventually produces. Search comes after the components underneath it have stopped moving. A workable trigger for when the ordering flips: read thirty recent failures and cluster them by cause, and if no cluster larger than three shares a non-prompt cause, the structural defects are exhausted for now and the remaining failures are diffuse enough that a search has something to do.

The threshold is doing one job, and knowing which job keeps it from being an arbitrary number somebody picked. A cluster means several failures share one cause, so one fix retires all of them together, and that is the whole reason component work is cheap per failure recovered. A cause appearing once or twice in thirty is about as likely to be two unrelated accidents as a defect, and chasing it costs the same engineer-hours to recover one case. So the trigger is not "is there a bug underneath" — there is always a bug underneath. It is "is there a bug whose fix pays for more than one failure". When the answer stops being yes, what remains is diffuse, and diffuse is the shape a search handles well and a reader handles badly.

The loop back through the baseline re-score is the edge people leave out, and leaving it out is how a stale margin ends up in a release note. After a component fix the baseline is worth more than it was, so every previously measured margin was computed against a number that no longer exists, and a two-point winner over the old baseline may be a zero-point winner over the new one. Re-scoring costs one holdout scoring and about twenty minutes; not re-scoring costs the credibility of every comparison downstream of it.

Not every change forces a re-run, and being able to say which do saves a great deal of pointless recomputation.

The first two rows are the expensive ones and they are also the most frequent, which is the practical reason a search belongs on a schedule that follows pipeline work rather than one that runs beside it.

Where the argument for reading stops#

Error analysis does not dominate search on every axis, and treating it as always-better is its own mistake. It does not scale — one engineer reads thirty cases an hour, and a hundred thousand production conversations cannot be read at all. It is not repeatable, in the sense that two engineers reading the same thirty failures find different things, and neither can hand their process to a script. The classification is scriptable once a cause has been discovered, which is what the signature function is, and committing it turns one afternoon's reading into a standing report: a regression in retrieval:wrong-doc-version shows up as a count rather than as a customer complaint, and the next person to sit down with thirty failures starts from last quarter's taxonomy instead of from raw text. What cannot be scripted is the step that produced the taxonomy, since a signature function only ever recognises causes somebody has already named.

Its sampling is also easy to get wrong in a way that manufactures work. The thirty cases above were the thirty lowest-scoring, which is the obvious way to pick them and biases what you find, because the bottom of the score distribution is enriched for grader failures and for cases whose reference answer is itself wrong. Five of these thirty had reference answers that did not match the source document, and an hour spent "fixing" those is an hour spent editing the evaluation set — worth doing, and not a system improvement, and dangerous to count as one because it moves the very number you are measuring against. Sampling in two strata avoids most of this: take half the cases from the bottom decile, where structural bugs concentrate, and half uniformly from everything scoring below the median, where the ordinary failures live. Neither stratum shows you cases that scored well for the wrong reason, which is a separate blind spot that only a read of passing cases will close.

Most importantly, reading is blind to exactly the failures a search is good at: diffuse, low-magnitude wording problems spread across every case, where no cluster exists to find and no single failure is interesting on its own. The two also consume different resources. A search is machine time that runs overnight without supervision; error analysis is an engineer's attention during the day. On a team of six this means both can be in flight, and the constraint is not headcount but sequencing: do not run a search against a component someone is actively changing, because the run's output will be invalidated by their merge and you will have spent $47 and ninety minutes of review to learn nothing.

The rule that survives all of that is narrow. Cluster before you search, because the histogram costs ten minutes and tells you both how much room a prompt change has and whether the room is currently full of somebody else's bug. Fix the structural clusters, re-score the baseline, and only then spend the budget — and treat any winner produced against a moving pipeline as a candidate with an expiry date rather than as a result.

What is left after the fix and the re-run is a prompt that is 2.4 times the length of the baseline, adds 180 ms to p95, wins by 2.1 points on a clean holdout, passes every constraint filter, and reads like text a machine wrote — which it did. Nobody on the team feels able to edit it confidently. The provider has announced a new model version in three weeks. Deciding what to do with it at that moment, in the room, under whatever pressure the week supplies, is how a prompt with no owner ends up in production for a year.