An optimiser proposes candidates, scores them, and keeps the winners

Automated prompt optimization means handing the writing of a prompt to a program that decides between wordings by scoring them rather than by reading a few answers and forming an impression. The program is called an optimiser, and its loop is short. It takes the current prompt as a starting point, produces a variant of it — usually by asking a large language model (LLM) to rewrite the instruction after reading cases the current prompt got wrong — runs that variant over a set of stored cases whose correct answers the team has already agreed on, has a grader turn each output into a number, and keeps the highest-scoring variants as the seed for the next round of rewriting.

The technique exists because the human version of the same loop is slow. An engineer hand-tuning a prompt gets through perhaps a dozen wordings in an afternoon, and the comparison between wording seven and wording six usually rests on a handful of outputs read by eye. A team that already has an evaluation set and a grader — the CI evaluation of topic 3.16, run over the versioned prompts of topic 4.10 — has everything the machine version needs to run unattended overnight across hundreds of wordings, each scored on the same cases in the same way, so the scarce input becomes provider spend rather than attention. It shows up in three forms. Frameworks such as DSPy expose optimisers by name — GEPA, MIPROv2, BootstrapFewShot — and rewrite the prompts inside a program you wrote. Managed services, Amazon Bedrock's prompt optimisation among them, take a prompt and hand back a rewritten one. Teams also write the loop themselves in a few hundred lines, because its shape is not complicated. The vocabulary below applies to all three, and so do the ways their output misleads a reader.

The four moving parts#

Strip away the framework and there are four. A proposer generates candidate prompts. An objective turns a candidate into a scalar by running it over a set of cases and letting a grader score the outputs. A selector decides which candidates survive to seed the next round. A budget decides when to stop. Everything else — reflective mutation, Pareto frontiers, bootstrapped demonstrations — is a choice about how one of those four is implemented.

The proposer is where the approaches differ most, and the differences matter because they change what the search can and cannot find. Three families dominate in practice.

The bootstrapped variant is the one people underestimate, because it does not look like prompt engineering at all. The instruction text stays fixed and the search runs over subsets of demonstrations, usually generated by running the current program over training inputs and keeping the traces where the grader said the answer was correct. On a classification task with a fiddly output schema this frequently beats every rewrite, because the model was never confused about the task and was only confused about the shape of the answer.

The step under that result is worth making explicit, because it is what tells you in advance which tasks demonstrations will help on. An instruction describes the shape of an answer, and a model reading one has to turn the description back into a shape before it can produce anything. A demonstration is the shape, already in the form the decoder emits. So on a task where the model understands what is being asked and gets the layout wrong, swapping a description for an exhibit removes the exact step that was failing. On a task where the model has misunderstood what is being asked, it removes nothing, because the examples now show the right layout wrapped around the wrong answer — which is why a bootstrapped run on a genuinely misunderstood task comes back with a tidier version of the same mistake.

The objective is where your judgement lives, and it is the part the framework will not supply. A run needs a function from (prompt, case) -> float, and in most real systems that function is either a deterministic check (exact match, schema validity, a regex over a field, a unit test on generated code) or a model-graded rubric. What the optimiser optimises is that function, exactly and literally, including any part of it you did not mean.

import dspy

def rubric_score(example, prediction, trace=None) -> float:
    """0.0-1.0 from a judge model; 0.0 if the answer is not parseable."""
    parsed = parse_answer(prediction.answer)
    if parsed is None:
        return 0.0
    return judge(question=example.question,
                 answer=parsed,
                 reference=example.reference)

optimizer = dspy.GEPA(
    metric=rubric_score,
    reflection_lm=dspy.LM("openai/gpt-4.1", temperature=1.0),
    auto="medium",
)

winner = optimizer.compile(
    program,
    trainset=tuning_cases,      # 200 cases the search may look at
    valset=tuning_val_cases,    # 60 cases used inside the loop for selection
)

Note what is absent from that call: the holdout. Whatever cases appear in trainset or valset have contributed to choosing the winner and can no longer tell you how the winner behaves on inputs it was not selected for — the same exhaustion that topic 3.5 works through for held-out sets and read budgets, arriving here at several hundred selections an hour instead of a few a day. The mechanical rule is that a separate holdout exists — a slice of the evaluation cases split off before the run, which no part of the loop is allowed to read — the optimiser is never given a handle to it, and it is scored exactly once, after the run is over.

Why a case that has been scored against cannot also report on the winner is a short chain, and walking it once is what makes that rule mechanical rather than a matter of taste. The selector reads scores. Those scores were computed on the tuning cases. The candidate that survives is therefore the one those particular cases happened to reward — including in the places where they rewarded it for a quirk of their own phrasing rather than for anything about the task. So the winner's score on those cases has the reason the winner was chosen mixed into it, and no operation performed afterwards separates the two. A case that took part in the choosing cannot also be a witness to it. Anyone who has trained a model will recognise the shape of that: a test set you tune hyperparameters against stops being a test set, by exactly this mechanism and for exactly this reason.

The selector matters less than the proposer but is worth knowing about, because it explains why two runs of the same optimiser produce winners of very different character. The simplest selector keeps the top k candidates by mean score, which drives the population towards whatever wording is currently ahead and tends to converge quickly on one lineage. Frontier-based selectors, of the kind GEPA popularised, instead track which candidate is best on each individual case and keep every candidate that is best on at least one, so a prompt that is mediocre overall but uniquely good at the three multilingual cases stays alive. That preserves diversity and is the main reason reflective evolutionary search can be sample-efficient — GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning reports reaching comparable or better quality than RL-based tuning with substantially fewer rollouts, and frontier selection plus natural-language reflection on failures is the mechanism it credits.

The step being skipped in that claim is why diversity should buy sample efficiency, since the two sound unrelated and one is a cost. A top-k selector throws away everything below the cut. A candidate that is behind on the mean is usually behind because it is worse on most cases, but sometimes it is behind because it is much better on a handful and slightly worse everywhere else, and the mean is precisely the statistic that cannot tell those apart. The second kind is where a new idea lives. Keeping every candidate that leads on at least one case keeps those in the population, so the next round of rewriting has more than one shape to work from. The efficiency follows: the scarce resource in this loop is evaluation passes, and a population that has collapsed onto a single lineage spends its passes re-measuring variations of one prompt.

Where the randomness lives#

If you run the same search again, do you get the same prompt? The answer depends on which of three independent sources of randomness you have pinned, and pinning one of them is a common and expensive mistake, because it produces a run that looks reproducible and is not. Two of the three are governed by the same control, the sampling temperature, which sets how far a model is allowed to stray from its most likely next token; at 0 it takes the most likely token every time, and at 1.0 it samples widely enough that the same request twice returns different text.

The first and third rows of that table both say temperature=0, and the same setting is doing two unrelated jobs in them. A judge is an instrument, and an instrument that returns a different reading each time it is used adds noise to every comparison the selector makes, so pinning it takes that noise out of the run. A proposer is not an instrument. It is the source of the variety the search consumes, and the same setting there removes the variety along with the noise. The row governed by neither of them is the middle one, which is exactly the row a team that has pinned its temperatures will believe it has covered.

A concrete illustration of why the middle row is not a detail. In a typical configuration, each candidate is first screened on a random 25-case minibatch, and only the 40 candidates that clear a threshold are re-scored on all 200 tuning cases. Sampling 25 from 200 has an enormous number of possible draws, and the standard error of a mean rubric score over 25 cases — the amount that mean moves from one random draw of cases to another, with the prompt held fixed — is around 0.06 when per-case scores have a standard deviation of 0.3. A candidate that is genuinely 0.02 better than its parent will fail that screen a substantial fraction of the time, and a candidate that is genuinely 0.02 worse will pass it a substantial fraction of the time. The seed that decides those draws therefore decides which lineage survives to round 2, which is most of what decides the final answer.

That 0.06 is one division, and it is worth doing rather than accepting. Take the spread of the individual case scores — a standard deviation of 0.3 means a typical case lands about that far from the prompt's own average — and divide it by the square root of how many cases are in the draw. The square root of 25 is five, and 0.3 divided by five is 0.06. The square root is there because averaging independent numbers makes their errors partly cancel: a draw that happens to contain an unusually easy case is likely to contain an unusually hard one as well, and the more cases in the draw, the more thoroughly the two offset. That is also why buying precision this way is so expensive. Quadrupling the minibatch halves the wobble rather than quartering it, so the screen would cost four times as much to become twice as trustworthy.

Setting the two figures beside each other is what makes the screen look different from the way it is usually described. The improvement being hunted is 0.02 and the noise on the measurement is 0.06, so the quantity the screen is trying to see is about a third the size of the quantity it cannot help seeing. The tempting reply is that a 25-case draw is small but unbiased, so across 400 candidates the errors wash out. They do not, because each candidate gets one draw and is judged on it. A lucky draw promotes a mediocre candidate into round 2 and an unlucky one retires a good candidate for good, and nothing later in the run revisits either decision.

The practical configuration is to hold all three in one place and record it with the run:

RUN = {
    "seed": 7,                        # drives all shuffling and minibatch draws
    "proposer": {"model": "gpt-4.1", "temperature": 1.0},
    "judge":    {"model": "gpt-4.1-mini", "temperature": 0.0, "n": 1},
    "budget":   {"max_candidates": 400, "minibatch": 25},
    "tuning_cases": "evalset@a41f2c9#tuning",   # content-addressed
    "holdout_cases": "evalset@a41f2c9#holdout",
}

The proposer runs at temperature 1.0 deliberately, because a proposer at temperature 0 explores almost nothing; the seed is what makes that exploration repeatable rather than the temperature. And even with all three pinned, bit-exact reproduction is not on offer from a hosted model — provider-side batching, hardware and routing make identical requests return different tokens often enough that you should not build a process on the assumption of determinism. What a seed buys is a run you can repeat, argue about, and compare against another seed, which is enough.

Fine-tuning, evaluation quality and the engineer's own reading#

The obvious neighbour is fine-tuning, and the two are frequently discussed as alternatives to each other, which obscures the more useful distinction. Fine-tuning and RL-based methods change model weights, so they need a serving path for a custom model, they are invalidated wholesale when you change base models, and their output is not readable. A prompt search changes only text. The artefact fits in a code review, transfers to a different provider by pasting it into a different request, and costs nothing to serve beyond its own tokens. That is the entire reason the technique is worth the trouble at senior level, and it is also why the failure modes in the rest of this topic are operational rather than statistical.

It is also not a substitute for having a good evaluation. An optimiser is a machine for climbing whatever gradient your metric provides, and if the metric rewards long confident answers, it will find long confident answers within about forty candidates. Teams reach for prompt search when their evaluation is weak, on the theory that the machine will figure out what good looks like; the machine cannot, and a weak grader converts a search budget directly into overfitting. The AWS guidance on optimizing generative AI prompts is blunt about this ordering: evaluation quality bounds what automated refinement can deliver.

Finally, it is not a replacement for the engineer's understanding of the failure. Later in this topic a search run and an hour of reading failures are put side by side on the same slice, with engineer time priced in, and the ordering that comes out of that comparison is not the one the tooling encourages.

Everything downstream of this loop is a reading problem, and the loop is what makes the reading possible. A winning prompt carries no record of which proposer wrote it, which cases screened it, how many siblings it beat, or which of its sentences the grader was actually rewarding — so a team that has the artefact and not the configuration has bought a text nobody can evaluate. Recording the four parts and the three seeds costs a dictionary in a file and is the difference between a candidate and an anecdote.

The immediate open question is narrower and needs settling first. That run reported a winner scoring 0.91 where the hand-written prompt scored 0.79, on the 200 cases it searched over. The number is real, it is correctly computed, and it is not an estimate of anything you care about.