Model-authored eval sets

An eval set — short for evaluation set — is a fixed collection of inputs to an AI system, stored together with the outcome each input is supposed to produce, and held somewhere stable enough that the whole collection can be re-run on demand. Running it means feeding every input through the system, comparing each output against the stored expectation, and reducing those comparisons to a score. It occupies the position a test suite occupies in ordinary backend work: it lives in CI and sits behind a release gate, the check a change has to pass before it ships. What an earlier topic established is that the comparison at the end has to be a judgement about whether an answer was good, because an equality assertion against generated text fails on wording that was never wrong.

A golden dataset, also called a golden set, is an eval set of one particular kind. The inputs are real, drawn from the traffic the system actually received, and the expected outcome recorded beside each one was decided by a person who read it. The name is borrowed from the golden files of ordinary testing, where a stored, human-blessed output is the reference every later run is diffed against. It exists because a generative system offers nothing that can tell you whether its own answer was right — no exit code, no schema violation, no assertion to run — so the reference has to come from outside the system. Building one is the work this topic covers. This chapter is about the alternative that gets built first, because it can be built in an afternoon: a set whose questions and reference answers were both produced by prompting a model. Taking those two halves apart in turn accounts for most of the difference between what such a set reports and what the product does.

The generator writes the questions it already knows how to answer#

A model asked to produce a realistic customer support question produces a question drawn from its own notion of what such a question looks like, and that notion is the average of a great deal of well-formed text. The generation prompt does not have to be naive for this to happen. Here is one that asks explicitly for variety and for a range of difficulty:

GEN_PROMPT = """You are generating evaluation data for a retail support assistant.
Write 20 realistic customer support questions covering a range of intents
(order status, returns, refunds, shipping, account access) and a range of
difficulty. For each, also write the ideal answer an excellent support agent
would give. Return JSON: [{"question": ..., "reference_answer": ...}]"""

Three of the rows it returned, unedited:

[
  {"question": "Hi, I ordered a pair of running shoes last week (order #48213) and the tracking hasn't updated in four days. Could you tell me where my package is?",
   "reference_answer": "I'm sorry about the delay. Order #48213 shipped on 12 March via DHL and is currently held at the Barcelona sorting facility. Packages usually clear within 48 hours..."},
  {"question": "I'd like to return a jacket I bought on 3 April. What is your return policy and how do I start the process?",
   "reference_answer": "Our return window is 30 days from delivery for unworn items with tags attached..."},
  {"question": "I've been charged twice for the same order. Can you help me get the duplicate charge refunded?",
   "reference_answer": "I can help with that. Duplicate authorisations are usually released automatically within 5 business days..."}
]

Every one of those is a single intent, correctly spelled, politely framed, and — the detail that matters most — it carries the identifier needed to answer it. The customer volunteers the order number in the first message, in the format the system expects, without being asked. Now three messages from a real Thursday at the same retailer, transcribed with the original spelling intact:

[1] "whers my stuff"

[2] "hola, compré unas botas la semana pasada y llegaron rotas — I want to
     return them but also I moved house last month, do you have my new
     address? and can you cancel the other order too"

[3] "Re: your last message — no. I said the SECOND one. the one from
     august not the one from september. this is the third time"

Message 1 has no order number, no name, no timeframe, and one intent expressed in two words. Message 2 switches language mid-sentence and contains three requests, one of which — the address change — is not a support question at all but an account mutation with its own authorisation rules, and one of which refers to an order the assistant has not been told about. Message 3 is turn six of a conversation whose earlier turns are load-bearing, contains a correction rather than a request, and is angry in a way that changes what a good answer looks like.

Why the generated collection could not have contained a message like any of those three is worth following one step at a time, because the answer decides what can be done about it. A model returns a score for every fragment of text that could come next, and something outside the model picks a high-scoring one, so what comes back from a request for a realistic support question is whichever continuation of that request scores well. A continuation scores well by being typical of text of that shape — the kind of question a reader would accept as a fair example of the genre. Every property that makes message 1 hard to handle is a property that makes it untypical: it was written by one person, on one keyboard, at one level of impatience, with no intention of being a fair example of anything. Generation therefore draws from near the centre of a distribution over what support questions look like, and a product fails at the edges.

There is a cheap way to put a number on how far apart the two collections are, and it is worth running before anyone argues about it in the abstract. Take a thousand generated rows and a thousand real first messages, embed them, label them by origin, and fit a logistic regression to predict which is which. If the two came from the same distribution the classifier should be near chance, which on the AUC scale — the area under the ROC curve, a standard summary of how cleanly a classifier separates two groups, running from 0.50 for a coin flip to 1.0 for perfect separation — means 0.50. On one team's data it reached 0.97 AUC with a bag-of-characters model and no embeddings at all, which means the two collections are separable almost perfectly on surface features alone: message length, punctuation density, the presence of a digit. A classifier that can tell your test data from your production data at 0.97 is telling you that a model can too.

Two steps in that recipe carry the whole diagnostic, and neither is obvious. The first is why chance is the right expectation when the two collections match. A classifier can only beat chance by finding a region of its input space where one label is denser than the other, and if the two collections were drawn from the same distribution there is no such region anywhere — every neighbourhood holds generated and real messages in the same proportion, so no rule does better than guessing. Separation is therefore evidence about the two distributions rather than evidence about the cleverness of the classifier, which is what licenses reading the number as a property of your data.

The second is what the number means, because "area under the ROC curve" describes a plot rather than a quantity. The AUC is the probability that a randomly chosen generated message is scored as more generated-looking than a randomly chosen real one. A coin flip gets that right half the time, which is the 0.50 floor; a rule that never overlaps gets it right every time, which is the 1.0 ceiling. At 0.97 the two collections come apart ninety-seven times in a hundred on surface features and nothing else.

The comparison table in the next section shows how little work that takes. An identifier is present in the opening message of essentially every generated row and in 41% of the sampled real ones, so the single rule no identifier in the first message means real is never wrong about a generated row and is right about nearly three in five real ones, without reference to meaning at all. Add message length, and a check for mixed language at 0% of generated rows against 12% of real ones, and very little overlap survives. Each of those features is also visible to the assistant on every request it serves, and that is the reason a separability figure is worth computing rather than being a curiosity: a property that distinguishes your test data from your production data is a property the system under test can condition on.

The mismatch is not a matter of the generator being lazy. You can push it toward messiness — ask for typos, ask for multi-intent, ask for missing identifiers — and it will oblige, in the way a model obliges, which is to produce a stylised imitation of messiness. Generated typos land on the letters a model considers typo-prone; real typos come from a specific keyboard layout, a specific autocorrect, a specific person typing on a phone while walking. Generated multi-intent messages tend to have two clean intents joined by "also"; real ones bury the second intent in a subordinate clause of the first. The gap narrows with effort and does not close, because what you are trying to sample is the joint distribution of ten thousand customers' circumstances, and the only place that distribution exists in full is your logs.

The reference answers are the other half of the problem#

Even if the questions were perfectly representative, the answers in a generated table came from a model and are then used as ground truth to grade a model. When the grader's notion of a good answer and the system's notion of a good answer come from the same training distribution, they agree on things that are wrong together. One such set contained a reference answer stating a 30-day return window; the actual policy had been 14 days for sale items since January. The assistant said 30 days too, so the case passed, and the same error was being repeated to real customers at a rate of several hundred a week.

The step that makes this failure possible, rather than merely likely, is worth separating out. Grading a generated answer against a generated reference asks one question: do these two pieces of text say the same thing? Both sides of that comparison were produced by scoring plausible continuations. A fact that sits outside what is plausible — a return window your company shortened in January, which no amount of general familiarity with retail would predict — is therefore absent from both sides at once. The comparison contains no term that could disagree with the world, because the world was never consulted at either end. That is a different failure from a weak grader: a weak grader is wrong in both directions and its errors partly cancel, while this one is reliably right about the wrong thing, and reliability is what makes the score look trustworthy.

This is a specific, checkable failure and it does not go away by using a different vendor for the grader. A competing model removes the shared-weights problem and leaves the shared-prior problem: both models believe that a plausible-sounding retail policy is a retail policy. Independence of weights is not independence of belief, and it is the belief that is doing the grading. Ground truth for what your return window is lives in a document your company owns, and a case whose expected answer was never checked against that document is not a test, it is two models agreeing.

The row that decides the argument is the last but one. A test suite exists to fail when the system is wrong, and a set that inherits the system's errors cannot fail for that reason.

The property a test needs, stated generally, is that its expected value was arrived at without consulting the thing under test. A golden file has that property because a person read the output and blessed it; an ordinary assertion has it because somebody wrote the expected value down from a specification. A model-authored reference has it only if the model's belief about your return window came from your return window, and nothing in the generation prompt made that happen. What is left is a comparison between two draws from one source. A comparison of that shape measures how much the source varies, which is a real quantity and is not the correctness of either side, and the closer the two draws agree the better the eval looks.

Sourcing from traffic instead means pulling more than the customer's text. A case has to carry enough context to reproduce the failure, which for a retrieval-and-tools assistant means the full turn sequence, the documents that were retrieved, the tool calls with their arguments and their responses, the prompt version and model version in force, and the trace id — the handle on the recorded execution of that one conversation — so you can go back to the original. A row containing only the question and the final answer will let you notice that something was wrong and will not let you find out why, and it will silently change its own verdict the next time the retrieval index is rebuilt.

The obvious tension is that such logs are full of names, emails, addresses and order ids, and an eval store is a different security context from production — more people read it, it gets copied into notebooks, it ends up in CI. Sanitising it is mandatory. Sanitising it badly destroys the case, because an assistant cannot look up an order id that has been replaced with a placeholder, and a retrieved document that no longer mentions the customer no longer matches the conversation. That problem has a real solution and it is involved enough to need its own chapter later in this topic; for now the instruction is to pull the cases with their context intact and to treat the sanitised store as a design problem rather than a regex.

The jobs synthetic cases genuinely do better#

Refusing to use generated data at all is the overcorrection, and it costs real coverage. Several jobs exist where sampling from traffic is either impossible or strictly worse.

Before launch there is no traffic. A new intent, a new market, a new tier of the product — you have zero conversations and a gate to pass, and generated cases are the only cases that exist. They are a scaffold rather than a benchmark: they check that the plumbing works and that the obvious paths are wired, and they get retired as soon as real conversations arrive. The claim such a set can support is a claim about the code path rather than about the population. It can show that a request reaches the order tool, that the tool's response reaches the answer, and that a refusal fires where a refusal was configured. It cannot show that the answers are the ones customers will need, because the population that would decide has not written in yet, and no amount of generated volume brings that decision forward.

Rare-and-severe scenarios may never appear in a month of logs and still must not be shipped broken. A customer asking the assistant to disclose another customer's order, a prompt-injection payload embedded in a returned document — instructions written to be obeyed by the assistant rather than read by the customer — a request that would trigger a regulated disclosure: you cannot wait for these to occur in production, and if they do occur you have already had the incident. Adversarial cases are authored on purpose, by people thinking about what would be bad, and generation helps produce variants once you have the first one.

Controlled perturbation is the strongest use of all, because it starts from a real case. Take a sampled conversation the assistant handles correctly, and generate twenty variants with the order number missing, the tone hostile, the language switched, the message length tripled with irrelevant narrative. The variants inherit the realism of their seed and isolate a single variable, which is what you want when the question is whether the system breaks when an identifier is absent rather than how good it is overall. The seeds have to be real for this to work; a perturbation of a fiction is a fiction.

Why that isolation works is the same reason a recorded fixture works in an ordinary test, and it is worth spelling out because it is the property the whole technique rests on. When a verdict changes between a seed and one of its variants, exactly one thing about the input moved, and every other thing — the account state, the documents that came back, the customer's history, the turns before this one — stayed at a value some real customer actually produced. The change in verdict therefore has one candidate cause. Generating a whole case instead moves all of those at once, so a failure has nothing in particular to be attributed to. What a perturbation set measures is not how good the assistant is; it is how much one variable matters, which is a narrower question and one that a small number of cases can answer honestly.

Volume itself is occasionally the point, for questions that are not about answer quality at all. Measuring p99 latency — the response time that only the slowest 1% of requests exceed — under a realistic token distribution, estimating what a prompt change does to the monthly bill, or checking that a new tool integration does not deadlock under concurrency needs thousands of plausible inputs and no ground truth whatsoever. Generated traffic is ideal there, because the only property required is that the inputs resemble production in length and shape. Generated data also travels where real data cannot: sharing a set with a vendor, publishing it, or running it in a lower-trust environment is straightforward when no sentence in it was ever written by a customer.

sampled real traffic  ->  the benchmark behind the release gate
perturbations of it   ->  robustness probes, one variable at a time
authored adversarial  ->  the things that must never happen
purely generated      ->  pre-launch scaffolding, retired on first real traffic

A generated set is a real instrument pointed at the wrong population. Its inputs are the model's prior over what a customer message looks like, which is clean, single-intent and carries its own identifiers; its expected answers are the model's prior over what a good answer is, which encodes whatever the model believes your policy says. Neither half can fail for the reasons a product fails, and the score it produces is precise, stable and about something other than your customers. Where no traffic exists, where the scenario must never occur, or where the question has no ground truth at all, that is exactly the instrument you want — and in each of those cases the reason it works is that nobody is asking it to stand in for the population.

There is one more reason to be deliberate about which conversations you pull. The next question everyone asks — how many rows the set needs before its number can be trusted — has an answer that surprises people, because adding rows stops helping much earlier than the statistics textbook suggests, and a hundred carefully chosen conversations can be a sharper instrument than ten thousand.