Sizing a golden dataset

A golden dataset is an eval set whose inputs are real conversations and whose expected outcomes were decided by a person. Sooner or later somebody looks at how few rows it has and asks how a system serving hundreds of thousands of conversations a month can be gated on a hundred-odd examples. The instinct behind the question is correct and it is applied to the wrong quantity: statistical power is a property of independent draws, and the ten thousand rows a generator produces are not ten thousand draws from anything.

This chapter works out what an extra row is worth, which turns out to depend almost entirely on where it came from. It concedes the part of the confidence-interval argument that is true, applies the correction survey statisticians use for clustered samples, and then deals with the deeper problem that no amount of n fixes: an interval is an interval around whatever the set measures, and bias does not shrink with sample size.

What another row actually buys#

Take the arithmetic at face value first, because the argument is stronger if you concede the part that is true. For a set scoring around 0.90, a 95% Wilson interval on the pass rate — the standard confidence interval for a proportion, which stays sensible near 0 and 1 where the textbook formula does not — has a half-width of roughly ±0.054 at n=120, ±0.028 at n=450, and ±0.006 at n=10,000. Those are real reductions, and if your question is genuinely what the pass rate of this system is on this distribution, more independent samples answer it better.

Those three figures are one law seen at three points, and knowing the law is what lets you predict a fourth without running anything. The width of an interval on a proportion is set by that proportion's standard error, which is the square root of the quantity p times one minus p, divided by n — where p is the pass rate being estimated and n is the number of independent cases behind it. The n sits inside a square root in a denominator, and that placement is the whole shape of the thing: quadrupling the cases halves the interval, and halving it again costs four times as much as the first halving did. Check the law against the row above. Going from 120 cases to 450 is a bit under fourfold, and the half-width comes down by about a half. Going from 450 to 10,000 is about twenty-two-fold, the square root of that is a shade under five, and the half-width falls to about a fifth of what it was. The Wilson form in the code below adds an adjustment that matters near 0 and near 1, where the textbook formula can put an interval outside the range a proportion is allowed to occupy, and it changes none of that shape.

The clause carrying all the weight is independent. Generated sets are clustered: the model produces a limited number of underlying scenarios and then varies the surface. Cluster ten thousand such rows by embedding and you typically find something like five hundred distinct scenarios with twenty near-identical variants each. Rows inside a cluster pass and fail together, so they contribute one observation between them rather than twenty. The survey-statistics correction for this is the design effect, which converts a clustered sample into the smaller number of independent draws it is actually worth:

from math import sqrt

def wilson_halfwidth(p, n, z=1.96):
    denom = 1 + z**2 / n
    return (z * sqrt(p * (1 - p) / n + z**2 / (4 * n**2))) / denom

def effective_n(n, cluster_size, rho):
    """Kish design effect: deff = 1 + (m - 1) * rho"""
    return n / (1 + (cluster_size - 1) * rho)

p = 0.90
print(round(wilson_halfwidth(p, 120), 3))        # 0.054  — 120 sampled cases
print(round(wilson_halfwidth(p, 10_000), 3))     # 0.006  — 10,000 rows, claimed
n_eff = effective_n(10_000, cluster_size=20, rho=0.85)
print(round(n_eff))                              # 583
print(round(wilson_halfwidth(p, n_eff), 3))      # 0.024  — 10,000 rows, honest

The design effect in that function is easier to trust once it has been checked at both ends. Write m for the number of rows in a cluster and rho for how strongly rows inside one cluster share a verdict. The effect is one, plus m minus one, times rho. Set rho to 0 — rows that happen to sit in the same cluster but behave independently — and the effect is one, so a sample of n rows is worth n draws, which is right. Set rho to 1 — rows that always agree — and the effect is m, so the sample is worth n divided by m, which is also right, because a cluster whose members never disagree tells you precisely what any one of its members would have told you on its own. Everything between those two ends is a straight line drawn between the two answers, and the straightness is the adjustment's approximation rather than a fact about anybody's data.

Running it on the figures in the code takes one division. Clusters of twenty give m minus one of nineteen; nineteen times 0.85 is a little over sixteen; so the design effect is a little over seventeen. Ten thousand rows divided by a little over seventeen is the 583 the code prints, and feeding 583 back through the Wilson function is where ±0.024 comes from. That is the entire correction, which is worth knowing, because the objection it answers usually arrives with a great deal more confidence than one division deserves.

An intra-cluster correlation of 0.85 — rho in the code above, which measures how strongly rows inside one cluster share a verdict, at 0 for rows that behave independently and 1 for rows that always agree — is not pessimistic for paraphrases of the same scenario, and it is measurable rather than assumed. Cluster the set by embedding at a similarity threshold you pick by eye, take fifty clusters, run the set under two configurations that differ enough to move verdicts — a prompt edit, a model version — and count how often the members of a cluster all flip together versus splitting. The proportion that move as a block is your rho, and on generated sets it typically lands between 0.7 and 0.9. Doing this once takes an afternoon and settles the question permanently for that generator.

What that procedure deliberately does not measure is how similar the rows look, even though embedding similarity is what formed the clusters in the first place. It measures whether rows share a verdict when something about the system moves, and those are different quantities. Twenty paraphrased wordings of one scenario are worth one draw because they pass and fail together, not because they read alike, and a generator that produced twenty wordings the assistant handled differently would be producing twenty draws whatever the embeddings said. The clustering is a cheap way to find candidate groups; the two-configuration run is what tests whether the group behaves as one.

The honest interval on ten thousand generated rows is therefore about ±0.024, roughly four times wider than the naive calculation and only about twice as tight as a hundred-and-twenty-case set. That factor of two costs the token bill for ten thousand generations, the storage, the eighty-fold increase in evaluation cost on every run, and the wait — at 1.8 seconds per case and 20-way concurrency, 120 cases finish in eleven seconds and 10,000 in fifteen minutes, which is the difference between running the gate on every pull request and running it nightly.

Every row costs the same to run and they differ by orders of magnitude in what they return, which is why row count is a bad unit of account.

Precision on the wrong quantity#

The deeper problem is that an interval, however correctly computed, is an interval around whatever the set measures. Statistics of this kind quantify sampling error and say nothing about bias, and bias does not shrink with n.

That last clause is carrying most of the argument, and the reason behind it is short enough to be worth stating rather than asserting. An estimate computed from a sample converges on the expected value of the thing you sampled. Taking more draws reduces how far any particular estimate wanders away from that expected value, and nothing about taking more draws can move the expected value itself, because the expected value is a property of the population you drew from rather than a property of how many times you drew from it. Every additional row is one more draw from the same place, so the interval closes tightly around whatever that place happens to produce.

If the set is drawn from a model's idea of a support question, then as n grows you converge — beautifully, tightly, with a shrinking error bar — on the system's pass rate over model-authored questions. The previous chapter put that quantity at 0.94 for one assistant whose production quantity was near 0.62. Ten thousand rows would have reported 0.94 ± 0.006 instead of 0.94 ± 0.054, which is a more confident statement of a number nobody needed.

The name for what you meant to measure is the estimand, and it is worth writing down in the set's README in one sentence, because the act of writing it exposes the mismatch. "The proportion of conversations in last month's production traffic that a support lead would judge correctly handled" is an estimand. "The proportion of generated questions the assistant answers in a way another model likes" is also an estimand, and once both are on the page nobody argues for the second. A tight confidence interval is a statement about the repeatability of a measurement, not about its accuracy, in exactly the way that a scale reporting your weight to three decimal places tells you nothing about whether it was zeroed.

This is where the comparison to a test suite is useful and where it also breaks down. A suite of 10,000 unit tests that all exercise the same three functions is worth less than 100 tests that touch a hundred code paths, and every engineer already believes this about tests. Coverage, not count, is the currency. The place the analogy breaks is that unit tests are deterministic, so one test per branch suffices, whereas a case here has a noisy verdict and needs several cases per behaviour before the per-behaviour rate means anything — one case in a slice gives you 0/1 or 1/1, which is compatible with almost any underlying rate.

Sizing by coverage instead of by count#

The practical procedure inverts the usual order. Rather than picking n and sampling, enumerate the things the set has to be able to say something about, then let n fall out of that. For one retail assistant the enumeration ran to fourteen entries: order status with an id, order status without an id, returns inside the window, returns outside the window, refunds on a split payment, address change mid-shipment, multi-account name collision, three-intent messages, mixed-language messages, hostile follow-ups after a wrong answer, questions about a policy that changed recently, conversations longer than twelve turns, retrieval returning nothing relevant, and injection attempts inside retrieved documents.

At six to ten cases per entry that is roughly a hundred cases. It is not a magic constant; it is fourteen behaviours times enough cases each to distinguish "usually works" from "usually does not". Six cases in a slice give a 95% interval of about ±0.30 on that slice's rate, which is coarse and still decisive for the thing you use it for: a slice going from 6/6 to 2/6 between releases is a signal, and no amount of aggregate score would have shown it.

The two statements in that sentence look contradictory and are not, and the reconciliation is the reason per-slice rates are worth buying. The ±0.30 says how badly you know the slice's true rate from six cases, which is badly. It does not say how well you can see that rate change, because the same six cases are re-run under both releases and most of them do not move. Four cases changing verdict in the same direction between two releases of a system that was otherwise stable is not something noise produces often. Estimation and detection are different jobs, and a set that is poor at the first can be sharp at the second.

The second half of the sizing question is which cases within a slice to keep, and the answer is the ones near the boundary where behaviour flips. A case the system passes under every configuration you will ever ship contributes nothing to any comparison; it is dead weight that inflates the denominator and makes every score look better. Those are worth keeping only as a smoke test, and worth counting separately, because the honest statement is that regressions are being detected by the cases that can move.

That also changes the statistics you should be running. Releases are compared on the same fixed set, so the comparison is paired, and the relevant quantity is the discordant cases — the ones that changed verdict. A release where 9 cases went fail-to-pass and 1 went pass-to-fail, on 120 cases, gives an exact McNemar p of about 0.021 — McNemar's test discards every case whose verdict was unchanged and asks only whether the split between the two directions of change is more lopsided than coin flips would produce — which is a defensible claim of improvement from a set whose unpaired interval is ±0.054 and would have called the change indistinguishable.

from scipy.stats import binomtest

fail_to_pass, pass_to_fail = 9, 1          # verdict changes between two releases
p = binomtest(pass_to_fail, fail_to_pass + pass_to_fail, 0.5).pvalue
print(round(p, 3))                          # 0.021

Throwing away the cases that did not change sounds like a way of flattering a result, and the reason it is legitimate is worth working through. Both releases were run over the same fixed set, so each case produces a pair of verdicts rather than two independent observations. A case that came out the same way under both releases is equally consistent with the new release being better, worse or identical, and so contributes nothing towards telling those three apart. A case that changed carries a direction. Under the hypothesis that the two releases are equally good, a case that changed is as likely to have gone one way as the other, so the 10 discordant cases are 10 coin flips and the question reduces to how surprising a nine-to-one split is. Counting splits at least that lopsided in either direction comes to about two in a hundred, which is the 0.021 the code prints.

What that buys is visible in the two intervals. The unpaired calculation treats all 120 cases as sources of noise on both sides of the comparison; the paired test looks only at the 10 that moved and ignores the rest. The comparison has stopped being between two scores and become a count of the cases that changed their minds, which is a much smaller thing to be uncertain about. It is available only because the set is fixed. The moment cases are added or removed between two releases the pairing is gone, and that is one more reason a golden dataset carries a version.

Where the small set genuinely cannot help#

A hundred-odd cases cannot detect a one-point drift in overall quality, and pretending otherwise is how these sets lose credibility. If the real question is whether a prompt change moved the aggregate from 0.910 to 0.899, a paired set will shrug, and the honest answer is that the question needs online measurement — an A/B test on live traffic with thousands of conversations per arm, scored by a proxy like escalation rate or thumbs-down rate — rather than a bigger offline set. Offline evaluation and online experimentation answer different questions, and a golden set is a diagnostic instrument for finding out what broke, not a precision estimator of aggregate quality.

There are two situations where the answer genuinely is more rows, and conceding them keeps the argument credible. Validating an automated scorer against human labels is a supervised learning problem in its own right, and estimating that scorer's agreement to within a few points needs several hundred labelled cases, not a hundred — the labels are the training and test data for a different model, and the sample size is set by that model's needs rather than by the product's failure modes. The other is a rare safety event where the acceptable rate is very low: demonstrating that a disclosure bug occurs in under 0.1% of conversations requires observing thousands of conversations of the relevant kind, and no hundred-case set can support that claim. Both are real, both are expensive, and both are different projects from the release gate.

A small set is also poor at anything about the tail beyond the slices you thought of. A behaviour nobody enumerated is a behaviour with zero cases, and zero cases produce no failures, so a small set is exactly as blind as its authors were. Generated volume does not fix this either — the generator is at least as blind — but it is why the enumeration deserves the time it takes, and why it has to be revisited when the product changes.

What a hundred real cases buy that ten thousand generated ones do not is independence and coverage, and both of those are properties of where the rows came from rather than of how many there are. Independence is what makes an interval mean what it says; coverage is what decides whether a regression has anywhere to show up. A set sized by enumerating behaviours and putting six to ten cases on each will have a wide aggregate interval and a sharp per-slice signal, which is the right way round for an instrument whose job is to tell you what broke rather than to estimate a population mean to three places.

The remaining problem is the one a sixty-conversation Thursday sample walks into directly. Even sampling from real traffic, if you draw conversations at random you get the traffic mix, and the traffic mix is dominated by the easy majority intent while the cases that hurt you sit at a fraction of a percent. Two hundred conversations pulled uniformly can contain zero instances of the bug that will cost you a customer, and the set will score 0.91 and say nothing about it.