Agreement is consistency; a gold standard is correctness

Confirming the order-number hypothesis took two people and two days. The support operations lead and the engineer who owns the taxonomy each labelled the same 400 items independently — 30 drawn from each of the 12 classes, plus 40 sampled from the model's current errors — and then sat down and adjudicated every item where they differed, which was 37 of them, writing a one-line rationale for each ruling. Six of those rulings changed the guideline. At roughly 110 seconds an item that is about 12 hours of labelling each plus 90 minutes of adjudication, so a little over 26 hours of senior time, or $2,500 at a loaded rate. The output is 400 items whose labels carry the authority of the people who decide what the labels mean.

That last property is the whole point, and it is what a reference sample has that no amount of reviewer agreement can supply. The vendor pool can be perfectly self-consistent while applying a convention nobody with authority over the taxonomy ever endorsed. A gold set breaks the circularity by introducing a labelling source that is not drawn from the pool being measured.

Measuring reviewers against the reference instead of against each other#

With the 400 items in hand, each of the six vendor reviewers labelled a 150-item stratified subset of them, blind, mixed into ordinary work. The result is a defect rate per reviewer, and separately a defect rate per slice.

The shape of that table is the diagnosis. The overall column is tight — 88.7% to 91.3%, a spread of 2.6 points across six people, which is what a well-calibrated pool looks like. The slice column is uniformly bad, with every reviewer between 20.8% and 33.3% on the same 24 items. A defect that shows up in all six reviewers at the same rate is not a reviewer defect, because six people do not independently develop the same wrong habit; it is a property of the document they all read, or of the training they all received.

The contrasting shape is worth holding in mind, because it calls for an entirely different response. If R4 had come back at 71% overall while the other five sat above 89%, nothing about the guideline would be implicated and the action would be to retrain or replace one person. Reviewer-vs-gold measurement separates those two situations in a way that reviewer-vs-reviewer measurement cannot, because in a pool of six the single divergent reviewer drags every pairwise agreement figure down slightly and the aggregate reads as mild, diffuse noise.

Keeping the measurement running matters more than the one-off audit. Gold items are seeded into the ordinary work queue at a low rate so that reviewers cannot tell which items are being scored, and the defect rate is computed on a rolling window.

GOLD_RATE = 0.03  # 3% of served items are known-answer

def next_item(queue, gold_pool, rng):
    if rng.random() < GOLD_RATE and gold_pool:
        return gold_pool.sample(), True
    return queue.pop(), False

def rolling_defect_rate(annotations, gold, window=200):
    """annotations: newest-last list of (item_id, label, is_gold)."""
    scored = [(a, gold[a.item_id]) for a in annotations if a.is_gold]
    recent = scored[-window:]
    wrong = sum(1 for a, truth in recent if a.label != truth)
    by_class = {}
    for a, truth in recent:
        s = by_class.setdefault(truth, [0, 0])
        s[0] += a.label != truth
        s[1] += 1
    return wrong / len(recent), {k: v[0] / v[1] for k, v in by_class.items()}

At a 3% seed rate and 400 items per reviewer per week, each reviewer hits about 12 gold items a week, which is far too few to trust weekly but enough to detect a real drift over a month. Raising the rate to 10% gives faster detection and costs 10% of the throughput, and that trade is a budget line rather than a methodological question. The gold pool has to be large enough that reviewers do not begin to recognise individual items, which in practice means retiring an item after it has been served four or five times and re-cutting the pool when the guideline versions.

Sizing it is a question about the smallest defect rate you want to be able to detect. Detecting a per-reviewer defect rate of 10% against a target of 5% at any useful confidence needs on the order of 150 to 200 scored items per reviewer, which at a 3% seed rate is about ten weeks of that reviewer's work — too slow for a running control and fine for a quarterly audit. Detecting a slice-level defect needs items in the slice, and a 24-item slice supports "this is badly broken" and nothing more precise. The practical shape most projects land on is a 400-item reference set, re-cut roughly every time the guideline accumulates enough rulings to change what a correct label is, with a one-off 150-item audit per reviewer at the start of an engagement and continuous seeding thereafter as a drift alarm rather than as a measurement.

Who holds the pen on the reference set is a governance question and it should be settled before anything is labelled. Rulings made during gold construction are guideline changes, so they belong to the same owner and the same versioned document as the rulings that come out of the adjudication queue; a gold set maintained by the model team in a spreadsheet, diverging quietly from the guideline the reviewers read, reproduces the two-conventions problem with the roles reversed.

Checking the labels on the examples the model keeps failing#

The gold set answers whether reviewers are applying the intended convention. A second instrument answers a different question: on the specific examples the model gets wrong, is the label right? Those are not the same population, because the model's residuals are concentrated exactly where the data is hardest and therefore where labelling defects cluster.

The procedure is to sample the residuals, read each one, and assign it a code describing why it is wrong, then count the codes. Hamel Husain's write-up "Why is error analysis so important in LLM evals, and how is it performed?" is the canonical description of this loop, and the discipline it insists on is that codes are invented while reading rather than chosen from a list in advance, then merged once the same code has been written three or four different ways.

On a sample of 100 residuals from the intent classifier, the counts came back like this.

Fifty-eight of a hundred errors are the model being marked wrong for having learned the vendor convention. Nineteen are the modelling problem the team was about to spend a quarter on, and 19% of 357 residuals is about 68 items out of 3,000, which caps the available gain from a bigger encoder at roughly 2.3 points. The eleven undecidable items are a data pipeline problem — the export truncates conversations at 4,096 characters and the intent is often stated in the last turn — and no label on them can be correct.

What a gold set is not#

It is not truth. It is the considered judgement of the people who own the taxonomy, which is exactly what you want when the question is "which convention should the corpus encode", and is worth nothing at all if those people have the downstream use wrong. A gold set built by an engineer who thinks the labels feed a report, for labels that actually feed a router, institutionalises the wrong convention with more authority than the vendor's version had.

It is also not a majority vote of the reviewer pool. Taking three reviewers from the same pool and adjudicating by majority feels like a rigorous upgrade and reproduces the shared error three votes to nothing, since the whole failure mode under investigation is that the pool agrees. If cost forces you to build the reference from pool members, the mitigation is to have the guideline owner review and sign each adjudicated item, which is a fraction of the effort of labelling from scratch and restores the property that matters.

And it is not a bigger held-out set. The held-out set measures the model; the gold set measures the labelling process. They are drawn differently — the gold set is stratified to over-sample rare classes and known-hard slices, which would bias any accuracy figure computed on it — and they age differently, since a held-out set is meant to stay frozen while a gold set has to be re-cut whenever the guideline changes enough to alter what a correct label is.

Turning the finding into an action with a price#

A finding without a price attached is a discussion. Each of the five codes above maps to a different intervention with a different cost and a different owner, and the reason to put them side by side is that the cheapest fix is rarely the one the team reaches for first.

The re-labelling line is the one that surprises people, because the instinct on discovering a guideline defect is to re-label everything. Restricting the sweep to items that match the ruling — here, training items containing an order number and labelled order_status, which is about 7,200 of 60,000 — turns a $19,600 re-run into a $2,350 correction, and the only thing that makes the restriction safe is that the ruling was written down precisely enough to be turned into a query.

Every figure in that table is a rate multiplied by a volume: seconds per item, defect rate, adjudication share, ramp curve. On this project the rates were available because four rounds had already been run and the timing data was sitting in the tool. On a project that has not started, none of them exist, and the standard move is to estimate them from the shape of the task, which is where estimates acquire the factor of two that shows up later as a schedule slip.