Annotation is funded work or your metrics are decoration

The slide said 89% of assistant answers rated acceptable, n = 10,412, and the sample size was the reason nobody in the room questioned it. The labels had been produced over three sprints by two engineers working through an exported spreadsheet in the gaps between tickets, marking each row good or bad on their own reading of what the product should do. There were no written guidelines, no adjudication of disagreements, and no point at which two people had labelled the same row.

Sofia, who had just taken ownership of the eval data, ran a small check before the next board cycle. She drew 200 rows at random, had both original labellers redo them blind against the rubric, and included 100 rows that one of them had already labelled four weeks earlier. The two labellers agreed with each other on 62% of the 200. The engineer who re-labelled his own earlier work agreed with himself on 71% of the 100. The labelling operations topic already covered why a headline agreement figure has to be chance-corrected before it means anything; the relevant fact here is simpler, which is that a measurement instrument that disagrees with itself three times out of ten is not measuring the system. The 89% described the labellers' varying moods about what counted as acceptable, and the 10,412 in the denominator made that description precise rather than true.

Volume buys precision, calibration buys validity, and they are not interchangeable#

The two properties get conflated because both feel like "more data makes it better", and only one of them responds to more data. Sample size controls how tightly you can bound a number: at 90% acceptable, a sample of 300 gives a 95% interval of roughly ±3.4 percentage points, and 10,400 gives ±0.6. Calibration controls whether the number is of the thing you meant. If uncalibrated labellers are systematically 8 points optimistic because nobody defined what to do with answers that are correct but omit the fee schedule, then the 10,400-item estimate is 8 points wrong with a ±0.6 interval, and the interval makes the error harder to notice rather than easier.

The last row uses two terms that recur below: a gold set is a small body of items whose labels have been argued out to a single agreed answer and are then treated as the reference, and a judge model is a model prompted to score outputs against the rubric so that people do not have to score all of them. Neither is worth anything without the property in the right-hand column. That column is what an evaluation set has to be to serve the role chapter 3 gave it. The left-hand column is what teams produce when annotation is something engineers do with their leftover attention, and it costs six times as many hours to produce the version that answers fewer questions.

Turnaround time decides whether the gate covers anything new#

The property that gets left out of these discussions entirely is how long it takes to get labelled data for a failure mode you discovered this week. Chapter 2's gate requires evaluation results on the affected suite, which works only when the suite contains cases for the behaviour the change touches. A new failure mode — the assistant confidently answering proration questions, say — needs sampled production traces, a guideline decision about what the correct behaviour is, labels from at least two people, and adjudication of the cases they split on.

When that pipeline is unfunded, it runs at the speed of whoever is least busy. On the billing team the first attempt took nineteen days from we should have cases for this to the cases are in the suite, during which four prompt changes and one retrieval change merged with evidence from a suite that could not see the problem. With a funded owner and a defined sampling and adjudication routine, the same request ran at 48 hours for 200 stratified items, which is inside the cycle time of a normal pull request. A gate that can only be applied to last quarter's failure modes is a gate that new behaviour walks straight through.

The recurring work is what the funding pays for, and it is not a burst.

# plan/eval-data-workstream.yaml — steady state, ~0.6 FTE at 8k sessions/day
weekly:
  - stratified sample of 250 production traces across 5 slices   # 1.5 h
  - primary labelling pass, 3.5 min/item                          # 6.0 h
  - double-labelled overlap on 15% of items                       # 1.0 h
  - adjudication of disagreements                                 # 1.5 h
  - agreement report to prompt/serving owners                     # 0.5 h
quarterly:
  - gold-standard refresh, 120 items, three-way labelled          # 12 h
  - guideline revision + re-label of affected historical items    # 8-20 h
  - judge revalidation against the refreshed gold set             # 4 h
on_demand:
  - new failure mode: 200 stratified items, 48 h SLA
  - onboarding a new labeller: shadow pass + calibration set      # 6 h

Hamel Husain's note on how many people should annotate LLM outputs makes the same point about staffing from the other direction: the number of annotators you need falls out of the throughput and the overlap you want, and the recurring calibration is the part that cannot be dropped, because guidelines that are not exercised stop being shared.

The line about a stratified sample is doing quiet work in that plan. A uniform random draw from production reproduces production's own proportions, so the out-of-corpus questions that caused the worst regression in chapter 2 arrive at roughly 3% of the sample, which is seven items in a draw of 250 and supports no conclusion about that slice at all. Stratifying by slice and then weighting back when you report an overall figure gets you 40 to 50 items in each of the five slices per week, which is enough for a slice-level trend within a month. On top of that sits targeted sampling of a suspected failure mode, drawn deliberately rather than randomly, and those items go into the suite as cases while being excluded from any population estimate, since a sample chosen because it looked broken cannot tell you how often things are broken.

One hire, six engineers, no annotation function#

The concrete version of this decision is a budget line. The team has six engineers, no annotation capability, and money for one more person. The argument for a seventh engineer is throughput: more hands, more changes, more features. The argument against it is that the team's constraint is not producing changes.

Last quarter that team merged 34 behaviour-affecting changes. Eleven carried evaluation evidence. The other 23 shipped on spot checks, because the suite did not cover the behaviour they touched and getting coverage took nineteen days. Two of those 23 caused measurable production regressions, one of which was the retrieval change from chapter 2, which ran for three weeks at 41% higher token cost — about $3,400 a month at their volume — while cutting correct refusals by twenty points. A seventh engineer takes 34 changes to roughly 39, of which perhaps 13 would carry evidence, so the team's rate of unverified behaviour change goes up.

The hire is not a data-entry role, which is the misreading that makes the argument sound like asking for a headcount to fill spreadsheets. The person owns the definition of correct as an operational thing: they write and revise the guideline, run calibration, maintain the gold standard, run the sampling, and hold the authority chapter 3 assigned to the eval-data owner. On a team this size the labelling volume itself is often better bought than staffed — a specialist vendor at $2 to $4 per domain item can absorb bursts — while calibration, adjudication and the gold standard stay in-house, because those are the parts where the judgement being encoded is your product's judgement.

The argument as laid out so far is an engineering argument, and the person holding the budget is usually not moved by measurement hygiene. The version that travels is about the rate of unverified change: a two-thirds unverified rate on a system that talks to customers about money, and two regressions last quarter that between them cost something in the order of $10,000 in excess tokens plus support handling nobody costed. Framed that way, the decision is whether to raise the volume of changes or the fraction of them that anyone has checked, which is a question a director can hold an opinion about.

Agree the evidence for the hire before making it, because otherwise the role gets judged on labels produced per week, which is the one number that does not matter. Three that do: the share of behaviour-affecting changes merging with evidence, the elapsed time from discovering a failure mode to having cases for it in the suite, and the measured agreement on the current gold set. None of the three could be reported at all in the previous state, which is itself part of the case.

The judge does not remove the annotation, it moves it#

The standing objection is that a model can do the labelling, at a thousandth of the cost, which is true for the bulk of the scoring and untrue for the part that makes it trustworthy. A judge's scores are only worth what its agreement with human labels on your data says they are worth, so a judge is an amplifier on a human-labelled gold set rather than a replacement for one. The Berkeley work Who Validates the Validators? is the research statement of this: LLM-assisted evaluation has to be aligned against human judgement on the actual task, and the criteria themselves drift as people look at outputs.

In practice that turns into a validation step the judge has to pass before its numbers are allowed into a pull request.

def validate_judge(judge, gold, min_agree=0.80, min_kappa=0.60):
    """gold: list of items with human-adjudicated labels, three-way where split."""
    preds = [judge.score(item.input, item.output) for item in gold]
    truth = [item.label for item in gold]

    agree = sum(p == t for p, t in zip(preds, truth)) / len(gold)
    kappa = cohen_kappa(preds, truth)
    fn_rate = sum(p == "ok" and t == "bad" for p, t in zip(preds, truth)) / \
              max(1, sum(t == "bad" for t in truth))

    report = {"n": len(gold), "agreement": agree, "kappa": kappa,
              "missed_failures": fn_rate}
    if agree < min_agree or kappa < min_kappa or fn_rate > 0.15:
        raise JudgeUnvalidated(report)
    return report

The missed_failures term is the one that decides whether the judge is usable as a gate, because a judge that agrees on 85% of items by being generous about the 15% that are bad will report every regression as fine. The 120-item quarterly gold refresh in the plan above exists to feed this function, and revalidation is scheduled rather than incidental, since the judge's alignment decays whenever the prompt, the model build or the input mix moves. Three hundred adjudicated items calibrating a judge that then scores 50,000 production outputs is a good trade; three hundred items that nobody refreshes calibrating the same judge a year later is not a trade at all.

None of this argues that sample size is irrelevant. A calibrated set of 40 items will give you an interval so wide that most changes are unmeasurable, and the answer there is to sample more, not to relax the calibration. The claim is narrower: precision without calibration is a well-specified estimate of the wrong quantity, and precision is the property that money buys easily while calibration is the property that requires someone whose job it is.

Nor does every team need a full-time person. Below a certain volume the workstream is a named half-day a week held by a specific engineer, protected in the plan the way an on-call rotation is protected. What makes it funded is that it appears in the plan with hours and a name attached, so that when a delivery deadline arrives it has to be explicitly cut rather than quietly evaporating.

Assume all of it is in place: gates on behaviour-affecting paths, owners with real authority, a calibrated eval set refreshed on a schedule. There is still no way to know from inside the team whether any of it is written down, because the people who built it can execute it from memory.