Ground truth for a judgement task is a stratified afternoon's work

You have taken the cost arithmetic and the calibration argument to the team that owns the compliance pipeline, and the response is the reasonable one: what would change your mind. The pipeline has 9,000 archived decisions, five policy areas, six months of audit records, and no labels. There is no benchmark for "does this supplier document meet our internal data-handling policy", there will never be one, and the team's position is that this is precisely why they built a debate — because the task is a judgement call that cannot be scored.

The task cannot be scored automatically. It can be scored, by the two contract managers who have been signing off on the policy for four years, and the amount of their time this needs is about a day and a half.

Sixty documents chosen on purpose#

The instinct is to grab a random hundred, and random sampling wastes most of them. Roughly 76% of the archive came back compliant, and four of the five policy areas produce a few hundred documents each while data handling produces four thousand. A random hundred gives you seventy-six compliant documents, forty-something from one policy area, and two or three from the area where the expensive mistakes happen.

Stratifying fixes this by deciding in advance which cells of the population the estimate has to cover, and sampling each cell to a fixed depth regardless of how common it is.

Sixty documents, six per cell, ten cells. The non_compliant half is over-represented by a factor of about three relative to the archive, which is deliberate, because those are the decisions with consequences and a rate estimated on twelve of them is uninformative. The distortion is undone at analysis time by weighting each sampled case by the size of the cell it came from.

from collections import defaultdict
import random

def stratified_sample(records, per_cell=6, seed=7):
    rng = random.Random(seed)
    cells = defaultdict(list)
    for r in records:
        cells[(r["policy_area"], r["verdict"])].append(r)
    sample, weights = [], {}
    for key, pool in cells.items():
        picked = rng.sample(pool, min(per_cell, len(pool)))
        sample.extend(picked)
        weights[key] = len(pool) / len(picked)   # cases represented per pick
    return sample, weights

def weighted_accuracy(results, weights):
    num = sum(weights[r["cell"]] for r in results if r["correct"])
    den = sum(weights[r["cell"]] for r in results)
    return num / den

The seed is in the code because the sample has to be reproducible. Six weeks from now someone will ask whether the result would have held on a different draw, and the answer is much better when you can rerun the same procedure with seed=8 and show it.

Two refinements are worth the extra ten minutes. Exclude anything from the first three weeks of operation, because prompts were being changed then and those decisions do not represent the system as it runs now. And record the document length for each pick, because if the sixty happen to be short documents the result will not transfer to the 90-page agreements where the debaters' quotes are hardest to check.

Labelling blind, and what blind has to mean#

The expert must not be able to tell what any system said about a document, which rules out the obvious shortcut of showing them the pipeline's verdict and asking whether they agree. Agreement rates collected that way run ten to twenty points high, because reading a verdict before forming one is the anchoring problem from chapter 1 with a human in the anchored position.

What the expert sees is the document, the policy text, a free-text box for the reason, and a three-point difficulty flag. What they do not see is any model output, any prior verdict, or the other expert's answers. Splitting thirty documents each with fifteen overlapping gives coverage of all sixty plus a measurable agreement rate, and the fifteen overlap cases are worth more than their share of the effort.

On this pipeline the two contract managers agreed on 12 of the 15 overlap documents, giving a Cohen's kappa of 0.71, kappa being an agreement score that discounts the agreement two labellers would have reached by guessing. That number is doing two jobs. It bounds what any system can score, because a target that two experts disagree on 20% of the time is not a target anything reaches 95% on. And the three disagreements, once the two of them sat down together, split into two that resolved immediately when one had missed an appendix and one that did not resolve at all — the retention-schedule document where both readings survived a twenty-minute conversation. That last one goes into the ambiguous class from chapter 4 rather than into the gold set, and its existence at roughly one in fifteen tells you what fraction of the archive is not decidable at all.

The time this took: an afternoon of engineering to pull the sample, strip the verdicts and build the labelling sheet; eight hours of expert time across the two of them at about eight minutes a document; two hours of adjudication. Total elapsed time was three working days, and the reason this work gets skipped is not that it is long. It is that it is the only part of the project that is not a model call, so it does not fit in the sprint the way another prompt iteration does.

Two arms, and the third one you should not add#

With sixty gold labels, replay the incumbent debate pipeline and a plain single-pass classifier over the same sixty documents, using the same model, the same policy text and the same document extraction. Then fill in the comparison from chapter 6.

One document of difference on sixty, and a discordant split of 5 against 4. The recall row, which counts what share of the genuinely non-compliant documents an arm managed to catch, is the only one with any shape to it, and eight versus five misses on thirty non-compliant documents is still inside what a reshuffle produces. Six months of operation at eight times the price has bought a result that a three-day experiment cannot distinguish from the classifier it replaced.

Sixty cases cannot detect a small difference, and that is the correct amount of power — the chance an experiment of a given size has of noticing a difference that is really there — for this decision rather than a limitation of it. The detectable difference on sixty paired cases is around ten percentage points, so this experiment answers "is the debate doing something large enough to justify 8x" and does not answer "is the debate 0.6 points better". Those are different questions and only the first one is on the table. A system costing eight times the baseline that cannot clear a ten-point bar has failed a test calibrated to what it charges.

The tempting third arm is a debate with the two debaters split across different model families, on the reasoning from chapter 2 that decorrelating their errors would fix the shared-blind-spot problem. Adding it here is a mistake for two reasons. It changes two variables at once — the ensemble structure and the model mix — so a result in its favour would not say which change produced it, and a result against it would not say either. And it answers a question that only becomes interesting after the current one is settled: if the incumbent debate is indistinguishable from a single pass, the finding is that this task does not benefit from adversarial structure, and a more expensive adversarial structure is not the indicated next step. Cross-family debate is a reasonable experiment to run second, on the gold set you have now built, once the first result is in.

Writing the decision rule down first#

The rule that turns this into an experiment rather than a demonstration is agreeing what the numbers mean before anyone sees them. Three sentences in the ticket, signed off by the team lead and whoever owns the budget:

Pre-registered decision rule — compliance debate evaluation

Primary metric: weighted accuracy against the 60-document gold set.
Ship/keep debate if weighted accuracy exceeds single-pass by >= 5.0
points AND non-compliant recall is not worse.
Otherwise retire the debate arm and revert to single pass.
Secondary (reported, not decisive): calibration, latency p95, cost.
Sample and seed fixed before labelling: stratified_sample(seed=7).

Without that, a one-document difference in the debate's favour becomes "debate is ahead" in the retrospective, and every ambiguity in the analysis resolves towards the system people have already spent six months building. Pre-registration costs ten minutes and removes almost all of that.

Standard practice is a claim about other people's tasks#

The team lead's position — that debate is standard practice for high-stakes decisions — is worth taking seriously rather than deflecting, partly because it is half true and partly because dismissing it converts a technical question into a status contest.

The half that is true is that adversarial structure has a genuine record on determinate tasks, and the original multiagent debate paper measured real reductions in arithmetic and factual errors, with later work such as the ICLR blogpost Multi-LLM-Agents Debate: Performance, Efficiency, and Scaling mapping out where those gains hold up and where sampling the same model twice does as well for less. Debate is not folklore.

The half that does not carry is the inference from "it works on some tasks" to "it works on this one". The published gains come from settings where a claim can be driven down to something checkable, and from ensembles whose participants fail in different ways, and this pipeline has neither: the debaters are one model with opposite instructions, and the judge cannot see the document. What the evidence supports is that debate helps under conditions, and the useful move is to name the conditions and ask which of them hold here.

The version of that conversation that works is short and concrete. Agree that the pattern is real and that high-stakes decisions deserve more than one pass. Note that the archive contains no evidence about this deployment either way, so both positions are currently unsupported. Propose the three-day experiment with the decision rule written first, and offer to run it yourself. Point out that the outcome is useful in both directions, since a debate that clears five points is now defensible with numbers instead of with reputation, and one that does not frees eight times the inference budget for something else.

If the debate loses that comparison, the money is available and the question becomes where to spend it. The answer is usually not another ensemble, because most failures in these pipelines are a specific and cheaply checkable thing, which is what the last chapter takes apart.