A model judge shares the generator's misreadings

The docs pipeline shipped with a grader. After the agent generated each of the 340 API reference pages, a second call scored it from 1 to 5 on accuracy and completeness against the source file, and anything below 4.0 was regenerated up to twice. The first full run produced a mean accuracy score of 4.3, 91% of pages at 4 or above, and 31 regenerations. It cost $6.40 and eleven minutes. Every dashboard the team had said the pipeline was working.

Six weeks later a customer's integration hammered a payment endpoint after a transient 502, and the postmortem traced it to this page:

retries (int, default 3)
    Number of times the client will retry the request after a failure.
    With the default, a failing request is attempted up to 4 times.

The client's retries parameter counts total attempts, not retries after the first, so the default produces three attempts and not four. The generator read for attempt in range(self.retries): and wrote down what the parameter is conventionally called rather than what this implementation does with it. The judge, given the same source file and the generated page, scored accuracy 5 and left a comment noting that the page "correctly documents the retry semantics including the total attempt count".

Two properties the judge does not have#

A model judge — a second model call handed the artifact and a rubric, returning a score for it — is stochastic, meaning the same input can produce a different verdict on a different call. Re-running the grader on that same page four more times produced scores of 5, 5, 4, and 5, and the variance is not a configuration mistake you can tune away — even at temperature 0, the setting that makes the model always take its highest-scoring next token, batching and kernel non-determinism move those scores enough to flip a borderline verdict, and any threshold turns a small numeric wobble into a different decision. Every safeguard in the retry loop from chapter 4 assumed a verdict that is a function of the tree; a grader that answers differently on Tuesday makes stall detection meaningless, because two identical trees now produce two different fingerprints.

The more damaging property is the one that is easy to miss, because it does not look like a defect in the judge at all. The judge is not independent of the generator. It reads the same source file, brings the same priors about what a parameter called retries means in the ecosystem, and is drawn to the same conventional reading when the implementation departs from convention. Its errors are therefore correlated with the generator's errors, and they are correlated most strongly on exactly the APIs whose behaviour is surprising — which are the APIs that most need documenting, and the ones where a wrong page does the most damage.

That is measurable rather than theoretical. Seeding 40 factual errors into generated pages and running the grader over them produced this split: of 26 surface errors — a renamed method, a wrong return type, a parameter that no longer exists — the judge flagged 24. Of 14 semantic errors, where the prose contradicted the code in a way that matched the conventional expectation, it flagged 4. A verifier that catches 92% of the easy class and 29% of the hard class is not a weaker version of a compiler; it is an instrument whose blind spot is aligned with the generator's.

Before trusting any grader of this kind there is a cheap calibration worth running, and it takes an afternoon. Have two people independently grade the same 100 pages against the rubric, then have the judge grade them, and compute agreement between each pair. On this pipeline the two humans agreed at a Cohen's kappa of 0.68 — kappa measures how often two graders agree after subtracting the agreement they would reach by chance alone, so 1.0 is perfect and 0 is no better than guessing — which is mediocre and workable, while the judge against either human came in at 0.31 — barely above what you get from a grader that always says 4. The number tells you two things at once: the rubric itself is fuzzier than its four decimal-place scores suggest, and the judge is not measuring the same thing the humans are measuring, whatever the correlation of the raw averages looks like. Teams routinely skip this step because the judge's scores look plausible in a spreadsheet, and a plausible-looking distribution of numbers is available from any process at all.

The human rubric from the previous chapter is slower, coarser and more expensive per verdict on every row of that table, and it beats the judge on the one row that determines whether a check is worth having. A designer or an engineer reading the page brings a different set of misunderstandings than the generator did, so when they are wrong they are usually wrong about something else. Independence is what makes a second look worth anything, and buying a second look from the same model family is closer to asking the author to proofread their own work than to a review.

The score changes what reviewers do#

There is a second-order effect that turns a merely weak check into a harmful one. Once a score column exists, reviewers use it, and they use it the way any human uses a confidence signal: they allocate attention away from the things marked safe. On this team's docs review, median time per page fell from 4.2 minutes before the scores were introduced to 1.6 minutes after, and the drop was steepest on pages scoring 5. Those are the pages where the judge and generator agreed, which is disproportionately where they agreed on a misreading. The system now routes the least human attention to the pages most likely to be wrong, and it does so through a mechanism nobody designed and everybody would recognise as reasonable if asked to justify each step.

The argument that keeps the judge alive in these discussions is an economic one, and it deserves a direct answer. Two cents and three seconds against twelve minutes of a person is a ratio that makes almost any comparison look silly, and the intuition behind it — some check must beat no check — is sound for checks whose failures are independent. It stops being sound once the failures are correlated with the generator's, because then the expected value of the check is not merely small, it can be negative: on the 40-page seeded audit, the judge would have caught 4 semantic errors and, through the attention effect, plausibly cost more than 4 by making the remaining 10 less likely to be noticed by a reader who saw a 5 next to them. A cheap check is only cheap if its false passes are free, and here they are the most expensive thing in the pipeline.

That is the precise sense in which manufacturing a verifier is worse than admitting you do not have one. Nine hundred pages with no scores get read at 4.2 minutes each by people who know they are the only line of defence. Nine hundred scored pages get read at 1.6 minutes each by people who believe something checked them.

Carve out the slice that has a real check#

The productive response is to stop trying to grade the whole artifact and to find the part of it with a mechanical oracle. In API documentation that part is the code samples, and the strong version of the idea is not to check the samples after generation but to source them from code that is already under test.

# payments/client.py — the sample lives with the implementation
def with_retries(self, retries: int = 3) -> "Client":
    """Configure the total number of attempts for each request.

    ``retries`` counts total attempts, so the default makes three.

    >>> c = Client(base_url="https://api.example.test").with_retries(2)
    >>> c.max_attempts
    2
    """

$ pytest --doctest-modules payments/ -q
............                                                      [100%]
12 passed in 1.9s

The >>> lines in that docstring are doctests, and --doctest-modules executes each one against the real implementation and fails when the printed result stops matching what the docstring claims. The docs build then extracts the docstring and its verified example into the reference page instead of asking a model to invent one. The sample is executed on every commit, so it cannot drift when with_retries changes, and the line of prose sitting next to it — counts total attempts — is now anchored to an assertion that fails if it stops being true.

Shipping only that slice is worth more than shipping graded whole pages, for three reasons that hold up under a product manager's questioning. Samples are the part readers copy into their own code, so an error there produces an outage rather than a misconception. The slice stays true without anyone re-checking it, because the check runs in CI alongside everything else, whereas a page graded once decays silently from the moment the code changes. And the boundary is legible: readers, and reviewers, can be told that signatures and examples are machine-verified and the surrounding prose is not, which puts human attention where it belongs instead of spreading it evenly over material of wildly different reliability.

Where a model in the loop is fine#

None of this makes model grading useless, and the distinction that matters is whether the model is the check or is producing candidate checks. A model reading client.py and proposing twelve assertions about its behaviour is a good use of a model; the assertions then run, deterministically, and the ones that fail are either bugs in the code or errors in the model's reading, and either way a human learns something concrete. That is the generator disposing of nothing and the compiler disposing of everything.

Model judges are also a reasonable triage instrument as long as they gate nothing. Ranking 340 pages by suspicion and sending the bottom 40 to a human first is a use where a stochastic, correlated signal still adds value, because the downside of a wrong ranking is a slightly worse queue order rather than a shipped error. The rule that keeps this honest is that the score must never be visible to the person doing the review, since the ranking's job is to schedule attention and the score's effect is to withdraw it.

The framing of the judge's task also matters more than the choice of model. Replacing "score this page's accuracy from 1 to 5" with "list every claim in this page that the source file contradicts, quoting the line" lifted the semantic catch rate in the seeded audit from 4 of 14 to 8 of 14, because an adversarial instruction fights the agreement bias rather than inviting it. Showing the judge only the source and one extracted claim at a time, without the surrounding page, helped again. Neither change makes it a gate, and a check that catches 8 of 14 while its misses cluster on your strangest APIs is a check you can use to prioritise and cannot use to accept.

The docs pipeline, rebuilt around the verified slice, produces fewer pages and a smaller number to report. That tension between what the system produces and what the system is worth is the last thing standing between a working loop and a programme that survives its second quarter.