Deterministic gates and an LLM judge

A rubric is a written scale saying what each score level means, and a judge prompt is a rubric plus an instruction to apply it. The arrangement is genuinely useful: it grades properties that no regex can express — whether an explanation is faithful to the document it cites, whether a tone is appropriate, whether a refusal was warranted — and it produces a number that can be tracked across releases. Every check in the seven chapters before this one is deterministic and free by comparison, and the two are usually presented as alternatives.

They are not alternatives, they are stages, and the order they run in decides what a failing suite tells you. This chapter is about that ordering. A grader handed input outside its intended domain does not decline; it estimates, and the estimate reads as evidence. Putting the cheap, specific, deterministic checks ahead of it means a malformed output is reported as malformed rather than as unhelpful, which changes the failure from a research project into a one-line defect.

What a scalar score discards#

Here is a judge verdict on a case from a 300-case support suite, one of 92 that came back at 2 out of 5 on helpfulness in a single run:

score: 2
reasoning: The response begins to address the customer's question but
does not provide a complete resolution or actionable next step. It reads
as unfinished and would likely require follow-up.

The reasoning is consistent, articulate and accurate. Here is the raw output it was grading:

{"order_number": "ORD-448192", "status": "in_transit", "message": "Your
order left the Rotterdam depot on Tuesday and is currently with the loc

Every one of the 92 carried stop_reason: max_tokens. The information that would have settled it in ninety seconds existed at the moment each response arrived: stop_reason said max_tokens, and json.loads on the body would have raised Unterminated string starting at: line 1 column 89 (char 88). Both were discarded, because the first act of the evaluation was to hand the text to a grader whose output type is an integer between 0 and 5.

That compression is the actual failure, and it is worse than it looks for three separate reasons. A scalar score has no room for a cause, so every distinct failure mode — truncation, a wrong tool, a policy breach, an unhelpful but well-formed answer — arrives at the dashboard as the same low number, and the next move has to be a guess. A grader handed input outside its intended domain produces a fluent assessment of whatever it was given, which reads as evidence and functions as noise. And because the score is stochastic, the 92 cases were not even a stable set, so two consecutive runs disagreed on the membership of the group being investigated.

The first two of those reasons share one mechanism, and it is a property of the grader's output type rather than of its competence, which was never in question. The grader returns one of six values. Six values have no room to carry a cause, which is the first reason stated as arithmetic: four distinguishable failure modes and every degree of each have to be encoded in the same small integer, and any of them can arrive as the same 2. Nor is there a seventh value meaning not applicable, which is the second reason stated as a mechanism: an input outside the rubric's domain has no exit from the grader except as a number that looks exactly like a graded one. A parser does have that seventh outcome, because it raises. An instrument that can report the limits of its own competence and one that structurally cannot are different kinds of tool, and the ordering in the next section is built entirely on the difference.

The sentence of reasoning attached to the score looks like the missing cause and is not. Where the rubric asks for a score and then a justification, the justification is text generated after the score and conditioned on it, so it explains the number rather than reporting what was consulted to arrive at it. More decisively, the judge never saw stop_reason. It was handed the answer text and nothing else, so an account of that text as unfinished was the most it could possibly have said, and within its own domain the account was correct. Accuracy about a symptom is not evidence about a cause the instrument was never shown.

Money is the least interesting part of the waste, and the arithmetic shows why it never raises an alarm. Those 92 judge calls at roughly 1,800 input and 200 output tokens cost about $0.77 per suite run, which spread over 92 cases is well under a cent each — two thousand tokens of grading per case, at whatever the grading model's input and output rates happen to be. A figure that size does not appear as a line anybody questions in a monthly bill, so the cost of grading malformed output is never what brings the problem to light. The check that would have prevented it exists in every JSON parser ever written, costs nothing per case, and runs in microseconds.

Gates in order of cost and specificity#

An evaluation that prevents this runs cheap, specific, deterministic checks first, stops at the first failure, and reports that failure as the case's result. A judge runs only on outputs already established to be well-formed, on-budget, and correct in their side effects.

Two properties of that ordering do the work. Each gate is cheaper than the one after it, so a run spends money in proportion to how far a case gets, and a run where a third of the outputs are truncated costs a third less to grade. More importantly, each gate is a precondition for the next measurement being meaningful at all: a helpfulness score on a truncated response is not an expensive correct answer, it is a cheap wrong one, because the quantity the judge was asked to estimate is undefined for that input.

"Undefined" is doing real work in that sentence and is worth unpacking, because it is a stronger claim than "hard to estimate". Helpfulness, as the rubric defines it, is a property of a complete answer delivered to a customer. The truncated string was never delivered to anyone and never could be: the pipeline that produced it fails downstream at the JSON parse, so no customer is ever on the receiving end of it. There is therefore no fact of the matter about how helpful it was, and the rubric's own scale has no reading for half a JSON object — nothing in it says what a 4 would mean for a fragment. A grader asked for a number under those conditions still returns one, because the loop it runs in produces a continuation at every step and has no way to emit an absence. The score that comes back is not a bad estimate of a real quantity. It is a well-formed answer to a question that does not have one.

The implementation is a list and a loop, and the interesting part is what it returns:

GATES = [parses, schema_valid, required_fields, forbidden_strings,
         tool_calls_correct, within_budgets]

def evaluate(record, case):
    for gate in GATES:
        verdict = gate(record, case)
        if not verdict.passed:
            return Result(passed=False, stage=gate.__name__,
                          reason=verdict.reason, judge_score=None)
    score = judge(record, case)          # only reached by well-formed runs
    return Result(passed=score >= case.min_score, stage="judge",
                  reason=score.reasoning, judge_score=score.value)

stage is what turns a failing suite into a triage list. A report saying 92 cases failed at parses with reason unterminated object is a single defect with a single owner; the same 92 reported as "low helpfulness" is a research project. Grouping the failure summary by stage, and sorting by count, is a ten-line addition that changes how a team reads its own results.

Short-circuiting has one cost that is easy to overlook, which is that a gate failing on many cases hides everything behind it. If truncation takes out 92 cases, you learn nothing about their tool calls or their budgets until it is fixed, and a team that fixes gates one at a time over successive runs discovers its problems serially rather than all at once. The cheap mitigation is to report a funnel — how many cases reached each stage and how many cleared it — so the summary shows 300 in, 208 past parsing, 205 past schema, 199 past tool checks, and so on. Where the checks are all local and free, running every gate and reporting the first failure while recording the rest costs nothing and gives you the full picture in one run; short-circuiting matters only for the stage that charges per call.

The ordering is also not purely a cost ranking, and one placement is genuinely arguable. A case that busts its token budget still produced a well-formed, correct answer, so a judge score on it is a valid measurement, and some teams deliberately run the grader on budget failures to learn whether the extra tokens bought anything. That is a reasonable choice; what is not reasonable is running it on output that failed to parse, where the measurement is undefined. The principle is that a gate goes ahead of the judge when its failure invalidates the judge's measurement, and everything else is scheduling.

Properties that are not in the answer text#

The proposal to delete the deterministic checks arrives sooner or later, and it usually arrives well made: one judge prompt is a single artefact to maintain instead of sixty assertions, it catches semantic failures no regex can express, and it produces a number that leadership can read on a slide. Each of those claims has something in it. Where the argument fails is subsumption, and it fails in the direction that matters. Consider what is actually in front of a grader: the input, the answer text, and whatever context you paste into the prompt.

The bottom two rows are the ones worth dwelling on, because they are the cases where a judge can technically do the job and still should not. A grader asked whether a response conforms to a schema will be right most of the time and will occasionally pass a document with a missing required key, because it is estimating rather than parsing. At 97% agreement with ground truth — a good figure for a well-calibrated rubric — a 60-check suite flips roughly two verdicts per run in one direction or the other, and none of the flips are reproducible. A gate that gives a different answer on the same bytes cannot be a merge gate; it can only be a metric.

Ground truth in that sentence is a specific artefact rather than a figure of speech, and the whole of the 97% rests on it. It is a set of cases a human labeller has graded by hand, whose labels are treated as correct because there is nothing better available to treat as correct — not because a human grader is infallible. Agreement is then the fraction of those cases on which the model returned the same verdict the labeller did. Two conditions travel with that number and are usually left implicit. The labelled cases have to resemble the traffic the judge will actually grade, since agreement measured on clear-cut examples says nothing about the borderline ones where a grader's opinion is worth having. And the labels have to have been produced without the model's score in view, because a labeller shown a score agrees with it more often than one who is not.

Where that "roughly two" comes from is arithmetic worth doing, because the figure is smaller than people expect and still fatal. Agreement of 97% means that on a case whose correct verdict is known, the grader returns that verdict ninety-seven times in a hundred and the opposite one the other three. Sixty checks graded in a run, each disagreeing three times in a hundred, give an expected number of disagreements of sixty multiplied by three hundredths, which is a little under two. Multiplying a per-item probability by the number of items is legitimate here with no assumption whatever: the expected size of a total is the sum of the expected sizes of its parts, and that holds whether or not the parts have anything to do with one another. It is the one step in this chapter that needs no licence.

A different question does need independence, and it is the question a merge gate actually asks: how often does a run come back with no flipped verdicts in it at all? That is the chance the first check agrees, and the second, and so on through the sixtieth — ninety-seven in a hundred raised to the sixtieth power, if the sixty verdicts are separate draws that do not influence each other. It comes to about one run in six. Five runs in six therefore carry at least one flip, and because the flips land on different checks each time, a build that was red yesterday comes back green today with nothing in the repository having changed. A red result that a rerun clears does not get read as a defect. It gets read as weather — the fate the wall-clock threshold met in chapter 6, where a gate that fails for reasons the author cannot address gets switched off within a month.

Underneath both figures is the property that decides what a judge may be used for. A test is a function from an artefact to a verdict that returns the same verdict on the same artefact, and that is what licenses the inference everybody draws from a red build: the bytes changed, the verdict changed, so the change is the thing to look at. A judge breaks that inference at its source, because a judge is a model call and a model call samples a piece of text at every step of its own loop. The verdict it hands back is one draw from a distribution over verdicts, in exactly the way the output being graded was one draw from a distribution over outputs.

Two sources of variation are therefore stacked inside every judge score, and the number on the dashboard cannot separate them. The system under test writes a different answer on the next run; the grader awards a different grade to a fixed answer on the next run. When a score moves, either of the two could have moved it. One of them can be held still — grade a frozen record, as the fixtures of chapter 7 do, and the answer stops varying — and the other cannot, short of forcing the grader's picking step to its highest-scoring candidate, which narrows the spread without closing it and does nothing at all about the model version underneath the rubric being replaced next quarter.

The tempting repair is to ask the judge several times and average, on the reasoning that the noise cancels. It half works, and which half is the difference between a calibrated grader and a comfortable one. Averaging several draws does shrink the spread of the result: errors falling either side of the number the grader is centred on partly cancel when they are added together, and the more draws go into the average the closer it sits to that centre. What averaging cannot do is move the centre. If the rubric is worded so that a polite non-answer reads as helpful, every draw is generous, the errors all point the same way, nothing cancels, and the mean of a thousand runs is exactly as generous as one run was. The part of the error that is identical on every draw is the part repetition cannot reach.

Whether a given judge is noisy or lenient therefore has to be measured, and it can only be measured against labels the judge did not produce. The diagnostic is to split the disagreements by direction rather than to count them. A grader wrong in both directions in roughly equal numbers has spread and no lean, and more repeats do buy a steadier number. A grader whose disagreements point one way has a lean, more repeats buy nothing, and what it needs is a rewritten rubric or a moved threshold. An aggregate agreement figure hides which of the two you are holding, because both shapes of error produce the same 97%.

None of that makes the deterministic checks ahead of the judge more accurate, and the reason to trust them is a different one. json.loads over a byte string is a total function of those bytes: the same input raises the same exception on any machine at any hour, because no step in it samples anything. A deterministic check can certainly be wrong — a regex can be written badly — but it is wrong identically every time, which is a defect a single fixture from chapter 7 will pin. A stochastic check that is wrong two runs in a hundred cannot be pinned by any fixture at all, since the fixture that catches it today accepts it tomorrow. That difference, rather than accuracy, is what puts one class of check on the merge gate and the other on a dashboard.

The forbidden-literal check has an additional property that has nothing to do with accuracy. When legal asks whether the assistant ever used a banned phrase, the answer needs to be a deterministic function of the output with a log you can show an auditor, and "a language model assessed 240 responses and found no violations" is a materially weaker statement than "the literal string does not occur in any of 240 responses, here is the grep". The determinism is the deliverable.

The two maintenance bills#

The maintenance objection deserves a straight answer rather than a defensive one, because it is the argument that actually decides these things. Sixty checks are not sixty artefacts if they are built the way chapter 4 built them — one generic checker plus per-case data, with a startup contract validation that fails once and loudly when a signature moves. The recurring cost is a YAML block per case, most of which is the case's expected values, which somebody has to write regardless of how the grading is done.

And a judge is not one artefact either. It is a prompt, a rubric with defined score levels, a pinned model version that will be deprecated on a schedule, a set of human-labelled examples to calibrate against, and a periodic re-measurement of agreement, because a judge whose agreement with human labels is unknown is producing a number rather than a measurement. Set the two bills side by side honestly and they are comparable, and only one of them can tell you that amount_cents was 42.

Keeping both instruments is what that list is really arguing for, and the item teams drop first is the last one on it. The 97% quoted earlier is not a property of judges in general. It is a measurement somebody took, of one rubric against one set of human labels, on one day. Every input to it moves afterwards: the model behind the judge is deprecated and replaced on the provider's schedule, the rubric is edited to settle an argument, and the traffic drifts towards cases the labelled set never covered. None of those events changes the number on the dashboard, because nothing recomputes agreement unless somebody has scheduled it. A judge that has drifted does not announce it — it goes on emitting scores in the same range as before — which is why the periodic re-measurement, and not the prompt, is where the maintenance cost of a grader actually sits.

The other half of the objection is true and worth conceding. Assertions do rot: 271 of 312 in the audit of chapter 7 had never rejected anything, and a proportion of those were incapable of it. That is an argument for fixtures and for a quarterly cull, not for replacing a deterministic check with a stochastic one, since a rotted assertion and a stochastic verdict fail differently — the first is silent and the second is loud and irreproducible, and neither is improved by the existence of the other.

What the ordering buys is that every measurement is taken on an input for which it is defined. Parsing, schema conformance, required fields, forbidden literals, tool identity and arguments, token budgets and completion length are all properties of artefacts you already have, cost nothing, and return the same verdict on the same bytes every time; they belong at the front, blocking the merge, reported by stage. A rubric-driven grader is the only practical instrument for the properties none of them can express, and it belongs behind them, scoring what survived, tracked as an aggregate metric rather than used as a per-case veto.

None of this makes the deterministic layer sufficient. It is a floor: it establishes that the output is parseable, complete, structurally valid, within budget, and correct in the actions it took. Whether the answer is any good is a separate question, and it is the question the next topic takes up — how to build a grader that agrees with humans, how to measure that agreement, and what to do when it drifts.

What you can do this week is smaller and entirely mechanical. Take one suite, add the run record if it is not there, put the gates in front of whatever grading you already do, and group the failure report by stage. The first run usually reclassifies a surprising fraction of what the team had been calling a quality problem.