Equal scores can hide two differently blind reviewers

3.71 against 3.66, with a confidence interval that contains zero. The recommendation written on the back of that run took one line: keep self-critique, since it scores the same, costs less to reason about and needs no second prompt to maintain. It is a reasonable-sounding inference from the only number anybody had looked at, and the per-class breakdown that the harness had also produced said something quite different.

The two arms are not close. They are differently blind, and the grader's five-point rubric happens to weight the classes in a way that nets them out to within a rounding error. Of the defects that either arm fixed, only 18% were fixed by both, so the overlap between what these two setups do is small enough that describing one as a cheaper version of the other is simply wrong about the mechanism.

Two of the numbers on that table can be divided into each other, and the answer is the sharpest thing in the chapter. Forty defects of difference across a hundred and twenty items is a third of a defect per review, and the grader prices that third of a defect at five hundredths of a point on a five-point scale. Since the interval around those five hundredths contains zero, the honest statement is stronger still: this rubric prices forty escaped defects at something it cannot distinguish from nothing. The grader is not broken. It is scoring how the review reads, and a wrong entitlement figure reads exactly like a right one to somebody without billing access.

Read the five rows as gaps rather than as levels and the shape of the trade appears. Four of the five rows point one way and one points the other, and the sizes are not similar. The separated arm leaves twenty-two fewer unsupported claims, sixteen fewer source contradictions and sixteen fewer constraint violations, and it leaves twelve more internal inconsistencies and two more arithmetic errors. Adding those five gaps gives the forty-defect difference in the totals row. So the arms differ by forty defects out of the hundred and sixty-odd either of them leaves standing, and the grader's single number moves by five hundredths of a point. Nothing is wrong with the grader; it is answering a question about how good the reviews read, and forty defects of these kinds are largely invisible to a reader who cannot check the sources.

Why the cheaper arm wins a row#

The internal_inconsistency row deserves explanation, because it reverses the direction of every argument made so far and it is not noise: 7 remaining against 19, over 31 labelled defects of that class, repeated across three runs.

Two things produce it. The self-critique pass has the writer's plan in context, and an internal inconsistency is precisely a divergence between what the plan intended and what different sections ended up asserting, so the material that makes reflection defensive about its recommendations makes it well-equipped to notice that section 3 assumed a $12,400 storage line while section 1 reported $9,800. The separated critic sees two numbers with no privileged account of which was meant, and it treats each claim against the sources rather than against the other claims.

That is the same property from chapter 1 read with its sign reversed, and it is worth saying so plainly. The plan in context is what makes reflection argue for its own recommendations, and it is also a statement of what the report was supposed to say. Against a defect where the intent is the thing being violated, having the intent in the window is an advantage rather than a liability. So the split is not an upgrade on every axis; it trades away a piece of context that was doing real work on one class in order to remove its influence on three others.

The second cause is downstream of the separated critic rather than inside it. When the critic flags an unsupported claim and the revision step rewrites that sentence, the rewrite frequently changes a figure or a framing that another section referred to, and nothing checks the rest of the document afterwards. Nine of the 19 remaining inconsistencies in the separated arm were not present in the first draft at all; revision introduced them. That is a property of a one-pass revise-what-was-flagged loop, and it means part of what looks like a weakness of separation is really a missing whole-document check after edits.

Take that nine out of the row and the two arms almost converge, which changes what the row is evidence for. The class has thirty-one labelled defects. The separated arm leaves nineteen standing, and nine of those were created by its own revision step, so it failed to detect ten of the original thirty-one and caught twenty-one. Self-critique left seven, so it caught twenty-four. The detection gap between the two arms on this class is therefore three items, and the visible gap is twelve. Nine of the twelve are creation rather than blindness. What that licenses is a redirection of effort: a project to make the separated critic better at spotting inconsistencies is aimed at three items, and a project to check the document after editing it is aimed at nine. The residual is that "not present in the first draft" was established by comparing the two documents, which cannot distinguish a contradiction the revision created from one it exposed by making a vague sentence specific.

A tie is a decision you have not made yet#

The tempting reading — same score, so take the cheaper one — is tempting for structural reasons rather than lazy ones. Aggregate scores are what dashboards carry, "no significant difference" is a familiar and comfortable conclusion, and when two options measure the same, cost is the textbook tiebreak. All of that is sound when the options are substitutes. These are not substitutes: choosing self-critique here is choosing to ship 31 unsupported claims and 22 source contradictions in order to avoid 12 internal inconsistencies.

That sentence states two of its three numbers as levels and the third as a gap, and both readings are worth having in front of you because they answer different questions. As levels, self-critique ships 31 unsupported claims and 22 source contradictions, which is what will actually reach customers. As gaps against the alternative, the choice costs twenty-two extra unsupported claims and sixteen extra source contradictions and saves twelve internal inconsistencies. The level is the right number for asking whether the system is fit to ship at all; the gap is the right number for choosing between the two arms, because the defects both arms leave standing are not on the table in this decision.

Stated that way the decision stops being about the score and becomes about consequence, which nobody had priced. Meridian's incident log had the numbers already.

The rows are not comparable in kind, which is the whole point of writing them out. Three of the five classes reach a customer as a false statement or a harmful recommendation; two of them are caught by a human who reads the review before sending and cost minutes. Meridian ships the separated critic on that basis, and the sentence written in the decision log is that the product can afford confusing reviews and cannot afford wrong ones. A product with a different shape — an internal analysis document read by an analyst who checks the sources anyway, where the deliverable's coherence is the deliverable — would read the same table and reasonably choose the other arm.

Notice what makes the two bottom rows cheap, because it is a fact about the workflow rather than about the defect. Both are caught by an account manager who reads the review before sending it. Take that human out — automate the send, or grow the volume until nobody reads them — and those two rows stop costing four minutes and start costing whatever a customer-visible contradiction costs. So the pricing table is a statement about the current process as much as about the defects, and it needs re-reading whenever the process changes. A decision log entry that records the reasoning rather than only the choice is what makes that re-reading possible.

Pricing defect classes this way has an obvious weakness, and it is better to name it than to present the table as more rigorous than it is. The costs in the right-hand column are estimates from a small incident log, they mix money with minutes with reputational damage that nobody can denominate, and two people at Meridian disagreed about whether an unsupported claim on an account call is worse than a wrong entitlement figure. The table's value is that it forces the disagreement into the open where it can be settled by whoever owns the product, rather than leaving it to be settled implicitly by a grading rubric written months earlier for a different purpose.

The general form is that aggregate parity between differently blind reviewers converts a measurement question into a product question, and the product question has to be answered explicitly. Inferring it from a tie means answering it by whichever failure class your grader happened to weight lower.

One more thing follows from the breakdown, which is where to look when a class is bad. A class where both arms leave many defects standing — arithmetic_error, at 12 and 14 — is not a case for or against separation at all; both arms are failing it for a shared reason, and the fix lives outside the choice being debated. Reading the breakdown column by column tells you which arm to ship; reading it row by row tells you which rows the debate cannot help with.

The measurement this chapter relies on#

Everything above rests on having per-class labels for the defects remaining in the final output, and it is worth being clear that this is the expensive artefact in the whole topic. Grading 240 final documents against five defect classes took two people the better part of two days. Nothing else produced the reversal on internal_inconsistency; the finding-level counts, the aggregate grade and the token bills were all consistent with the arms being minor variants of each other.

The 240 is one hundred and twenty items graded under both arms, which is the smallest set that can produce this table at all — dropping either arm leaves nothing to compare, and dropping items shrinks the per-class counts until a row of seven against nineteen becomes a row of two against five and stops being distinguishable from noise. That is the reason the labelling cannot be sampled down the way the precision measurement in chapter 4 could. Precision is one rate over one pool of findings; this is five rates over two arms, and each class needs enough items of its own.

The labels also age. They describe the arms as they were configured that week, and the revision-induced inconsistencies in particular are an artefact of a specific revision prompt, so a change to how revision handles a flagged span invalidates that row and no other. Knowing which row a change can move is what keeps the re-labelling bounded.

That property is worth generalising, because it is what makes a per-class table maintainable at all. Each row is produced by a mechanism, and each change touches a mechanism. A change to the critic's categories moves the rows whose defects those categories name. A change to retrieval moves the rows whose evidence lives in documents. A change to the revision prompt moves the rows revision can create. A per-class table therefore ages row by row rather than all at once, and a team that records which mechanism produced each row can re-label a fifth of the set after a change instead of all of it.

There is one objection to all of this that none of the measurements so far can answer, because every arm compared has been a writer plus some form of critic. A reviewer who thinks the critic is the wrong place to spend the calls entirely — that two calls to the same model to produce and then assess one draft is worse than three calls to produce three drafts and a cheap way to pick among them — is proposing something the harness has never run.