A critique with a location and a severity gets acted on

The ticket asked why a chargeback had cost the merchant $15.00 on top of the disputed amount, and the drafting system produced a five-paragraph answer that was correct in every particular: the fee, the 15-day evidence window, where to upload documents, and what happens if the issuer rules against them. The critique pass returned 180 words of prose, of which the operative part was the explanation of the evidence window is vague and the closing paragraph reads as abrupt. The revision that came back had changed three of the five paragraphs. Paragraph two, which nobody had complained about, now said the merchant would be charged "the standard dispute fee" instead of $15.00.

That system was running a reflection loop, also called a self-critique loop: the model is called three times over a single request, once to write a draft, once to read that draft back and say what is wrong with it, and once to rewrite the draft in light of those objections. Topic 4.8 covered the version that fits inside one prompt, where a model is asked to answer and then check itself within a single call; the loop is that idea unrolled into separate calls, so each step has its own prompt, its own output and its own record, and the cycle can run more than once. The reason anyone builds one is that a first draft is often close to right and wrong in ways that are easier to notice while reading than while writing — a vague sentence, a missing precondition, a closing paragraph that lands badly — so a second look catches some of it with no human in the path and no change to the model itself. The pattern turns up wherever a draft is cheap to produce and expensive to get wrong: support answers reviewed before an agent sees them, coding assistants that read their own patch and act on their own objections before showing a diff, summarisers that check a summary back against the document, and the reflection step that agent frameworks such as LangGraph ship as a built-in node. The chargeback answer went through exactly those three calls, and the revision is what the merchant would have read.

That failure is not exotic. Across 300 tickets replayed through the loop, 61% of revisions modified at least one section the critique had never mentioned, the median revision rewrote 340 characters when the critique had identified a single sentence, and 9.3% of the runs introduced a new factual error in a section that had been correct before the pass ran. There was also no way to answer the simplest audit question about the loop — which criticisms were acted on — because the critique was English and the revision was English and nothing connected them.

Those three figures describe one behaviour from three angles, and it helps to say what the behaviour is before naming a fix. The critique named one sentence; the revision moved 340 characters, which is several sentences; and in three revisions out of five the movement reached a section the critique had not mentioned at all. The last figure is the one that turns cost into damage. A revision that only rearranges untouched text wastes tokens, and a revision that rewrites untouched text has a chance of getting it wrong, which is what the 9.3% counts. The step between the second figure and the third is worth being explicit about: an edit that stays inside the sentence the critic named cannot damage a paragraph it never reaches, so the width of the edit is the mechanism and the new errors are the consequence.

The shape of the fix is familiar from static analysis. A linter that printed a paragraph about how your module could be tidier would be ignored; what makes a linter actionable is that every complaint arrives as a file, a line, a rule identifier and a severity, so the thing consuming it knows exactly what to change and the thing measuring it knows exactly what was fixed. A critic writing about a draft has the same job and benefits from the same discipline, and the rest of this chapter is that discipline applied to a three-call loop.

That comparison is an analogy, and it is worth saying where it stops before anyone leans on it. A linter's rule set is fixed and its output is deterministic: run it twice over an unchanged file and the second list is the first list, and a rule the code does not violate is never reported. A critic's findings are sampled from a model, so two runs over one draft give two different lists and neither of them is the list, and a finding can name a defect that is not there. The borrowed part is the format — a location, an identifier, a severity — and the format is what makes a finding actionable and countable. The reliability of the underlying judgement is not borrowed with it, and the last section of this chapter is about what that costs.

What the loop has to keep#

A generate-critique-revise loop is three model calls with a memory, and the memory is the part that is usually missing. The naive version threads everything through one growing conversation: the draft is an assistant turn, the critique is the next user turn, the revision is the next assistant turn, and at the end you take whatever is last. That structure makes every intermediate state a slice of a transcript rather than an object you can score, compare, store or return. Sequential chaining, from 11.2, already gives the alternative — each step is its own call with its own inputs and its own recorded output — and a critique loop is that chain with a cycle in it.

The reason a transcript is not enough is worth walking rather than asserting. Scoring a draft means passing that draft, on its own, to a scorer. A transcript holds the draft as a span of text with no boundary marked around it, so recovering the draft means finding where the assistant turn began and where it ended, and that boundary is a parsing problem that changes shape whenever the prompt changes. Comparing pass two against pass three means recovering both, which is the same problem twice. Returning something other than the last draft means having addressed the earlier one, which a transcript has no way to do. Each of those operations needs the draft as a value with an identity, and a conversation gives you a sequence of turns instead. Storing the drafts as separate objects is what makes all four operations trivial, and every measurement in this topic depends on at least one of them.

Every arrow into the ledger is a row you will want later. The draft at each pass is what lets you return something other than the last one; the findings at each pass are what let you compute how much of the critique the reviser actually absorbed; the model, temperature and prompt hash for each call are what let you tell, three weeks after a regression starts, whether the critic prompt changed under you. Storing the trajectory is cheap — a five-pass run over a 900-word answer is roughly 40 KB of JSON — and it is the difference between a loop you can debug and a loop you can only rerun.

Put that storage figure beside what it buys. A 900-word answer is a few kilobytes of text, and five passes hold five of them plus the findings and the per-call metadata, which is how the total reaches tens of kilobytes rather than hundreds. Against the cost of the model calls that produced it, the storage is not a line item at all. What it licenses is narrow and worth stating exactly: it licenses reconstructing any pass of any past run, which is what every diagnosis in the later chapters of this topic starts from. It does not licence reconstructing why the model produced what it produced, since the ledger holds inputs and outputs and the model's internal state is not an output. A ledger tells you that pass three rewrote a paragraph and pass four rewrote it back; it will not tell you why.

from dataclasses import dataclass, field

@dataclass(frozen=True)
class Finding:
    defect: str          # taxonomy label, e.g. "unsupported_claim"
    location: str        # verbatim quote from the draft it applies to
    severity: str        # "blocking" | "major" | "minor"
    detail: str          # what is wrong, in one sentence
    fix: str             # what the reviser should do about it

@dataclass
class Pass:
    index: int
    draft: str
    findings: list[Finding] = field(default_factory=list)
    resolved: list[bool] = field(default_factory=list)
    calls: int = 0
    cost_usd: float = 0.0

@dataclass
class Run:
    run_id: str
    prompt: str
    passes: list[Pass] = field(default_factory=list)
    critic_prompt_sha: str = ""
    model: str = ""

Three details in those declarations do work that is easy to skim past. Finding is frozen, so a finding recorded at pass two is the same object at pass four and cannot be edited to match whatever happened afterwards. Pass.resolved is a list of booleans parallel to findings, which is what makes "how many of this pass's findings were resolved" a count rather than a search. And Run.critic_prompt_sha records a hash of the critic prompt rather than the prompt itself, which is enough to detect that the prompt changed between two runs without storing a copy of it on every request.

A finding is a defect, a location and a severity#

The critic's output format decides how the reviser behaves, because a reviser given a paragraph of impressions has been handed a rewrite brief, while a reviser given four anchored findings has been handed four edits. Constrain the critic to emit a list of objects, each of which names the defect, quotes the span of the draft it applies to, rates severity, and says what to do. The quote is the load-bearing field: it converts "the closing is abrupt" into a character range the reviser is allowed to touch and the harness can verify.

Follow that conversion one step at a time, because the whole design rests on it. The quote is a substring of the draft. Searching the draft for that substring returns a start offset and an end offset. Those two offsets cut the draft into three pieces: the text before the span, the span itself, and the text after. Only the middle piece is in scope for this finding, so the two outer pieces are a prediction about what the revised draft must contain — byte for byte, in the same order. A prediction of that kind is checkable by comparison, with no model involved and no judgement required. The prose critique offered nothing to compare, because "the closing is abrupt" does not cut the draft anywhere.

{
  "findings": [
    {
      "defect": "unsupported_claim",
      "location": "Visa will usually side with the merchant in these cases",
      "severity": "blocking",
      "detail": "No source in the retrieved policy supports a claim about dispute outcomes.",
      "fix": "Remove the prediction and state that the issuer decides."
    },
    {
      "defect": "missing_precondition",
      "location": "you have 15 days to submit evidence",
      "severity": "major",
      "detail": "The 15-day window runs from the dispute notification, not from the transaction.",
      "fix": "Name the event the clock starts from."
    },
    {
      "defect": "tone",
      "location": "That is simply how the network rules work.",
      "severity": "minor",
      "detail": "Dismissive close on a ticket where the merchant lost money.",
      "fix": "Replace with a sentence offering the escalation path."
    }
  ]
}

The reviser then works one finding at a time, in severity order, and each call sees the current draft plus a single finding and returns the edited draft along with a status. Statuses are worth reading: applied means the reviser changed the anchored span, rejected means it disagreed and gave a reason, and not_found means the quote did not match the draft, which happens on around 7% of findings in practice — usually because the critic paraphrased slightly. Normalising whitespace and falling back to a fuzzy match with a similarity threshold recovers most of those; the ones that still fail should be dropped rather than passed through as free text, because a finding without a location is exactly the thing this design exists to eliminate.

That 7% is worth reading as a rate rather than as a nuisance. It says roughly one finding in fourteen was produced by a critic that had the draft in front of it and still did not copy from it exactly, which tells you the model treats the location field as a description of a span rather than as a transcription of one. Dropping the survivors after fuzzy matching costs a small number of real findings, and that is the trade being made deliberately: a dropped finding is a defect that reaches the reader, and a location-free finding accepted as free text is a licence to rewrite anything. The first failure is bounded by the drop rate and the second is not bounded at all.

Serialising the edits costs calls, so the arithmetic is worth having in front of you. Three findings applied one at a time is three reviser calls at roughly 1,100 input tokens and 950 output tokens each. Batching them into one call is cheaper and faster, and it is defensible if the prompt still names each finding explicitly and tells the reviser to leave everything outside the quoted spans byte-identical. What is not defensible is handing over the whole critique and asking for an improved version, because the model has then been asked to rewrite, and rewriting is what it will do.

The shape of that arithmetic matters more than the particular figures. Serial revision costs one call per finding, so its bill is linear in how much the critic found; batched revision costs one call whatever the critic found, so its bill is flat. The input is nearly the same either way, since every call carries the whole draft, and the output is nearly the same, since every call returns the whole draft. What serial buys for the extra calls is that each call sees one finding, so nothing else in the critique can influence the edit. What batching gives up is exactly that isolation, and the prompt's instruction to name each finding is an attempt to recover it by asking rather than by construction.

Two constraints in the reviser prompt do most of the work of keeping edits local. The first is an explicit instruction that text outside the quoted spans must be returned unchanged, which the harness then verifies by diffing the untouched regions and rejecting the revision if they moved; on the payments drafter that check fired on 8% of reviser calls and was cheap enough to run on all of them. The second is a length guard, because the reliable direction of drift is upward — a reviser asked to clarify one sentence will happily add two — and capping the revision at the original length plus 15% removes the slow inflation that otherwise turns a 240-word answer into a 600-word one over four passes.

The first of those two figures is the one to sit with, because it names a belief most people hold without examining it. The instruction to leave the untouched regions alone is in the prompt, it is unambiguous, and it is ignored on roughly one reviser call in twelve. Nothing about that is surprising once it is stated plainly: an instruction shifts what the model is likely to produce and does not constrain what it can produce. So the instruction is not the defence. The diff is the defence, and the instruction's job is to reduce how often the diff has to fire. A team that ships the instruction and skips the check has bought the appearance of locality, and its rejection rate will read as zero because nothing is measuring it.

The length guard hides a second decision of the same kind. Capping at the original length plus 15% and capping at the incoming draft's length plus 15% are different rules, and they are easy to confuse because they read almost identically. The first is an absolute ceiling that four passes cannot move. The second compounds, because each pass measures against a draft that already grew, and four rounds of fifteen percent is most of the way to a doubling — which is the inflation the guard was written to prevent, arriving through the guard. The cap has to name the original.

Measured over the same 300 replayed tickets, the two critic formats separate cleanly:

Read the first three rows as gaps and not as levels. The gap on the first row is the mechanism: five revisions in nine stopped reaching sections nobody had complained about. The gap on the second row is that mechanism measured in characters, and the two are the same fact — an edit confined to a quoted sentence is a small edit. The gap on the third row is the consequence, and it is the row that decides whether any of this was worth building. What the third row also says, and this is the part a summary of it would lose, is that the structured column is not zero. Anchoring the edit reduces the chance of damaging correct text by a factor of about four and does not remove it, because the anchored span itself can be edited badly and because a sentence can be made wrong by a change to the one before it.

The bottom two rows are the reason to do this even if the quality numbers had come out flat. A resolution rate is a number the loop produces about itself on every request, without labels and without a human, and it is the first instrument that tells you whether the critique step is doing anything at all. A pass where the critic emits four findings and the reviser resolves zero of them has burned two model calls; a pass where the critic emits nothing has burned one and is a signal in its own right. Both are invisible when the critique is prose.

Divide one of those two rows by the other and the loop's own verdict on itself falls out. A pass emits 3.4 findings on average and resolves 0.58 of them, so a little under one finding in six survives contact with the reviser. That ratio is worth stating carefully, because it is easy to over-read. It licenses one conclusion firmly: most of what the critic raises does not end up fixed, so a design that assumes the reviser clears the list is describing a loop nobody has measured. It does not license calling the other five sixths wasted, for two reasons. A finding the reviser rejects with a reason may have been wrong, and a finding recorded as unresolved at the last pass has simply never been checked, since resolution is only observable when a later pass looks again. Both means are also means over passes, so the ratio of the means is not the mean of the per-pass ratios, and a single pass that emits ten findings and resolves one drags the figure differently from ten passes that emit one each.

Whether a finding is resolved is checked by feeding the revised draft back to the critic on the next pass and looking for the same defect at the same location. A finding that reappears at pass three having been marked applied at pass two means the reviser changed the text without changing what the critic objects to, which is the loop's most common way of spinning in place. Matching a reappearance is fuzzy work, since the anchor quote will have changed by definition, so the practical rule is to match on the defect label plus the paragraph index, and to treat a same-label finding within the same paragraph as the same complaint until proven otherwise.

Where a well-formed finding is still wrong#

None of this makes the critique correct. The findings are still generated by a language model reading its own work, and on a 50-ticket sample where a domain expert labelled each finding as real or spurious, 18% of pass-one findings were objections to text that was already right — a confident, well-anchored, precisely located complaint about a correct sentence. Structured output improved the plumbing around the critique and left the accuracy of the critique exactly where it was. Topic 4.8 established why that ceiling exists for a single self-review prompt, and it does not lift because the review now arrives as JSON.

Nearly one pass-one finding in five, then, is a well-formed instruction to damage a correct sentence, and the anchoring that makes findings actionable makes those actionable too. This is where the linter comparison finally breaks in a way that costs something. A linter's false positive is a rule you can disable; a critic's false positive is indistinguishable in form from a true one, because both arrive as a defect label, a quotation and a severity, and the format carries no information about which is which.

Severity is self-reported and uncalibrated, so treat it as an ordering rather than a threshold. Sorting findings by severity is useful because it decides what gets fixed first under a budget. Writing a rule that says "auto-apply anything marked blocking" hands a model the authority to escalate its own opinion, and in the sample above, 4 of the 61 blocking-severity findings were spurious. Severity earns its place in the schema by shaping the reviser's attention and by giving you something to slice metrics on later, not by being a trustworthy score.

Those two rates are worth comparing, because the comparison is what supports the recommendation and what limits it. Four spurious findings out of sixty-one blocking ones is a rate well under the eighteen percent measured across pass-one findings, so severity is carrying some signal — the findings the model rates highest really are more often real. That licenses using severity to order the queue, which is the recommendation above. It does not licence auto-applying the top of the queue, because roughly one blocking finding in fifteen still objects to correct text, and auto-application is precisely the design that removes the reviser's opportunity to disagree. The residual on that comparison is the denominator: sixty-one findings from a fifty-ticket sample is a small count, and a rate estimated from four events moves a great deal when one of them is relabelled.

This is also not a patch protocol. There is a stricter design in which the critic emits unified diffs or JSON Patch operations and the harness applies them mechanically, and it is attractive because it removes the reviser call entirely. Models are unreliable at producing byte-exact context lines for a diff, so those pipelines spend their savings on retries and repair logic; quote-anchored findings with a model doing the edit is the version that survives contact with real drafts. If your artifact is structured — a JSON object, a config file, a table — the calculus flips, because an anchor there is a key path rather than a quotation and the patch applies deterministically.

The difference between those two cases is the reliability of the anchor rather than the quality of the model. A key path either resolves or it does not, and a model that emits a wrong key path fails loudly. A quotation has to match text the model is reproducing from memory of something in its own context, which is the operation that fails 7% of the time above, and a diff's context lines are that same operation carried out several lines at a time.

Nor does any of it bound the loop. Structured findings make each pass smaller and attributable, which reduces the blast radius of a bad revision without removing the possibility, and the ledger holds every draft without yet giving anyone a reason to prefer one over another. A loop built exactly as described here will still hand back the last draft it produced, whatever that draft is worth.

The loop now runs, keeps its intermediate state, and produces per-pass numbers about its own behaviour: findings emitted, findings resolved, findings re-raised, characters changed, calls and cost. Those numbers exist per pass, which means they can be plotted against the pass index over a fixed set of tickets, and the question of how many passes to run stops being a matter of taste. The shape that plot takes is consistent enough across tasks to be worth treating as a property of self-critique rather than a property of any one system.