Reflection / self-critique loop

Read these first

• Sequential chaining
• Reflection and self-critique in a single prompt, and its limits

What you will know

• Build a generate-critique-revise loop

  Preserve each intermediate artifact and make the revision consume a structured critique rather than an untracked second prompt.

• Show it improving output, then show it plateauing

  Use a fixed evaluation set and report both quality gain and added calls, since later passes often add latency without meaningful progress.

• Explain why a model is weak at finding its own errors

  The same weights that made the mistake are judging it, which limits how much it can catch.

• Bound the loop so it cannot degrade the output

  Require an acceptance gate against the prior draft, with a maximum iteration or budget limit and a safe fallback to the best version.

• Detect a critique loop making things worse

  Track per-iteration scores, reversions, and disagreement patterns so regressions trigger an alert or stop condition rather than another rewrite.

Chapters

1. 1

   A critique with a location and a severity gets acted on

   Generate, critique, revise — the mechanics matter more than the slogan. Keep every intermediate draft, and have the critic emit findings carrying a defect, a location and a severity rather than free-form prose, so the reviser fixes one thing at a time instead of rewriting sections nobody complained about. That structure also turns the fraction of findings resolved into a number you can measure.

   Read chapter
2. 2

   The same weights that wrote it are the ones judging it

   Score a fixed set after each pass and the gain arrives in the first two, then stops however many passes you allow. The reason is mechanical: the critique comes from the weights that produced the draft, so errors invisible during writing are largely invisible during judging. A richer critic prompt with defect categories and worked examples raises the plateau's height without moving where it sits.

   Read chapter
3. 3

   An acceptance signal has to come from outside the weights

   The gate in the next chapter keeps a revision only if it scores better, and the chapter you just read argued that the writer's own judgement is exactly the signal you cannot trust for that. Here you enumerate what can actually score a draft against its predecessor — an objective check or test suite where one exists, tool or retrieval output the critic can verify against, a rubric-scored judge that is not the writer, a heuristic on the artifact, a human — and what a self-scored gate does and does not guarantee. The same move is what extends the useful number of passes, since only a vantage point outside the weights sees the errors the model cannot. It also separates the offline evaluation set that produced the per-pass curve from the runtime gate that must make the same call per request, without labels.

   Read chapter
4. 4

   Return the best draft the loop saw, not the last one

   A loop scoring 62, 71, 73, 72, 70 across five passes and returning the fifth draft is handing back something worse than it already had at pass three. Lowering the cap does not fix it, because different drafts peak at different passes. Add the acceptance gate — keep the prior draft unless the revision scores better against the same check — plus an iteration or budget limit and a safe fallback.

   Read chapter
5. 5

   A flat mean hides a loop that helps half and harms half

   A weekly aggregate flat against the single-pass baseline is exactly what a loop that improves half its cases and damages the other half produces, which is why four escalations about answers contradicting themselves went unconnected for a month. Instrument per-case, per-iteration scores and the sign of each delta so gains and regressions are visible separately, and wire a stop condition to the regressions.

   Read chapter
6. 6

   Three passes or three samples: the same budget, different bets

   Four points above single-pass at 3.1x the calls is not yet a decision, because the same budget buys three independent samples with a selection step. Where errors come from sampling variance rather than misunderstanding, independent attempts routinely beat serial polishing of one attempt. This chapter covers the per-pass curve, the alternative arm, and what belongs in a write-up beyond the two headline numbers.

   Read chapter