Why code is the highest-leverage agent domain

Read these first

• What an agent is
• Verifier-first design

What you will know

• Run a representative code change through the compiler, type checker, linter, and test suite, and map each failure to the defect it exposes.

  These checks cover different failure classes, and passing them still does not prove behavior is correct or requirements are complete.

• Build a minimal agent loop that generates a patch, invokes deterministic checks, and retries from their diagnostics.

  The loop needs isolated execution, bounded retries, and safeguards against stale, incomplete, or misleading feedback.

• Inject defects that pass one verifier but fail another, then trace why the loop accepted or rejected the patch.

  A green test suite can coexist with type errors, style violations, security flaws, or untested edge cases.

• Compare the same agent workflow with an unverifiable task using a stated rubric for correctness, feedback quality, and recovery.

  Subjective tasks force humans to supply the missing oracle, making failures harder to localize and iteration less predictable.

• Measure how verifier coverage changes useful patch rate, iteration cost, and human review burden on a fixed task set.

  The commercial advantage depends on reduced review and debugging effort, not merely on producing more candidate code.

Chapters

1. 1

   Compilers and suites are verifiers nobody on your team built

   Explaining to a product manager why the first agent investment is a coding agent rather than a release-notes writer comes down to one property: for each individual output a cheap, deterministic check already exists, so most generation errors surface as observable failures in seconds instead of review opinions. This chapter pins that property down precisely, and separates it from weaker claims like "code is structured" or "engineers adopt tools quickly".

   Read chapter
2. 2

   Four checks catching four disjoint failure classes

   An agent ships get_entry annotated as returning dict, which returns None on an expired key down a branch no test walks. Ruff was green, black was green, pytest was green — and a type checker would have rejected it without running anything. Run a real change through compiler, type checker, linter and suite, and map each failure to the defect class it exposes, so you stop treating these as overlapping samples of one quantity.

   Read chapter
3. 3

   Compiling, typing, linting and passing is still not correctness

   The combination establishes that the code is well formed, internally consistent, stylistically conformant and behaves as the existing assertions expect. It says nothing about whether the change does what the ticket asked. Practise seeding defects that clear one verifier and fail another, and defects that clear all four, then decide what to do about the third bucket when no mechanical check can be added for it.

   Read chapter
4. 4

   A retry loop is only as honest as its diagnostics

   An agent burns its whole retry budget on a generated protobuf module: the same three errors repeat verbatim, the diffs get stranger, and a human's clean build compiles fine. The incremental daemon's cache was keyed on inputs it did not know had changed, so every attempt patched a failure that no longer existed. Build the minimal generate-check-retry loop, then cover isolated execution, bounded retries and detecting stale or misleading feedback.

   Read chapter
5. 5

   Cheap checks go inside the loop, slow ones at its edge

   An eight-second tsc --noEmit and an eleven-minute Jest suite are not competing for the same slot. The fast, broad-but-shallow check belongs on every attempt because it is the only one an agent can afford to consult repeatedly; the expensive behavioural one runs once at the end because paying eleven minutes per retry destroys the loop. Covers why running both in both places is the wrong resolution of that tension. Deciding a verifier's place is also something you can measure rather than argue: hold the task set fixed, add or remove one check from the loop, and record useful patch rate, attempts per task and reviewer minutes — which is the experiment that answers a teammate who wants the type check deleted because the suite already rejects things.

   Read chapter
6. 6

   Without a mechanical oracle the human grades in batches

   Run the same harness on fixing failing tests and on rewriting user-facing error strings against a designer's rubric, and acceptance rates can match while the loops differ completely. A diagnostic names a file and a line, so rejection attributes to one edit and the next attempt is informed; a rubric score arrives late over eight changed strings at once, and iteration becomes guesswork. Applies the same test to a docs build that only compiles code samples.

   Read chapter
7. 7

   A model judge shares the generator's misreadings

   The obvious move when a domain has no mechanical oracle is to score each output with a strong model and regenerate anything below threshold, and it counterfeits the exact property that made code attractive. A judge is stochastic rather than deterministic, and worse, it is not independent: it reads the same source the generator read and misreads it in the same places, so the check is most likely to pass precisely where the generation went wrong. A batched human rubric is slow and coarse and at least fails independently. A passing score also licenses reviewers to read less carefully, which makes the wrong page more likely to ship. The productive response is to carve out the slice that does have a real check — extract the code samples from the source and compile them — and ship that rather than a graded whole.

   Read chapter
8. 8

   Merged patch count is the wrong number to celebrate

   Throughput up 3.1x with a 94% check-pass rate, and senior engineers reporting their weeks got worse, is not a contradiction — it is what happens when the constraint was review capacity and you scaled generation. Measure engineer minutes per merged change, review plus rework plus incident time, on comparable work before and after. Includes changes worth making that would lower the patch count outright.

   Read chapter