Designing around an imperfect component

Read these first

• Architectural mitigations
• Uncertainty, confidence and abstention

What you will know

• Write an error budget for a feature and justify it against the cost of wrong outputs

  The threshold must distinguish harmless retries from irreversible, abusive, or regulated mistakes.

• Bound the blast radius of a wrong output

  A shared queue, account, or customer-facing response can turn one model defect into an outage.

• Make a consequential action reversible

  Undo may require idempotency keys, versioned state, or an approval hold; a disclaimer is not reversibility.

• Design where the human sits in the loop

  Low-confidence cases can route to review, while high-volume low-risk cases may need sampling rather than a person on every request.

• Trace how uncertainty reaches the user and implement a calibrated fallback or disclosure

  A confidence score that users interpret as probability can be more dangerous than a plain needs-review state.

Chapters

1. 1

   2,000 wrong outputs a month is the number, not 99% accuracy

   A feature that runs 200,000 times a month at 99% accuracy produces around 2,000 wrong outputs, and the only useful question is what one of those costs. 97% over 100,000 runs stays unanswerable until you know what a single error does, who absorbs it, and whether anyone notices. Two categories break that arithmetic rather than feeding it: a regulated error costs what it costs regardless of how many there are, since one is a reportable event, and an abusive case is adversarial, so its rate is chosen by an attacker rather than measured from your traffic. The error budget follows from cost and category, not from a percentage that sounds respectable.

   Read chapter
2. 2

   Errors cluster, so the aggregate can hide one customer's outage

   Three thousand errors in 100,000 runs can be three in every hundred for everybody, or every single request from one tenant, one language, one document layout or one long-tail entity type. The second is an account-level incident with a named customer on the call, and no budget computed on the aggregate can tell the two apart. Slice accuracy by segment, tenant, input type and language before accepting a headline number, and ask of a 97% feature which 3% and whether that 3% is stable. The answer also sizes the review queue later, since a queue built on the aggregate rate is the wrong size when errors arrive in bursts from one source.

   Read chapter
3. 3

   One defect multiplied by a shared queue is an outage

   A wrong output confined to a single request is an error; the same output applied as a template across a support queue, written to a shared account, or cached for every customer is an incident. Blast radius is a property of where the output lands, not of how often the model is wrong. Applying a generated auto-reply to a sampled 1% and checking the outcomes first is the cheapest bound available.

   Read chapter
4. 4

   A hold before commit is reversibility; a disclaimer is not

   Undo needs three things an interface cannot fake: a window in which the action has not yet committed, a cancel path someone can reach in time, and the prior state kept in a version you can restore. Some actions offer none of that. A message posted to a public channel cannot be unposted, so for those the guardrail has to move in front of the action rather than behind it.

   Read chapter
5. 5

   Retries duplicate side effects unless the key comes from the request

   An agent action that times out and gets retried will refund the customer twice, because the second attempt cannot tell whether the first one landed. Deriving an idempotency key from the originating request, instead of minting one per attempt, collapses those duplicates at the boundary that performs the effect. This is what makes a tool call safe to repeat, and it is a precondition for any automatic retry policy.

   Read chapter
6. 6

   Review capacity, not review policy, is the actual control

   Routing 3% of 50,000 daily items to humans means 1,500 reviews a day, and whether that counts as a control depends entirely on whether reviewers can clear them and still catch anything. Six months in, a 99.4% approval rate at four seconds median is not evidence of an improved model; it is a queue being rubber-stamped. Reviewer disagreement rates and seeded known-bad cases tell you which one you have.

   Read chapter
7. 7

   A needs-review state beats an uncalibrated 87% confident

   Printing "87% confident" beside a result invites the user to read it as a probability of correctness, which it generally is not, and the two-digit precision makes the miscalibration more persuasive rather than less. Coarse states the system can defend, such as verified against a source, unverified, or needs review, carry the same signal without the false promise. Then expect the objection, because it always arrives: the badge depresses conversion, the competitor who shows nothing looks stronger, product argues the model is usually right so the label is noise users will learn to ignore, and support argues that any visible hedge buys tickets. Arrive with the counter-argument and an honest price for the label, not just the design.

   Read chapter
8. 8

   Under pressure, drop the guardrail another one already covers

   An AI step that changes customer-visible pricing on 4% of orders at 98% accuracy needs a set of controls, and when the date moves you will keep some and lose others. Confidence thresholds, review queues, commit holds, caps on absolute price change and reconciliation against the previous price are not equally load-bearing, and two of them may guard the same failure. Decide which is redundant before the schedule decides for you.

   Read chapter