Dry runs and confirmation

The gap between an agent deciding on an action and the action happening is where this topic's controls belong. A tool call is the model emitting a structured request — a tool name and a set of arguments, such as delete_stale_branches(repo, older_than_days, …) — that your own code then executes against a real system, and tool execution safety is the set of controls placed in the path of that execution: what a call is permitted to touch, how long it may run, whether a person has to authorise it before it takes effect, and what record it leaves behind afterwards. Those controls exist because the actor choosing the arguments derives them from text, some of which arrived from a retrieved document, a web page or a ticket comment that somebody outside your organisation wrote, so an ordinary-looking tool call is an operation whose parameters an outsider may have influenced. In practice they live in the tool server, meaning the code sitting between the agent's loop and the repository, database or payment API it acts on, which is the same layer where a backend service already carries authentication, request validation and rate limiting.

This chapter is about the control that stands between proposing a write and performing it, and about the two things that get merged under one name. A dry run is a property of an execution: the call computes the full effect of an operation and reports it without applying it. A confirmation is a property of a decision: some actor other than the one proposing the action inspects a specific, pinned effect and authorises that effect before it is applied. They are frequently discussed together because a confirmation is close to useless without a preview to confirm, but they solve different problems, and a design can have one without the other.

Two different things wearing the same name#

Here is a tool that has a dry run and no confirmation. delete_stale_branches(repo, older_than_days, protected_patterns, dry_run) takes a boolean with dry_run defaulting to true, and a docstring instructing the model to always preview first and show the output to the user before deleting. A repository-hygiene agent built on it deleted sixty-three branches on a Thursday afternoon, having shown a preview of forty-one, and the engineer on duty had done exactly what the process asked: he read the list of forty-one branches, none of which looked alarming, and typed "yes, go ahead."

The two calls did not match. The preview had been issued with older_than_days: 90; the deletion went out with older_than_days: 60. Nothing anywhere in the stack compared the arguments of the second call against the arguments of the first, because from the tool server's point of view they were two independent invocations that happened to arrive from the same session eleven seconds apart. The system had a preview and it had a human saying yes, and it still had no mechanism that made the yes apply to anything in particular.

The two counts account for each other exactly, and that is what makes the incident readable rather than mysterious. Forty-one branches were shown, sixty-three were deleted, and the difference is the twenty-two nobody saw. Every one of those twenty-two is explained by the cutoff alone. A branch is stale when its last commit is older than the cutoff, so lowering the cutoff can only enlarge the set: a branch that was stale at ninety days is still stale at sixty, and a month's worth of additional branches joins it. The previewed list is therefore contained in the deleted list, with nothing in it that the deletion left alone. No second fault is needed to close the arithmetic — no race, no misparsed argument, no bug in the deletion path. One argument took a different value on the second call, and the two lists reconcile with no residual row. What that licenses is the sharp version of the reviewer's position: he checked a subset and was silent about the rest, and there was no signal anywhere in the preview that a rest existed.

What it did not have was a confirmation, in the sense that mattered, because the actor deciding whether to preview and the actor deciding what to execute were the same actor. When dry_run is a parameter in the tool schema, it is an argument the model fills in, exactly like older_than_days. Every reason the model has for choosing older_than_days: 60 over 90 — an instruction in the system prompt, a sentence from the user, a line of text in a file it just read, a comment inside a pull request description — is a reason it can have for choosing dry_run: false on the first attempt. The docstring saying "always preview first" is a strong prior, not an enforcement point, and priors are precisely what prompt injection is in the business of overriding.

The default value is the other half of the same illusion, and it fails one step earlier. A schema default is a rule about absence: it says what the server should assume when a field does not appear in the call. Emitting fields is exactly what the model does, and a model that has decided to delete emits dry_run: false rather than omitting the field, so the default is never consulted. dry_run defaulting to true therefore protects against a model that forgets to mention the flag, which is not a failure anybody was worried about, and it is silent about a model that mentions it. The safe-looking half of the schema is doing work on a case that does not arise.

The docstring is worth examining as a defence in its own right, because the shape of its failure recurs through the rest of this topic. "Always preview first" is one fixed string, attached to the tool definition, sent identically on every call. It read the same on the call that previewed and on the call that deleted. A defence whose job is to tell two cases apart has to vary with something that differs between them, and this one varies with nothing at all: between the two calls the conversation had grown, the cutoff had changed and the intent had changed, while the docstring was byte-for-byte what it had been eleven seconds earlier. That is the general form of the mistake to carry forward. A control whose value is constant across the cases it exists to separate is not separating them, however true its contents are.

This is worth stating plainly because the shallow version of the design is genuinely tempting: it looks like confirm-before-write, it demonstrates well, and in the happy path a well-behaved model does preview first and does show the output. The failure mode is not that the model is stupid. It is that the safety property depends on the model's cooperation, and the entire premise of the threat model developed in topic 7.4, on architectural mitigations, is that the model's decisions are attacker-influenced. A control the model chooses to invoke is a convention with a flag attached.

Separating them means the preview and the commit stop being the same operation with a boolean, and become two operations with different shapes:

# Before: one tool, one boolean, one actor deciding everything.
{
  "name": "delete_stale_branches",
  "parameters": {
    "repo": "string",
    "older_than_days": "integer",
    "protected_patterns": "array<string>",
    "dry_run": "boolean"   # the model fills this in
  }
}

# After: the model can only ever produce a proposal.
{
  "name": "propose_delete_stale_branches",
  "parameters": {
    "repo": "string",
    "older_than_days": "integer",
    "protected_patterns": "array<string>"
  }
}
# and the commit is not addressable by arguments at all
{
  "name": "commit_proposal",
  "parameters": { "proposal_id": "string" }
}

The second tool takes no operational parameters. There is no older_than_days to re-derive, no repo to substitute, nothing for an injected instruction to widen. The only thing the model can say is which previously computed plan to execute, and that plan is held on the server side, where it was produced.

Counting the fields is the whole of the argument. The original schema offers the model four places to write a value: a repository, a cutoff, a pattern list and a boolean. Each of those is derived from the model's context, and that context contains text somebody outside your organisation wrote, so each is a place an injected instruction can push. The committing call in the second schema offers one place, and what goes into it is not a description of an operation but a reference to one the server has already written down and cannot be talked out of. Drive the operational fields on the committing call to zero and there is nothing left for the second call to disagree with the first about. Note what that removes rather than adds: the design does not detect a discrepancy between the two calls, it deletes the channel through which a discrepancy could be expressed. The only shape of delete_stale_branches(...) the model can still emit is a proposal.

Pinning the plan so the yes means something#

Arguments drifting between the preview and the commit is only one of the two things that can change in that gap. The other is the world. Even if the model replays older_than_days: 90 byte-for-byte, the repository it is describing has moved on: someone pushed to feature/checkout-v2 at 14:11, which takes it out of the stale set, and continuous integration created eleven ci/retry-* branches that were not in the preview and now qualify. This is the ordinary time-of-check-to-time-of-use gap, and it is worse for agent tools than for a shell script because the interval is not milliseconds. It is however long the model takes to render the preview, however long the human takes to read it, and however many other tool calls the agent makes in between, which in the branch-deletion incident was eleven seconds and in a task where a person goes to lunch mid-approval is forty minutes.

The proposal is the unit that gets approved, so it has to be a real object with an identity, a body, an expiry and a fingerprint. When propose_delete_stale_branches runs, the tool server resolves the arguments against the live repository, produces the concrete list of refs it intends to delete, stores that list, and returns an identifier along with a digest over the resolved plan. Approval attaches to the identifier. Execution re-resolves the plan and refuses if what it would do now differs from what was approved.

import hashlib, json, time, uuid

PROPOSALS = {}  # in practice a table with a TTL, not a dict

def propose_delete_stale_branches(repo, older_than_days, protected_patterns, principal):
    refs = resolve_stale_refs(repo, older_than_days, protected_patterns)
    plan = {
        "op": "delete_refs",
        "repo": repo,
        "refs": [{"name": r.name, "sha": r.sha, "last_commit": r.committed_at.isoformat()}
                 for r in sorted(refs, key=lambda r: r.name)],
    }
    digest = hashlib.sha256(json.dumps(plan, sort_keys=True).encode()).hexdigest()
    pid = f"prop_{uuid.uuid4().hex[:12]}"
    PROPOSALS[pid] = {
        "plan": plan, "digest": digest, "proposed_by": principal,
        "expires_at": time.time() + 900, "approved_by": None,
    }
    return {"proposal_id": pid, "plan_digest": digest[:12], "plan": plan}

def commit_proposal(proposal_id, principal):
    p = PROPOSALS.get(proposal_id)
    if p is None:
        raise ToolError("unknown_proposal")
    if time.time() > p["expires_at"]:
        raise ToolError("proposal_expired")          # re-propose, re-approve
    if p["approved_by"] is None:
        raise ToolError("not_approved")
    if p["approved_by"] == p["proposed_by"]:
        raise ToolError("self_approval")             # the agent is never the approver
    fresh = replan(p["plan"])
    if fresh["digest"] != p["digest"]:
        raise ToolError("plan_drifted", diff=diff_plans(p["plan"], fresh["plan"]))
    return execute(p["plan"], approval=p)

Three of those checks earn their place immediately. The expiry exists because a plan resolved against a repository at 14:02 is a statement about the repository at 14:02, and the older it gets the less it describes. Fifteen minutes is a defensible default for branch deletion and would be far too long for a plan over a pricing table that a merchandiser is editing. The self-approval check exists because the approval has to come from a different actor, and the cheapest way for that requirement to quietly evaporate is for the agent's own service principal to be able to call the approve endpoint. The re-plan comparison exists because approval was given against a specific set of refs, and if the set has changed the approval no longer covers it.

Two of those three are requirements and one is a decision, and knowing which is which is what stops the wrong one being tuned. The self-approval check and the digest comparison are requirements: remove either and the property the design exists for is gone. Without the first, the agent's own service principal can approve the agent's own proposal, and the second actor the whole design turns on has quietly become the first one. Without the second, an approval given over forty-one refs authorises whatever the set has since become, which is the original incident with an identifier in front of it. The expiry is a design decision. Nothing in the mechanism breaks a minute after the quarter-hour is up; what the number sets is how much of the world's movement an approval is permitted to span. That makes it a property of the resource rather than of the approver — a branch list drifts slowly, a pricing table a merchandiser is editing drifts by the minute — and the correct response to an approver who is too slow for it is proposal_expired and a re-propose, not a longer window.

The flow through those three actors is easier to see drawn than described:

Infrastructure tooling arrived at the same shape a decade earlier and for the same reason. terraform plan -out=tfplan produces a plan file rather than a printed summary, terraform apply tfplan executes that file rather than re-deriving intent from the configuration, and the apply refuses when the recorded state no longer matches what the plan was built against. The plan file is the proposal, the state comparison is the digest check, and the reason both exist is that the industry spent years discovering that a printed preview followed by a separate command re-reading mutable inputs is a preview of something other than what runs. Agent tools reproduce that gap and add an actor whose choice of arguments is influenced by untrusted text, which makes the pinned-plan design more necessary rather than less.

The parallel is exact on the mechanism and stops at the actor, and the place it stops is the place a reader is most likely to over-claim. Terraform's state comparison protects an apply against the world having moved since the plan was built, which is precisely what the digest check does. What Terraform never had to survive is an intent that re-derives itself. Its configuration is a file somebody wrote and then left alone, so between plan and apply the inputs can move and the request cannot. An agent's cutoff is recomputed from a conversation that has grown since the preview, so both can move. The pinned plan therefore imports all of Terraform's protection against the first kind of drift and none of a protection against the second, because there was never one there to import; the protection against the second is the separate fact that the committing call has no arguments to re-derive.

Note what the agent cannot do in that diagram. It cannot approve, it cannot commit an unapproved proposal, and it cannot commit a different plan than the one shown, because the only handle it holds is an opaque identifier that the server binds to a fixed body.

What a preview has to contain to be worth confirming#

Pinning the plan fixes who decides. It does not by itself fix whether the deciding was informed, and this is where a large share of otherwise well-built confirmation flows fail. Consider a bulk_update_prices tool whose dry run produces this:

Preview: will update prices for 1,240 products in the Summer category.
Proceed? [y/N]

A reviewer looking at that has been asked to authorise something they cannot see. The sentence is a summary of the plan rather than the plan, and every property a reviewer would need in order to catch a mistake has been summarised away. Which stock-keeping units (SKUs), specifically — is the Summer category the 1,240 items they think it is, or does it include the clearance range that was folded into it last quarter? From what price to what price — is this the intended 15% seasonal reduction, or is it 15% of the original list price applied on top of an existing markdown, taking a €89 jacket to €31? Is 1,240 the count the reviewer expected, and if they have no idea what to expect, the number is not evidence of anything. Approving that text is a ritual, and rituals are worse than nothing because they produce an audit trail showing that a human reviewed a change they never saw.

A preview is adequate when it lets a reviewer catch the specific mistakes that this operation can make. That gives three concrete requirements. It must carry the resolved arguments rather than the ones the model proposed, so that category: "Summer" has been turned into the actual member list and any filter that expanded unexpectedly is visible. It must carry before-and-after values for the fields being written, at the granularity the reviewer reasons in, with extremes surfaced rather than buried in the middle of a long list. And it must carry the side effects, because a price write in most commerce systems is not a price write.

{
  "proposal_id": "prop_8b31f0c4a2d7",
  "op": "bulk_update_prices",
  "resolved_from": {"category": "Summer", "rule": "list_price * 0.85"},
  "affected": 1240,
  "sample": [
    {"sku": "SUM-4410-BLK", "name": "Linen Jacket", "was": 89.00, "now": 75.65, "currency": "EUR"},
    {"sku": "SUM-9902-NVY", "name": "Beach Towel",  "was":  19.00, "now": 16.15, "currency": "EUR"}
  ],
  "extremes": {
    "largest_drop_pct": {"sku": "SUM-1180-RED", "was": 240.00, "now": 26.40, "drop_pct": 89.0},
    "increases": 0,
    "below_cost": [{"sku": "SUM-1180-RED", "unit_cost": 61.20, "now": 26.40}]
  },
  "side_effects": [
    {"kind": "webhook", "target": "search-index", "count": 1240},
    {"kind": "webhook", "target": "partner-feed.google", "count": 1240, "note": "propagates in ~4h, not revertible from here"},
    {"kind": "email", "template": "price_drop_wishlist", "recipients": 8934},
    {"kind": "recompute", "target": "open_carts", "count": 217}
  ],
  "expires_at": "2026-03-11T14:17:22Z"
}

That preview does work the sentence could not. The extremes block shows one SKU that had already been marked down once, so the 15% rule compounded and put it 57% below unit cost, which is the kind of thing a merchandiser spots in two seconds and a summary hides completely. The side_effects block shows that saying yes also sends 8,934 emails announcing the price drop and pushes the prices into a partner feed that will not be pulled back by an internal rollback. A reviewer who understood only "1,240 products in Summer" would have approved all of that without knowing it was in the request.

Two rows in that block can be checked against the rule printed a few lines above them, and doing the check is what a merchandiser's two seconds actually consist of. The linen jacket's before and after stand in the ratio the rule states: take fifteen per cent off eighty-nine euros and seventy-five sixty-five is what is left, so that row is the rule behaving. The red item is not. Its new price is a little over a tenth of its old one, which is not a fifteen per cent reduction by any reading, and the drop_pct field on the same line agrees with that arithmetic rather than with the rule. Then set the new price against the unit cost on the row below it. The gap between them is a bit more than half the cost, which is the fifty-seven per cent figure, and it is the one number on the page that needs no interpretation at all: the company would be paying customers to take the item away. The reviewer never had to work out how the compounding happened. Two independent quantities on the same row — a price computed from a rule and a cost recorded by purchasing — disagreed, and either one of them alone would have looked unremarkable.

The side_effects rows repay reading as ratios rather than as a list. Two of them carry exactly the affected count, one webhook per product, which says their size is fixed by the size of the operation: approve a change to a tenth as many products and a tenth as many fire. The other two are not fixed by it. The wishlist mail is counted in people, and it comes out at something over seven recipients for every product touched. The cart recomputation is counted in carts, and there is roughly one for every six products. That difference is the reason side effects belong in a preview at all rather than in a runbook. A reviewer who has calibrated on "this is about a thousand products" will read every number on the page as being about the size of the operation, and the largest one is several times larger than the operation and lands in customers' inboxes. The rows that equal affected are the ones the reviewer can already estimate; the rows that do not are the ones the preview is earning its place by carrying.

Partial writes, authorisation and the cost of stopping#

Separating propose from commit is not a transaction, and it is easy to slide from one to the other in your head. The plan digest tells you the world has not changed between approval and execution; it says nothing about what happens if the execution fails at ref 700 of 1,240. Partial writes are their own problem with their own remedies, and they are the subject of chapter 5, on the integration tests that force these failures.

Neither is a confirmation an authorisation. The commit handler above checks that somebody other than the agent approved this specific plan. It never asks whether that person was entitled to approve a deletion in this repository, or whether the proposal should have been possible to construct at all. A confirmed call that was never in scope is still a call that was never in scope, and the approver's yes does not repair it.

Confirmation is also expensive in a way that a chapter about safety is prone to understate. Every proposal that stops for a person adds a human's response time to the task, and on a high-frequency operation that cost is what eventually gets the control removed. Chapter 6, on what makes a control survive its first month, takes that problem seriously; for now it is enough to know that "confirm everything" is a design decision with a failure mode, not the safe default it appears to be.

Separate the two every time you see them offered together, because the pairing is what hides the missing half. The dry run is cheap, useful, and entirely compatible with a system in which the model decides whether to use it. The confirmation is the expensive half, and it exists only when a different actor authorises a specific plan the server is holding, described in enough detail to be wrong in front of somebody.

What remains open is the question the commit handler never asked. The proposal was well-formed, pinned and approved — and nothing established that the agent was permitted to touch that repository, that category or that customer's data in the first place. That boundary is drawn by scope, and scoping it correctly is harder than listing the operations an agent may perform.