A hold before commit is reversibility; a disclaimer is not

The CRM cleanup agent shipped with an undo button. It read stale opportunities, decided which ones were dead, and archived them, and the design review had specifically asked for a safety net, so the engineer added a toast in the corner of the screen — "12 opportunities archived. Undo" — with a five-second timer. The button worked exactly as written: it restored the twelve rows to their previous stage. What it could not touch was the opportunity.archived webhook that had fired the moment the write landed, which the customer's marketing automation consumed, and which had already sent twelve "sorry we couldn't help this time" emails to twelve accounts, three of which were in active negotiation.

The undo restored the database and nothing else, because the action had already committed before the button appeared. That is the ordinary shape of the failure: undo is treated as a UI affordance rather than as a property of the system, and a button that reverses one of an action's effects reads to everybody involved as though it reverses the action.

Three things, all of which have to be true#

An action is reversible when a specific and checkable set of conditions holds. There is a window during which the action has not yet taken effect anywhere outside your own boundary. There is a cancel path that someone can actually reach inside that window — reachable meaning a human or an automated check can both see that the action is pending and stop it, in the time available, with enough information to decide. And the prior state is retained somewhere restorable, which usually means a version, a snapshot or an event you can replay, rather than a field that has been overwritten in place.

Missing any one of the three makes the other two decorative. The CRM agent had a cancel path and a form of prior state and no window, so the webhook left before anyone could act. A system with a window and prior state but no reachable cancel path — a 24-hour hold that only fires an alert into a channel nobody reads — commits everything by default and calls it review. A system with a window and a cancel path but no versioning can stop the action before it lands and cannot put anything back if it has already landed once, which is the common state of any pipeline that writes with an UPDATE and keeps no history.

Applying the test to real actions is more useful than the abstraction, because the answers are not intuitive.

The Slack row is the one worth dwelling on, because it is the clearest case of an action that engineering cannot rescue. Deleting a message removes it from the channel and does not remove it from the push notification on 200 phones, the email digest that has already been generated, the mobile client of anyone who read it offline, the third-party archival tool the compliance team installed, or the screenshot someone took. There is no window, because delivery is the effect and it is immediate. There is no prior state to restore, because the prior state is "these people had not read this", and human memory does not support rollback. Every technical control that appears to make it reversible is really making it deniable, which is a different and much weaker property. The same reasoning applies to any output that reaches a third party's system or a person's attention: an email that has left your relay, a payment that has cleared, a tweet, a webhook a partner has already processed.

The effects you did not know the action had#

The CRM team believed archiving was a single-column update, and by the time the incident was written up the action had six effects. The row changed. A database trigger wrote an audit record. A change-data-capture stream published to Kafka, which fed the search index and the analytics warehouse. The application emitted a webhook, which the customer's marketing tool consumed. An in-app notification went to the deal owner. And a nightly forecast job, reading the warehouse copy, revised the quarterly pipeline number that the sales director presents on Mondays.

Undo was designed against the first of those. Enumerating the rest is not difficult, and it is the step that gets skipped because the primary write is so obviously the whole story. The places to look are consistent across most systems: database triggers, CDC or outbox streams and everything subscribed to them, application-level event emitters, webhook registrations including ones customers configured themselves, search and cache invalidation paths, notification and email services, audit logs, and any third-party sync running on a schedule. For each one the question is whether reverting the primary write also reverts it, and the answer is usually that it emits a second event which downstream consumers may or may not handle as a reversal. The marketing tool in this incident had no concept of un-archiving, so the compensating event arrived as an unknown type and was discarded.

There is a cheap test that catches most of this before launch. Write the restore script — the actual code that takes one committed instance of the action and puts the world back — and run it in staging against a real instance, then check every downstream system by hand to see whether it agrees that nothing happened. Teams that do this discover the webhook in an afternoon rather than in an incident. Teams that cannot write the script in an hour have learned something more valuable, which is that the action they were describing as reversible is not, and the design needs a pre-commit control instead.

Prior state has a retention dimension that the three-part test does not capture on its own. A restorable version is only restorable while it exists, so the retention window has to exceed the time it takes to detect the problem. Row-level history kept for seven days is worthless against a defect that surfaces in the monthly reconciliation, and this mismatch is common because retention is usually set by storage cost and detection latency is usually not measured at all. Pairing the two — how long until we would notice, how long do we keep the old value — is a two-line calculation that belongs in the design document.

It is also worth separating soft deletion from versioning, since they are often treated as the same safety net. A deleted_at column restores existence and nothing else: if the same operation also normalised an address, recalculated a score or cleared a relationship, marking the row undeleted brings back the post-change values. Versioning keeps the whole prior tuple, which is what an actual restore needs, and it costs more storage and a slightly more complex read path. Choosing soft deletion and describing it as reversibility is the version of this mistake that survives longest, because it works perfectly in the demo where the only thing that happened was a delete.

Compensation is not reversal, and a disclaimer is neither#

For most irreversible actions there is a compensating action available, and it is genuinely useful as long as nobody confuses the two. A wrongly issued refund can be followed by a charge; a wrongly sent email can be followed by a correction; a wrongly archived opportunity can be un-archived and an apology sent. Compensation creates a new event that offsets the first one in whatever ledger you care about, and it leaves the original event in place along with everything the original event caused. In accounting that is exactly right, because the ledger is the thing that matters and a reversing entry restores it. In customer-facing systems it is much weaker, because the state you damaged lives partly in a person's head. The 8,400-recipient correction from the previous chapter was a compensating action, and it made the incident worse.

The disclaimer deserves a specific mention because it is the control that most often appears in a design document under the heading of risk mitigation. "AI-generated — please verify before acting" attached to an output does not create a window, does not create a cancel path and does not retain prior state, so by the test above it provides no reversibility at all. What it does is shift responsibility, and it does that less effectively than teams assume: in most consumer and financial contexts the fact that a customer was warned does not remove the obligation to make them whole, and in an incident review "the banner said to check it" has never once been accepted as an answer. Disclaimers are worth having for expectation-setting and for the honest reason that some users will in fact check. They are not a substitute for a mechanism, and putting one in the mitigations column of a risk register instead of a mechanism is how features ship with nothing behind them.

Building the window you do not have#

Where the underlying action has no natural window, the usual approach is to introduce one: the model's decision produces an intent record rather than an effect, and a separate committer turns intents into effects after a delay, during which anything can cancel them.

from datetime import datetime, timedelta, timezone

def propose(conn, action_type: str, payload: dict, prior: dict, hold: timedelta):
    """Record the intent and the state it will overwrite. Nothing has happened yet."""
    return conn.execute("""
        INSERT INTO action_intents
              (action_type, payload, prior_state, commit_after, status)
        VALUES (%s, %s, %s, %s, 'pending')
        RETURNING id
    """, (action_type, payload, prior, datetime.now(timezone.utc) + hold)).fetchone()[0]

def cancel(conn, intent_id: int, reason: str) -> bool:
    """Only succeeds while still pending — the committer takes the same row lock."""
    rows = conn.execute("""
        UPDATE action_intents SET status = 'cancelled', cancel_reason = %s
         WHERE id = %s AND status = 'pending'
    """, (reason, intent_id)).rowcount
    return rows == 1

def commit_due(conn, executors):
    for row in conn.execute("""
        SELECT id, action_type, payload FROM action_intents
         WHERE status = 'pending' AND commit_after <= now()
         FOR UPDATE SKIP LOCKED
    """):
        executors[row.action_type](row.payload)
        conn.execute("UPDATE action_intents SET status='committed' WHERE id=%s", (row.id,))

The prior_state column is the part that is easy to leave out and expensive to add later. Storing it at proposal time is the difference between being able to restore the twelve opportunities to stage='negotiation', owner=42, next_step='call Tue' and being able to restore them to "not archived", which is not the same thing and will be discovered during the recovery rather than before it.

The hold duration is a product decision with a cost on both sides, and it should be set from the latency of whatever will catch the mistake. A hold shorter than the detection path is a pause with no observer, which is the same defect the canary in the previous chapter had. If the catcher is a scheduled reconciliation that runs every fifteen minutes, a five-minute hold catches nothing. If the catcher is a human who reviews a queue twice a day, a two-hour hold is theatre and the real hold is eight hours. On the CRM agent the team settled on 30 minutes for archives, which covered the automated consistency check and was short enough that the cleanup still finished within its nightly window.

The failure this introduces#

A hold-and-commit design converts one write into a state machine with a background worker, and background workers time out. The committer calls the CRM's API, the call takes longer than the client's 10-second deadline, the worker never learns whether the write landed, and the row stays pending until the next pass picks it up and calls again. On an archive that is survivable, since archiving something twice leaves it archived. On the refund path from the first chapter it means the customer is paid twice, and the retry that caused it is the same retry policy that makes the rest of the system resilient.