Durability guarantees you never exercise still cost you daily

Most arguments about adopting a durable engine are conducted as though the question were whether durability is good. It is, and that framing settles nothing, because the machinery in the previous chapter is not free and its bill arrives on days when no run has failed. A determinism rule constrains every review. A versioning discipline constrains every deploy. A worker fleet needs alarms whether or not it is doing anything interesting. Those charges are levied on the calendar, and the guarantee they buy is only redeemed when a process dies holding something worth keeping.

So the question worth asking is not whether the guarantee is real but whether your runs ever hold anything worth keeping. That is a measurable property of a system you already operate, and the sections below turn it into four numbers.

The proposal that was accurate and wrong#

The proposal was two pages and the technical content was accurate. It suggested moving the support-triage agent onto a durable workflow engine, and it cited the purchase-order incident from 8 August as the motivating example. The triage agent is a different animal: it reads an incoming ticket, classifies it, searches the knowledge base, drafts a reply, and posts that draft as an internal comment for a human agent to send or edit. It runs 6,000 times a day, its median run takes 11 seconds and its p99 takes 38, it has one external write, and that write is a comment posted to the ticketing system with a client-supplied external id that the vendor deduplicates.

Everything the engine guarantees, this agent already has or does not need, and it is worth walking the four guarantees one at a time rather than asserting the conclusion. Position surviving a deploy matters when a run is likely to be in flight during one; the longest run here is shorter than the time between two deploys by three orders of magnitude, so a rolling deploy at worst loses 40 seconds of work on the handful of runs in flight, and the recovery is to re-run the ticket from the top for about 9,000 input tokens and €0.03. Durable timers matter when a run waits; there is no human pause, because the human enters after the run has finished. Uniform handling of irreversible effects matters when there are irreversible effects; there is no payment, no booking, nothing that cannot be repeated. And losing a run entirely is a non-event, since the ticket is still in the queue and the next poll picks it up.

What the commitment actually is#

Adoption is not a library import, and the costs are ongoing rather than one-off. They fall into five groups, and each of them lands on somebody's week.

Read the third column before the first two. Four of the five lines say "Ongoing", which changes the arithmetic of the whole decision: a one-off cost is compared against a benefit once, and a rate has to be compared against a benefit rate for as long as you run the system. That asymmetry is what the last section of this chapter turns on.

The determinism row is the expensive one for an agent specifically, because the code most likely to change every week is the loop that assembles context, picks tools and calls a model, and that is precisely the code the engine wants to be stable and deterministic. The discipline that resolves this — thin workflow, everything volatile inside activities — is good design and it is also a constraint that a team new to it will violate on the third prompt change, discover in staging if they are lucky, and discover as a stalled production run if they are not.

The failure surface deserves a number. Over the first six months of running an engine for the purchase-order service, the incidents attributable to the engine rather than to the business logic were: two non-determinism stalls after deploys, each blocking every in-flight run for 20 to 50 minutes until a rollback; one run stuck at the history event limit after an agent loop ran away; and one afternoon of degraded throughput from an undersized worker pool that nobody had connected to task-queue backlog.

None of those was catastrophic and all of them were novel, which is the actual cost of a new component: not that it fails more often, but that it fails in ways your on-call rotation has never seen. An engineer paged at 03:00 for a stalled workflow task has no prior for it, cannot reason about it from experience with the API or the database, and has to read a runbook written by somebody who was also learning.

The conditions that reverse it#

Arguing against adoption is only useful if you also say what would change your mind, and each trigger below is observable in a system you already run rather than a feeling about the future.

The purchase-order service triggers four of those six, which is why the same argument that rejects the engine for triage accepts it there. Approvals wait days. There are three irreversible writes. And finance asks for per-order reconstruction every month-end.

The last two rows are the ones the two-number summary above does not cover, and it is worth saying so rather than pretending the arithmetic is complete. An audit obligation is not about how much work is in flight; it is about what somebody has to be able to reconstruct months later, and a history that exists by construction beats a reconstruction from application logs however short the runs are. A team with five long-running processes is not making a per-run argument either — the machinery is being amortised across teams, so the cost line is divided while the benefit line is added up. Both are real reasons and neither is visible in a p95.

The first row is the one worth computing rather than eyeballing, because it is the only one whose trigger is a ratio. The share of runs in flight when a deploy lands is roughly the run's duration divided by the interval between deploys — a run occupies that fraction of the timeline, so a deploy arriving at an arbitrary moment catches that fraction of them. Runs are six minutes and deploys happen six times a day, which puts the interval at four hours and the fraction at a few percent: roughly 3% of runs are in flight at any deploy. On the triage agent the same division puts it near zero, and the two systems are not close enough for one argument to cover both.

A single trigger is usually enough on its own if it is the human pause, because chapter 5's inventory — pause table, sweeper, reminders, expiry policy, race handling, authorisation grants — is the largest body of hand-written code in this topic and the one most likely to be subtly wrong.

The middle of the range#

The choice is not binary anymore, and treating it as engine-or-nothing is what makes the decision feel bigger than it is. Libraries now offer the durable-execution guarantee against a database you already run: DBOS records step outputs in Postgres tables inside your own schema and replays the remainder without a separate cluster, LangGraph's checkpointer persists an agent graph's state per node and resumes at a node boundary, and several queue products have grown workflow features that cover fan-out and joins. Each of these takes a different position on the trade between how much of your code is constrained and how much operational surface arrives with it. A team that needs resumability and does not need three-day pauses can often buy exactly the part it needs.

The portable observation underneath all of them is that the schema of your run is the migration cost. Steps with names, values recorded at step boundaries, effects with keys and a ledger — those survive a move from hand-rolled to library to engine.

The reason they survive is that they describe your business process rather than any particular runtime. Every system that offers durability has to express "this named unit produced this result" and "this external write happened under this key", because those are what durability is about, so a mapping from your shape onto theirs always exists. Code that scatters external calls through an 800-line function has no named units to map and no keys to carry, so there is nothing to migrate — and that is true whether or not you ever adopt anything, which is why the shape is worth having on its own.

# the part worth writing on day one, whatever you adopt later
async def run_order(ctx, req):
    supplier = await step(ctx, "resolve_supplier", lambda: resolve(req.buyer_id))
    order    = await step(ctx, "open_order",       lambda: mint_order_ref(req))
    quote    = await step(ctx, "quote",            lambda: price(req, supplier))
    await effect(ctx, "charge", key=effect_key(ctx.run_id, "charge"),
                 call=lambda k: payments.charge(quote.total_cents, idempotency_key=k),
                 resolve=lambda k: payments.find_by_key(k))
    return order

Two functions carry the whole discipline: step, which memoises a named unit's result, and effect, which insists that any external write names a key and a way to find out whether it already happened. Read effect's three arguments as the three questions chapter 4 asked. key is what the resumed run can reproduce. call is what to do if the effect has not happened. resolve is how to find out whether it has — and a step whose integration cannot supply the third argument is a step you have just discovered needs reconciliation, which is a useful thing to learn at design time rather than during an outage. Whatever runs underneath — your own loop over a checkpoint table, a Postgres-backed library, or a replaying engine — reads the same shape.

What "we'll need it eventually" is worth#

The argument that ends most of these discussions is that durability is good practice and the requirement will arrive later. It is worth answering directly rather than conceding, and the answer is the "Ongoing" column.

Adoption cost does not fall by adopting early. The three to four weeks are the same in March and in November, because they are a fixed amount of setup work rather than something that gets harder with time. What does change with time is the accumulated rate: adopting in March means eight extra months of determinism constraints and version patches on an agent whose loop is rewritten twice in that period. So the quantity people imagine they are saving is constant, and the quantity they are spending grows, which makes early adoption strictly worse than adoption at the moment of need. The thing that genuinely reduces future cost is the shape above, which costs days rather than weeks and is useful immediately because it is what makes a run inspectable.

There is also a version of "eventually" that is a real forecast rather than a hedge. If the roadmap for next quarter contains an approval step, or a multi-day supplier negotiation, or an agent that books physical logistics, that is a dated requirement and adopting ahead of it is planning. The distinction is whether anybody can name the feature and the quarter.

The honest limitation on everything in this topic is that none of it produces exactly-once effects on systems you do not control, and no engine changes that. Chapter 4's reconciliation job still runs at 03:00, the settlement file is still the arbiter, and a compensating reversal is still an external write with the same gap one level down. Durable execution moves the uncertainty into a small, named set of places and gives each of them a resolution procedure, which is a large improvement over uncertainty distributed across an entire codebase, and it is not a proof.

What remains is to run the decision on a system you actually have. Take the agent you are working on this week and write down four numbers: its p95 run duration against your deploy interval, the number of steps that write to a system you cannot roll back, the longest wait it contains that involves a person, and how many runs are in flight during a typical deploy. Those four are not a checklist; they are the two quantities the whole topic reduces to — how much work is in flight when something dies, and how expensive that work is to lose — split into the parts you can actually measure. Then walk the recovery boundary for its most expensive step and say out loud what happens if the process dies immediately after the call.

If the answers are 38 seconds, one deduplicated write, no human, and a handful of runs, the queue and a run table are the right system and the argument is over. If any of them is a day, a payment, or a person, the machinery in chapters 1 through 5 is being written by you or by something else, and the only question left is which.