Rebuilding one piece by hand is how you learn its real price

The argument about whether to adopt an agent framework is usually conducted in adjectives. One side says a runtime is heavyweight, the other says writing it yourself is a distraction, and neither position carries a quantity. There is a way to produce one, and it is narrow enough to fit in a sprint: take a piece the framework provides, rebuild exactly that piece for an agent you already run in production, and record what the rebuild costs.

Durable execution state is the piece to pick, for the reason chapter one gave — it is the largest of the four and the only one whose build cost is measured in weeks rather than lines. The brief is deliberately small. Strip the framework's checkpointer out of one agent and write the persistence layer by hand, so that a run interrupted at any point resumes correctly on another worker. Nothing else changes: same tools, same prompts, same model, same traffic. What follows is the four walls that exercise runs into, in the order they arrive, because the order is itself the lesson — the first is visible in an afternoon and each of the others is invisible until the one before it is solved.

The version that works in an afternoon#

import pickle, psycopg

def save(thread_id: str, step: int, state: dict) -> None:
    blob = pickle.dumps(state)
    with psycopg.connect(DSN) as conn:
        conn.execute(
            "insert into checkpoints (thread_id, step, blob) values (%s, %s, %s)"
            " on conflict (thread_id) do update set step = excluded.step,"
            " blob = excluded.blob",
            (thread_id, step, blob),
        )

def load(thread_id: str) -> tuple[int, dict] | None:
    with psycopg.connect(DSN) as conn:
        row = conn.execute(
            "select step, blob from checkpoints where thread_id = %s", (thread_id,)
        ).fetchone()
    return (row[0], pickle.loads(row[1])) if row else None

That is sixty-one lines in full and it demonstrably works: kill the process at step 5, start it again, watch it continue at step 6. It is worth being clear about what it gets right, because the four walls below are not a verdict on it. The upsert keyed on thread_id means a resume reads the newest state rather than an arbitrary one. The step number travels with the blob, so the resumer knows where it is. Serialisation and storage are separated from the loop, which is the structure the framework has too.

What it assumes is that the state is data, that a run either completes or dies, that resuming means continuing, and that a trace is a thing that happens inside one process. Each of those four assumptions holds in a demo and fails in production, and each failure is a wall.

The state contains things that are not data#

Tuesday's failure was TypeError: cannot pickle 'SSLSocket' object, raised from inside save, three steps into a run. The agent's state dictionary had grown a field during a routine change: a tool that reads from an internal search service had started caching its client on the state object so it could reuse the connection between steps, which is an ordinary optimisation and completely invisible in a diff that adds four lines. The state was no longer data. It contained a live socket, and by the end of the week it also contained a logging.Logger, a tokenizer instance held by a message-trimming helper, and a functools.partial closing over a database session.

Nothing in that sequence involves anybody being careless, and the mechanism is worth following because it will happen again in any codebase. The run state is a plain dictionary, so any step can add a key to it. Adding a key is the cheapest way to share something between steps. A connection, a logger and a tokenizer are all things you want to share between steps for good performance reasons. And no tool in the language distinguishes a dictionary entry that can be written to a disk from one that cannot, because the distinction is not about type — a Logger is a perfectly ordinary object — but about whether the thing it refers to exists outside the process. So the population of state fields drifts from data toward handles under normal engineering pressure, with nothing pushing back until a serialiser is asked to write one.

The correction is that a checkpointed state has a schema and a codec, and both of them are yours to maintain. Every field is either serialisable data or a handle that has to be reconstructed on load, and there is no way to have the compiler tell you which — the distinction is semantic. What worked was an explicit registry with a version tag on the blob, so that unknown fields fail loudly at write time instead of quietly at read time.

CODECS = {
    "messages": (dump_messages, load_messages),      # role/content/tool_call_id triples
    "customer": (lambda c: c.id, lambda i: Customer.get(i)),
    "budget_cents": (int, int),
    "started_at": (lambda d: d.isoformat(), datetime.fromisoformat),
}
TRANSIENT = {"search_client", "logger", "tokenizer"}   # rebuilt on load, never written

def encode(state: dict) -> bytes:
    unknown = set(state) - set(CODECS) - TRANSIENT
    if unknown:
        raise ValueError(f"unserialisable fields in agent state: {sorted(unknown)}")
    body = {k: CODECS[k][0](v) for k, v in state.items() if k in CODECS}
    return msgpack.packb({"v": 3, "body": body})

The raise is the whole design. An allow-list that rejects a field nobody declared turns the drift above into a failing test on the pull request that introduces it, rather than an exception in production three steps into a customer's run. The engineer who adds a cached client now has to write one line saying which of the two kinds it is, and one line is a proportionate price for a decision only a human can make.

Pickle had a second problem that only appeared at the first library upgrade, and it is a different problem from the first one. Blobs written before the upgrade referenced a message class by its old module path, and after the upgrade pickle.loads raised ModuleNotFoundError on every in-flight run — around 340 threads, all of them dead, none of them recoverable without pinning the old version back and draining.

The mechanism there is worth a sentence of its own, because "pickle is fragile" is the kind of assertion that leaves a reader unable to predict the next case. Pickle does not store the shape of an object; it stores instructions for rebuilding one, and those instructions name the class by module path. Loading therefore runs an import. An import of a path that no longer exists cannot succeed, and no property of the data affects that, since the bytes were never the problem. A pickled checkpoint is not a data format; it is a snapshot of your import graph, and it makes every dependency bump a data migration. Moving to an explicit codec with a version integer cost two days and made upgrades boring, which is what those two days actually bought.

Cancelling in the middle of a tool call#

The second wall was cancellation, and it is the one that most reliably surprises people, because in a demo a run either finishes or the process dies. In production a run is cancelled while it is doing something: the user closed the tab, a deadline fired, an operator killed a thread that had gone into a loop. The cancellation arrives as asyncio.CancelledError at whatever await is currently suspended, and on an agent that is very often the HTTP call inside a tool.

The Unknown state is the one a hand-built version tends not to have, and its absence is what makes cancellation dangerous rather than merely fiddly. A state machine with only success and failure forces every real outcome into one of the two, and a request that was cancelled after it was sent is neither: the charge may exist. Recording it as a failure is a lie that a later step will act on. Recording nothing is worse, because the resume then has no way to know an effect was ever attempted. Cancelling during POST /v1/charges leaves a request that may or may not have been processed, and the naive handling — let CancelledError propagate, skip the checkpoint because the step did not complete — records nothing at all. The version that works records the attempt before the call and treats cancellation as a transition into an unknown outcome rather than into a failure, which means the cleanup path has to run even though the task is being torn down.

async def run_tool(state, tool_call) -> dict:
    key = idem_key(state["thread_id"], tool_call["id"])
    await effects.mark(key, tool_call["name"], status="started")
    try:
        result = await TOOLS[tool_call["name"]](**tool_call["args"], idem_key=key)
    except asyncio.CancelledError:
        with anyio.CancelScope(shield=True):        # the write must not be cancelled too
            await effects.mark(key, status="unknown")
        raise
    await effects.mark(key, status="succeeded", result=result)
    return result

The shielded scope is the detail that took most of a day to arrive at, and the belief it corrects is one almost everybody holds. The belief is that an except block runs to completion once it has been entered, which is true of ordinary exceptions and false of cancellation. Cancellation is delivered at await points; the await effects.mark(...) inside the handler is an await point; the task is already cancelled when the handler starts. So the recovery write is suspended and immediately cancelled itself, the record never lands, and the code reads as though it handles cancellation while doing nothing at all — the most expensive kind of wrong, because it passes review and it passes the demo. Shielding declares a region that cancellation may not interrupt, which is what makes the write survive the thing it exists to record. Every runtime that offers cancellation has solved this somewhere in its internals, and none of them advertise it.

Replay is not the same as re-execution#

The third wall is the one worth the exercise on its own. Resuming from a checkpoint means re-entering the graph at the step after the last completed one, and the naive implementation gets that right for pure steps and wrong for every step that touched the outside world.

The reason is a consequence of the write ordering from chapter two rather than a bug. The state store records completions, and it records them after the fact, so the only two things it can say about a step are "this completed" and nothing at all. A step that charged a card and then crashed before its checkpoint was written falls in the second category, and so does a step that never started. One record, two worlds. No amount of care reading the checkpoint table separates them, because the distinguishing event happened after the last write and before the next one.

Breaking that tie needs a second record whose write happens before the effect rather than after it, and that is exactly what the effects table is. It is the same derived key from the previous chapter used for a different purpose: not to make the provider ignore a duplicate, but to make your own code decline to issue one.

create table agent_effects (
  thread_id  text        not null,
  effect_key text        not null,   -- sha256(thread_id:tool_call_id)
  tool_name  text        not null,
  status     text        not null,   -- started | succeeded | failed | unknown
  result     jsonb,
  updated_at timestamptz not null default now(),
  primary key (thread_id, effect_key)
);

With a row written at started and updated at succeeded, the two worlds above become two different rows, and the ambiguity has moved rather than vanished. On replay, a tool call whose key is succeeded returns the stored result without touching the network, which is what "replaying without re-executing effects" means concretely. A key in started or unknown is the interesting case, and the honest answer is that no generic mechanism can resolve it — the runtime cannot know whether the charge went through.

It is fair to describe that table as a write-ahead log for things you did to other people's systems, so long as the analogy is labelled and its limit is stated. A database's write-ahead log works because replaying its entries reconstructs the state they describe; the log is authoritative over the thing it logs. This one is not. The row says a charge was attempted, and the authority over whether a charge exists sits with the provider. So the entry that a real write-ahead log would resolve by replaying, this one can only resolve by asking. What resolves it is a per-tool probe: get_charge_status(idem_key) for payments, a read-back for anything that creates a row, and for tools with no probe available, a decision to fail the run and hand it to a human. Writing those probes is not framework work at all; it is a contract you negotiate with every downstream service you call, and it is the most expensive part of durable execution by a wide margin.

Verifying replay is its own body of work, and the test that found the most bugs was crude. For a run of eleven steps, kill the process after each step index in turn, resume, and assert that the final state matches the uninterrupted run and that the effects table contains exactly one succeeded row per tool call. Eleven kills times four representative conversations is 44 cases, they run in about ninety seconds against a local Postgres, and they caught four defects the ordinary tests did not: a delta write that lost a field when the interruption fell between two snapshots, a resume that replayed the model call and therefore paid for it twice, an effect row written outside the transaction that recorded the checkpoint, and a messages reducer — the function a runtime uses to merge a step's output into the run state, which for a message list normally appends — that appended the same tool result twice on resume. Each of those is a duplicate charge or a corrupted transcript in production, and none of them is visible in a test that lets runs finish.

That is the property of the test worth carrying away. Every one of the four defects lives strictly between two steps, and a test that lets a run finish never visits the interval where they exist. Interrupting at every index is not thoroughness for its own sake; it is the only way to sample the state space the defects occupy.

The exercise also priced the read path. A checkpoint blob for this agent starts at 41 KB after step 4 and reaches roughly 210 KB by step 11, because the message history accumulates tool results, and at 210 KB the write costs 8 ms at p50 and 34 ms at p95 against a Postgres instance in the same availability zone. Notice which way that grows: each step appends its output to a list that every later step also carries, so the cost of a step rises with how many steps came before it, and the last write of a run is the most expensive one. That is acceptable per step and unpleasant at eleven steps a run, so the version that shipped writes deltas and keeps a full snapshot every fifth step — which is another two days, and another mechanism that has to be correct on the resume path or the entire store is silently corrupt.

The snapshot interval is a dial between two costs rather than a tuning detail. Longer intervals write fewer bytes and make a resume replay more deltas; shorter intervals write more bytes and make a resume cheap. The correctness risk is entirely on the delta side, since a resume that applies deltas in the wrong order or misses one produces a state that is plausible and wrong, which is the failure the eleven-kill test above exists to catch.

Owning the store adds an operational surface that never appears in a design document. At 1,700 runs an hour with a snapshot every fifth step, the checkpoint table grows by roughly 9 GB a week, so it needs a retention policy, and the retention policy needs to know the difference between a run that finished and a run that has been paused for three days waiting on a human — deleting the second class is a data-loss incident. Postgres also has opinions about tables written this way: 200 KB rows go to TOAST storage, updates in place produce bloat that autovacuum on default settings does not keep up with, and the index on thread_id has to be there before the table is large rather than after. None of this is hard, and all of it is a page in a runbook and an alert that someone has to own.

The span that has to cross a process#

The last wall is smaller than the others but it is the one that makes hand-built tracing look easy and behave badly. Once a run can resume in a different worker, the second half of the run is a different trace unless the trace context is persisted with the state and restored on load. The reason is mechanical: a span's parent is identified by a value held in memory, memory does not survive the process, and nothing else in the checkpoint carries it. The W3C traceparent header exists for exactly this, and the fix is to store it in the checkpoint and re-establish a link on resume, so the two invocations are joined rather than merely adjacent.

from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

carrier: dict[str, str] = {}
TraceContextTextMapPropagator().inject(carrier)          # before writing the checkpoint
state["_trace"] = carrier                                # {"traceparent": "00-4f9c…-a1b2…-01"}

# on resume
ctx = TraceContextTextMapPropagator().extract(state["_trace"])
with tracer.start_as_current_span("graph.resume", links=[Link(ctx)]):
    ...

A span link rather than a parent is the right relationship, since the resumed invocation is a genuinely separate unit of work that happens to continue the same logical run. The distinction has teeth: a parent span contains its children in time, and a resume that happens four hours after the original invocation ended is not contained by anything. Declaring it a child would produce a tree whose root spans four hours of which it was doing nothing for almost all of them, and every latency number computed from that tree would be wrong. Getting this wrong produces traces that look complete and are not, which is worse than traces that are obviously broken, because the missing half is only missing when something went wrong.

What the exercise priced#

Setting the four sub-problems against what each cost turns an argument about frameworks into a number.

Nine days for one team on one agent, against a framework where the same behaviour is compile(checkpointer=saver). That is the number to carry into an adoption argument, and it cuts both ways: it is a strong reason to buy durable execution rather than build it, and it is also a measurement of how much you are trusting to code you did not write — which is why the next few chapters are about reading and leaving that code rather than about admiring it.

The last column carries a qualification that is easy to read past. Three of the four costs are stated as durations and one is not: the replay row ends with a clause about negotiating a contract per downstream service, which is work whose size depends on how many services you call and on how cooperative each team is. That is why the total is a floor rather than an estimate. Every figure in that column is bounded by your own codebase except one, and the unbounded one is the row that decides the answer.

Nine days is not a benchmark#

Two readings of that number are wrong in opposite directions, and both come up whenever the exercise is described.

The first is that the exercise is a recommendation to ship the result. It is not, in the ordinary case. What the exercise produces is knowledge about a dependency, and shipping the artefact converts that knowledge into a second thing to maintain, on a team that has just measured how much maintaining it costs. There are situations where building it for real is right — the following chapters describe several — and none of them is "we already wrote most of it".

The second is that the exercise is a performance comparison. The write latencies above are not one. Eight milliseconds at p50 and thirty-four at p95 are measurements of one hand-built writer against one Postgres instance, taken while learning what to measure, and nothing here says whether the framework is faster. That question needs identical traffic, a workload described in advance, and both builds running against it, which is a different exercise with its own cost.

The durable answer to "which piece would you never willingly rewrite" is replay-without-re-execution, and the reason sits in the last column of the table rather than in the day counts. Serialisation, cancellation and tracing are all problems inside your own process, so their worst case is bounded by how much code you have. Replay is bounded by how many external systems your tools touch and by whether each of those systems offers a way to ask what happened, and neither of those is under your control.

Having written the piece, the framework's version of it stops being opaque. You know what put has to do, you know why a checkpoint is written after the node rather than before, and you know which parts of its behaviour are policy choices rather than laws. That knowledge has a direct operational use the first time the runtime does something its documentation does not describe, which is a thing runtimes do on ordinary Tuesdays, for reasons that are readable if you know where to look.