A half-written run.json passes every version check

The file was 2,048 bytes and ended in the middle of a string:

$ wc -c /var/lib/pipeline/run_9d4a.json
2048 /var/lib/pipeline/run_9d4a.json
$ python -c 'import json,sys; json.load(open(sys.argv[1]))' /var/lib/pipeline/run_9d4a.json
json.decoder.JSONDecodeError: Unterminated string starting at: line 34 column 16 (char 1893)

The document had been 6,104 bytes an hour earlier. An agent had opened it for writing, the kernel had flushed the first page, and the container had been evicted before the rest of the write reached disk. Every mechanism from the previous chapter was in place and working. The write carried the correct base version. No other writer had raced it. Compare-and-swap has nothing to say about this, because it validates which write wins among writes that completed, and this write never completed; the file's content is neither the old document nor the new one, and there is no version of the run that it corresponds to.

This one file was the only copy of the run's state. Twenty-three minutes of work across four agents, the pointers to everything they had written to object storage, and the ownership records that would have let a replacement resume, all of it referenced only from a file that no longer parses. The parsed contracts still exist in the bucket, but nothing knows which run they belong to or which stage consumed them.

Three failures live in this area and they get conflated constantly. They have different mechanisms, different fixes, and different costs, and a system that has handled one of them is not protected against the other two. The first of them has a name worth having: a torn write is one that stopped partway, leaving the stored bytes as neither the old document nor the new one.

Replacing the document instead of overwriting it#

A write to a file is not atomic. write(2) can return having transferred some of the bytes, the page cache — the kernel's in-memory holding area for file data that has not reached the disk yet — can be flushed in pieces, and a process that dies partway leaves the file in a state that never existed as a document. The operation that is atomic on a POSIX filesystem is rename(2) within the same directory, which is why the standard shape is to build the new document somewhere else and then swap it into place.

import json, os, tempfile

def replace_atomically(path: str, doc: dict) -> None:
    directory = os.path.dirname(path)
    fd, tmp = tempfile.mkstemp(dir=directory, suffix=".tmp")
    try:
        with os.fdopen(fd, "w") as f:
            json.dump(doc, f)
            f.flush()
            os.fsync(f.fileno())          # bytes are on disk, not just in cache
        os.replace(tmp, path)             # atomic within the same filesystem
        dirfd = os.open(directory, os.O_DIRECTORY)
        try:
            os.fsync(dirfd)               # the rename itself is durable
        finally:
            os.close(dirfd)
    except BaseException:
        os.unlink(tmp)
        raise

A reader at any instant sees either the complete old document or the complete new one. The two fsync calls are the part people skip, and they are doing different jobs: the first makes the temporary file's contents durable before the swap, and the second makes the swap itself durable, because without it a crash can leave the directory entry pointing at a file whose data never landed. Together they cost roughly two milliseconds on local SSD and considerably more on network storage, which is the reason they get removed by someone optimising a hot path and the reason the truncation reappears six months later.

The same directory matters, and the same filesystem. os.replace across a mount point is not a rename; it degrades to a copy, and the atomicity is gone. This is the specific way the pattern breaks when someone moves the artifact to a mounted volume and keeps the temporary file in /tmp.

On an object store the situation is better and not free. A single PUT of a whole object is atomic at the object level — readers see the previous object or the new one, never a mixture — so a client that dies mid-upload leaves the old object intact. The failure returns as soon as the artifact stops being one object. If the run's state is a document plus a separate index, or a document plus a sidecar of appended events, a crash between the two writes leaves them disagreeing, and no per-object atomicity helps. Keeping the authoritative state in exactly one object, and treating everything else as derived, is what preserves the guarantee.

The other half of losing the run is having no earlier copy to fall back to. Object versioning, or simply keeping the last N documents under keys like run_9d4a/v61.json, turns a catastrophic loss into a rollback to a version a few minutes old. At 4 KB a document and roughly sixty versions a run, retaining every version costs about 240 KB per run, which is not a number anyone needs to optimise. The reason to do it is not history in the sense chapter 1 ruled out; it is having something to restore.

The retry that applies your change twice#

There is a bug in the retry loop from the previous chapter, and it only shows up when the network misbehaves rather than when agents contend.

An agent sends a conditional write. The store receives it, validates the version, applies it, increments to version 63, and starts writing the response. The connection drops. The client sees a read timeout and knows only that it never got an answer, which covers two different worlds: the write applied, or it did not. Retrying is correct in one world and wrong in the other, and the client cannot tell which it is in.

Retrying naively re-reads the document — now at version 63, containing the agent's own change — and calls the mutation function on it again. If the mutation is "set my section pointer to this URI", reapplying is harmless because setting a field twice is the same as setting it once. If the mutation is "append my finding to the findings list" or "increment attempts", reapplying corrupts the document in a way nothing will flag:

{
  "attempts": 2,
  "findings": [
    {"clause": "9.4", "severity": "high", "note": "auto-renewal, 60d notice"},
    {"clause": "9.4", "severity": "high", "note": "auto-renewal, 60d notice"}
  ]
}

Two identical findings, and an attempts counter that will trip the give-up threshold one retry early. The document is valid, versioned, and wrong.

What the retry needs is for the update to be idempotent, meaning that applying it a second time leaves the document in exactly the state the first application left it in, so the client can retry without knowing which of the two worlds it is in. The fix is to make the update carry its own identity, and to have the document remember which updates it has already absorbed:

def apply_once(doc, update_id, change):
    if update_id in doc["applied_ids"]:
        return doc                      # already absorbed; this is a no-op
    change(doc)
    doc["applied_ids"].append(update_id)
    doc["applied_ids"] = doc["applied_ids"][-200:]   # bounded
    return doc

The identifier is generated once, before the first attempt, and reused across every retry of that same logical update — a UUID, the 128-bit random value whose collision odds are small enough to ignore, or a deterministic key such as agent-vendor:section:v3. Reapplying then becomes exactly as good as applying once, which is what lets the client retry freely without knowing whether the previous attempt landed.

Two details decide whether this actually holds. The identifier must be recorded in the same write as the change, because if the two are separate writes there is a window where the change landed and the record of it did not, and the whole problem returns one level down. And the list has to be bounded, since an unbounded ledger of applied identifiers grows the document every agent reads on every poll; two hundred entries at about forty bytes each is 8 KB, which is already larger than the state it protects, so keeping the window at the last few dozen updates per run is more sensible than keeping everything.

What this buys is effectively-once application, not exactly-once delivery, and the distinction is worth holding onto because the second thing is not available. The client still cannot learn whether its first attempt succeeded; it has simply arranged for the answer not to matter. That is the general shape of every solution in this area, and any design that claims to guarantee a message is processed exactly once has hidden a deduplication step somewhere and stopped talking about it.

When the view you read has already moved#

The third failure produces no error anywhere and is the hardest to see. The report agent reads version 62 at 14:31, finds six sections marked done, and spends ninety seconds composing a report from them. At 14:31:40 a reviewing agent marks the tax section rejected because its source document turned out to be a draft, and writes version 65. At 14:32:30 the report agent writes a finished report that includes the tax section. No write was lost, no document was torn, every version check passed, and the output is wrong because it was derived from a view of the world that stopped being true while the agent was working.

Compare-and-swap does not help, because the report agent's write does not conflict with anything — it is writing to a different part of the document. What it needs is a way to notice that the inputs it read have changed:

result = compose_report(doc)              # 90 s of model work
ok = store.put_if_match(
    key,
    with_report(doc, result, based_on_version=62),
    if_match=etag_of_version_62,
)
if not ok:
    fresh, _ = store.get_with_etag(key)
    if changed_under(fresh, doc, paths=["tasks[*].status", "tasks[*].output.uri"]):
        raise StaleInputs(read_version=62, head=fresh["version"])
    # nothing the report depended on moved; retry the write

The paths argument is the design decision. Making the write conditional on the whole document means every unrelated write by another agent forces a ninety-second re-derivation, which under six writers is most of the time. Making it conditional on nothing means the report agent never notices. Naming the parts of the document the work actually depended on is the middle, and it requires the agent to declare its dependencies, which most agent frameworks do not do for you.

When the check fires there are two honest responses. Re-derive, which costs the ninety seconds and the tokens again, and is right when the inputs are cheap to reprocess. Or record the output with based_on_version attached and let the consumer decide, which is right when re-derivation is expensive and the staleness may be tolerable — a report noting that it was composed from version 62 while the run ended at version 71 is at least a report whose provenance is inspectable. What is not acceptable is writing the result with no record of what it was derived from, because that produces exactly the artifact that caused the Thursday conversation: correct-looking, well-formed, and quietly built on something that has since been retracted.

Four mechanisms are now in play — conditional writes, atomic replacement, idempotent updates, and dependency-checked reads — and they interact. A lease has different properties from a compare-and-swap, an append-only log removes some of these problems and introduces others, and the single-writer proposal that keeps coming up in review has a shape that deserves examining on its own terms.