Quiescence is an obligation count reaching zero, not a quiet log

run_9310 was marked complete at 15:22:03 and torn down over the following four hundred milliseconds: worker slots returned to the pool, the run's scratch prefix scheduled for deletion, the trace closed and flushed. At 15:22:04 the writer received a message.

15:22:01.418  reconciler  send  -> writer  {revise_summary, clause_table v3}   [enqueued]
15:22:01.902  observer    poll  reconciler: idle
15:22:02.140  observer    poll  extractor-1: idle
15:22:02.361  observer    poll  extractor-2: idle
15:22:02.588  observer    poll  writer:      idle
15:22:02.9xx  observer    ledger scan: 14/14 terminal, 0 claims outstanding
15:22:03.005  observer    run_9310 -> COMPLETE
15:22:04.117  writer      recv  {revise_summary, clause_table v3}
15:22:04.230  writer      ERROR  run context closed; artifact prefix unwritable

Every observation the observer made was true when it made it. The reconciler really was idle at 15:22:01.902, because it had handed its message to the broker — the queueing service that accepts a message from one agent and delivers it to another, so that neither has to be running at the same moment — and was waiting for a reply. The writer really was idle at 15:22:02.588, because the broker had not delivered yet. The message existed the whole time, in neither agent, and the observer had no way to see it. The customer got the summary built from clause table v2, and the revision that the reconciler had already decided was necessary never happened.

An obligation exists between the send and the acknowledgement#

The unit that has to be counted is not the message and not the task, it is the gap between them. Define an obligation as something the run owes: created at the moment an agent dispatches work — a message to a peer, a task onto the ledger, a sub-agent spawn, a tool call whose result the agent intends to consume — and discharged only when the receiving side has acknowledged taking responsibility for it. A run is quiescent when no agent is executing, no claim is outstanding, and the obligation count is zero. Silence is the first of those three; quiescence is all of them together.

The word "acknowledged" carries the weight. An obligation is not discharged when the sender's send() returns, because the broker accepting a byte buffer says nothing about delivery. It is discharged when the receiver has durably recorded that it now owns the work, which in practice means the ack rides on the same commit that creates the receiver's claim.

def dispatch(bus, ledger, run_id, sender, target, payload):
    ob_id = uuid7()
    ledger.obligation_open(run_id, ob_id, sender, target,
                           kind="message", at=now())      # durable, before send
    bus.send(target, {"obligation": ob_id, "from": sender, "body": payload})
    return ob_id

def receive(bus, ledger, run_id, agent):
    msg = bus.receive(agent)
    ledger.transaction(                                   # one durable commit
        obligation_close=(run_id, msg["obligation"]),
        claim_create=(run_id, agent, msg["body"]["task_id"]),
    )
    return msg["body"]

Opening the obligation before the send is what makes the count conservative in the safe direction. A crash between obligation_open and bus.send leaves an obligation that will never be discharged by a receiver, so the run stays non-quiescent and eventually hits a budget — the failure that hangs, which chapter 3 established is the one worth preferring. Reversing the order gives the opposite bias: send first, and a crash before the open leaves a message in flight that the count does not know about, which is the run_9310 defect with extra steps.

Applied to run_9310, the reconciler's dispatch at 15:22:01.418 opened obligation 01H…9f, and at 15:22:02.9xx the ledger scan would have found it open. The run is not quiescent; the observer waits; the writer acks at 15:22:04.117 and starts work. Completion happens after the revision, at 15:23:40, and the customer gets clause table v3.

Why an outside observer cannot do this by sampling#

Adding a counter to the ledger works here because there is a shared durable store that every agent already writes to, which quietly reintroduces a coordination point. It is worth understanding why the alternative — an observer that samples each agent's local state and adds it up — cannot be made to work by sampling harder, because that is the design people write first and its failure mode is intermittent enough to survive months of testing.

The observer asked the reconciler first and the writer second, and the message crossed between the two questions. No amount of speed fixes this, because the window is not a performance problem — it is that the observer's view is assembled from moments that are not simultaneous, and any pair of observations has a gap a message can travel through. Sampling in the other order fails on a message travelling the other way. Sampling twice narrows nothing on its own, since a message can cross during the second pass exactly as it did during the first.

This is the problem that distributed termination detection was invented for, and the two classical answers are both worth knowing because both show up, renamed, in agent frameworks. Dijkstra and Scholten's 1980 algorithm builds a spanning tree over the computation: every agent remembers the parent that activated it, keeps a deficit count of children it has activated and not yet heard back from, and signals its parent only when it is idle and its deficit is zero. Termination is the root's deficit reaching zero, and it is exact — no sampling anywhere. Mattern's four-counter method keeps sampling but repairs it: take two consecutive snapshots, and conclude termination only if all agents were idle in both and the total messages sent equals the total received in both. A message in flight during the first pass shows up as sent-but-not-received in the second, so the crossing that fooled the naive observer is exactly what the counters catch.

The ledger scheme is the one this pipeline uses, and the reason is not elegance. There is already a durable ledger that every agent commits to, the run has a few hundred dispatches rather than a few million, and an extra pair of durable writes per dispatch is affordable: a typical run here makes 340 dispatches, so 680 writes at about 2 ms each adds roughly 1.4 seconds to a run that takes six to nine minutes, which is under half a percent. A system with a million messages per run would pay a different price and should use the deficit scheme, where the cost is one ack the receiver was probably sending anyway.

Obligations that are not messages#

Once the count exists, it is the natural home for every other kind of outstanding commitment, and folding them in is what makes the completion condition mean what its name says.

A tool call that takes four minutes is an obligation opened at dispatch and discharged when the result is recorded, which is how a run that has released every claim while waiting on a callback stays non-quiescent. A sub-agent spawn is an obligation opened by the parent and discharged when the child writes its terminal outcome, so a run cannot complete while a descendant is still executing — the property chapter 5 will need when it cancels a tree. A human approval is an obligation too, and one that should be explicitly marked as not billable, so that a run parked for eleven hours waiting on a compliance sign-off does not read as eleven hours of stuck agent time.

Partial results are the case that most repays modelling this way. When extractor-2 wrote three of five sections and stopped because it hit its own limit, chapter 3's ledger showed a done item with a verified artifact behind it, which is true and incomplete. Under obligation accounting, the extractor's claim carried an obligation to produce five sections, and finishing three discharges three of them and leaves two open. The run cannot go quiescent while they are open, so the choice becomes explicit at the moment it arises: dispatch the remaining two to another agent, or close them deliberately with a recorded reason. Both are fine outcomes and both are visible; what disappears is the third option where nobody decides and the run reports success.

Who gets to declare it, and when#

With the count in the ledger there is no need for an observer process at all, which removes the sampling race rather than working around it. The agent whose commit takes the count to zero is the agent that evaluates completion, in the same transaction, because at that instant it holds the only view that is guaranteed consistent — its own write. Completion becomes edge-triggered on the transition to zero — evaluated by whoever causes the change, at the moment it happens — instead of level-triggered, which is the arrangement where a separate poller keeps asking whether the count is currently zero, and the run's terminal record is written by whichever peer happened to finish last.

UPDATE runs SET state = 'complete', completed_at = now(), completed_by = $2
 WHERE run_id = $1
   AND state = 'running'
   AND NOT EXISTS (SELECT 1 FROM obligations
                    WHERE run_id = $1 AND closed_at IS NULL)
   AND NOT EXISTS (SELECT 1 FROM ledger_items
                    WHERE run_id = $1 AND status NOT IN ('done','failed_permanent'));

The state = 'running' predicate makes the write idempotent under contention, meaning that running it twice leaves the same result as running it once: if two agents discharge their last obligations within the same millisecond, both evaluate the condition, one row is updated and the other statement affects zero rows and moves on. Nothing is lost by the loser, since the run is complete either way and completed_by is bookkeeping.

An external observer still has a job, and it is a different one. It watches for runs that have been non-quiescent for longer than expected and reports them, which is monitoring rather than control. The distinction matters because an observer that can declare completion is a second writer racing the agents, and an observer that can only notice an unfinished run is a reader that cannot corrupt anything by being slightly out of date.

What a zero count does not tell you#

Quiescence is a statement about activity, not about quality. A run whose obligations all reach zero because every extractor produced an empty clause table is perfectly quiescent and perfectly useless. Termination detection tells you that nothing further will happen without new input, and whether what happened was any good is a separate evaluation over the artifacts, using acceptance criteria that have nothing to do with counting.

The sharper limitation is that a run can stay non-quiescent forever by legitimately generating new obligations. Every discharge is accompanied by a new dispatch — the reconciler asks for a revision, the writer asks for a clarification, the reconciler revises again — and the count oscillates between one and four without ever touching zero. Nothing here is stuck by the definitions of chapters 1 and 2: no cycle in the wait-for graph, no repeated identical action, every agent making calls and spending tokens. The run is alive and it is not going to stop, and the obligation count, being a measure of outstanding work rather than of consumed resource, will never object.

That is the case a termination detector cannot handle by design, because there is nothing anomalous to detect. Stopping it requires the other half of the problem: a bound on what the run is permitted to consume in total, enforced across every agent and every hop, with a defined thing that happens when it runs out.