Dependencies, success criteria and recovery points make a plan

A finance lead at a payments company asks for one thing, in one sentence: put settlement currency on the March reconciliation report, filling it in from the processor's export, and make sure the dashboard query does not time out. An agent turns that sentence into four steps. Each step, read on its own, does what it says. The run fails anyway, on a constraint that was present in the task from the moment it was written and recorded nowhere in the plan, and it fails in a way that costs ten minutes of a locked production table before anybody hears about it.

A plan, at this point in the topic, is the object the previous chapter built: a list of steps, each naming a tool and the arguments that tool will be called with, emitted before anything executes and validated against the tool list and against policy. That validation asks whether each step is permitted. Whether the steps can happen in the order they are written is a different question, it has a different answer, and nothing in a flat list of tool calls carries the information needed to ask it. Three additional fields per step are what make it askable, and the rest of this chapter is what each one is for, what it catches, and where it stops.

The step that returned in a second and the statement that ran for eighteen minutes#

The agent produces four steps, in a schema that passes every check from the previous chapter. The tools exist, no step touches a protected target, no destructive tool is mislabelled as a read.

{"steps": [
  {"n": 1, "tool": "run_sql", "effect": "write",
   "args": {"q": "alter table ledger.charges add column settlement_currency text"}},
  {"n": 2, "tool": "run_sql", "effect": "write",
   "args": {"q": "update ledger.charges c set settlement_currency = v.ccy from staging.vendor_invoice_202603 v where v.charge_id = c.id"}},
  {"n": 3, "tool": "run_sql", "effect": "write",
   "args": {"q": "create index ix_charges_settlement on ledger.charges (settlement_currency, day)"}},
  {"n": 4, "tool": "update_view", "effect": "write",
   "args": {"view": "reports.march_recon", "column": "settlement_currency"}}
]}

Step 1 runs in 0.4 seconds. Step 2 runs in 1.1 seconds, which is the first thing that should look wrong — this is a 61-million-row table — and the tool result explains why: {"job_id": "wh-8842", "status": "RUNNING"}. The warehouse dispatches long DML — data-modifying statements, an UPDATE of this size being the usual case — asynchronously and hands back a handle. Step 3 fires 1.1 seconds later, against a table with an eighteen-minute UPDATE still writing to it, and sits behind that lock until the statement timeout kills it at 600 seconds.

Every link in that is checkable, and it is worth walking them rather than accepting the summary. An UPDATE that has to visit sixty-one million rows and write a value into each matching one cannot complete in about a second on any warehouse; that is a fact about how much data has to move. So a call that returns in about a second has not done the work. What it has done is accept the work: the RUNNING status and the job handle are the warehouse saying it has taken the statement and will tell you later. The status field is the whole message, and it says the statement is in flight rather than finished.

Two quantities are worth separating here, because they are the two the plan confuses. One is the tool call returned, which became true after about a second. The other is the statement completed, which became true eighteen minutes later. The first is a property of the request; the second is a property of the world. They are related for a synchronous tool and unrelated for an asynchronous one, and run_sql is asynchronous for statements of this size.

That is the shape of the mistake, and it is worth stating in its general form because it recurs for the rest of this topic. The call returned is a quantity that stays constant while the dangerous property varies. It reads the same whether the UPDATE took a second or eighteen minutes, whether it succeeded or was still queued, whether the table is locked or free. A criterion computed from it is computed from inputs that exclude the thing being measured. A run guarded by that criterion is not unguarded by accident or by carelessness — it is guarded by a quantity that cannot move when the thing it guards against moves.

The ordering fault then has an arithmetic consequence the transcript makes visible. The UPDATE needs eighteen minutes. The statement timeout is 600 seconds, which is ten minutes. Step 3 begins waiting on the lock a second or so after step 2 started, so the index build is killed with roughly eight minutes of backfill still to run. The index can never win that race by waiting longer, because the ceiling on its waiting is lower than the floor on the work it is waiting for. Retrying step 3 immediately fails the same way, for the same reason, with the same timing, and a retry policy tuned for transient errors will spend its entire allowance discovering that.

There are two numbers in this chapter that look like each other and are not the same quantity, and separating them now prevents a confusion later. The vendor's March export has 61,400 charge lines in it, and that count lands in staging.vendor_invoice_202603. The table being migrated, ledger.charges, holds tens of millions of rows accumulated over the life of the ledger, which is why it is described above as a 61-million-row table. One is a count of lines on one month's invoice; the other is a count of rows in a ledger. The resemblance between the two figures is a coincidence of scale and nothing else, and no calculation in this chapter turns one into the other.

The run failed on an ordering constraint that was present the whole time and written down nowhere. Step 3 depends on step 2 having finished, and the plan expresses only that step 3 comes after step 2 in a list. Those are different claims, and the gap between them is where multi-step agent runs go wrong.

The reason a reader glides over that gap is that written order and temporal order usually do coincide. In a shell script, in a notebook, in a numbered runbook a person follows by hand, line four does not begin until line three has come back, because the interpreter or the person is blocking on it. Adjacency in a list has meant "after" for as long as anybody has been writing procedures down. The belief that it still means "after" here is not a careless one, and it is wrong for one narrow reason: the thing being waited on is no longer the tool call, it is a job the tool call started.

Three fields that close the gap#

The list needs each step to carry what it waits on, what it means for it to have worked, and what state the world is left in if the run stops there.

Dependencies turn the list into a graph. depends_on: [2] says step 3 cannot start until step 2 is satisfied, and satisfied is defined by step 2's own success criterion rather than by its tool call having returned. Making the dependency explicit also makes the absence of a dependency explicit, which is how you find missing prerequisites before the run: step 2 reads staging.vendor_invoice_202603, and nothing in the plan creates that table, so a checker walking the graph can ask which step produces each input a step consumes and report the ones with no producer.

The second half of that is the part worth slowing down on, because it is the only one of the three fields that finds something the plan does not mention. A list can only be checked for what it contains. A graph can be checked for what it lacks, because a graph makes a claim of the form these are all the things step 3 waits on, and a claim of that form can be falsified by naming a thing step 3 also waits on. Turning a list into a graph converts silence into an assertion, and an assertion is something a checker can disagree with.

Success criteria are the checkable statement of what "done" means for one step. Without them, the executor's criterion is whatever the harness does by default, which is almost always "the tool call did not raise", and that is exactly the criterion that mistook a dispatch receipt for a completed backfill. A criterion is a probe: a query, a file-existence check, an API read whose result the harness can evaluate. It has to look at the world rather than at the tool's return value, because the tool's return value is what lied.

Recovery points name the states the run can legitimately be resumed from, together with how to tell whether the work up to that point is already done. Step 1 is not re-runnable — ADD COLUMN on an existing column is an error — so a naive restart after an interruption fails immediately and, worse, fails in a way that looks like the schema change is broken rather than already applied. The recovery point turns each step into something that can be asked "have you happened yet?" before it is asked to happen again.

The three answer three different questions and none of them substitutes for another. A dependency says not yet. A success criterion says that worked. A recovery probe says that already happened. Written out per step, they look like this.

The table is more interesting read as a set of gaps than as a set of levels. Every row's success criterion is a different question from its tool call, and the size of that difference is what the row is worth. Step 1's criterion is almost the same question as the call, so the row buys little. Step 2's criterion is two questions, neither of which is the call: the job's own status, and a count of rows the job was supposed to have touched. That row is where the eighteen-minute failure gets caught, and it needs both halves — the job status alone would pass on a job that succeeded while matching nothing, and the null count alone would pass on a table that was already backfilled last week.

Step 0 is not a step the model wrote. It appeared because the dependency check asked what produces staging.vendor_invoice_202603 and found nothing — which is what "expose a missing prerequisite before the run" means in practice. The March export has to be pulled from S3 and loaded before anything else in this plan can work, and the plan reads perfectly fluently without it.

The invalid-index row is worth reading twice. A CREATE INDEX that is interrupted partway leaves an index object behind that exists, has the right name, and is not usable by the planner. A recovery probe that checks for existence resumes into a state where the dashboard is silently doing sequential scans; the probe has to check indisvalid, because that is the property the step was actually trying to establish.

The causal chain under that is short and each link is checkable. Building an index writes a catalogue entry first and fills the index afterwards; an interruption between those two events leaves the entry with the fill incomplete; the database marks such an entry invalid so the query planner will not consult it; an invalid index is therefore present by name and absent by function. A probe that asks is there an object called ix_charges_settlement gets a true answer to a question nobody needed answered. Nothing raises, no query fails, and the only symptom is that the dashboard query the whole request was about goes back to scanning the table.

That generalises, and the general form is the residual to carry out of this section. A probe that checks a name is checking the cheapest observable thing that correlates with the step having run. A probe that checks a property is checking the thing the step was for. They agree on every run that completes normally, which is exactly why the difference between them is invisible until the run that matters. Every recovery probe you write has a name-shaped version that is easier to write and passes more often, and choosing it is the commonest way a step ends up guarded by a criterion that cannot fail.

The graph the flat list was hiding#

The dotted edge is the one the flat list dropped entirely. Step 2 reads from the staging table as well as writing to the charges table, so it has two prerequisites, and only one of them is adjacency in a list.

The solid edges and the dotted one are two different kinds of claim, and the diagram is drawing them differently on purpose. A solid edge is a sequencing claim the model made: do this after that. The dotted edge is a data claim recovered from the step's own arguments, because the UPDATE names staging.vendor_invoice_202603 in its from clause. Nobody had to be thorough for the dotted edge to exist. It is derivable from the SQL the step was already carrying, which is what makes it the kind of prerequisite a checker can find and a reviewer reliably will not.

Checking the graph before anything runs#

The dependency check is small, and it catches three distinct classes of problem in one pass over the plan.

def check_plan(plan, tools):
    problems, produced = [], set()

    ids = {s["n"] for s in plan["steps"]}
    for s in plan["steps"]:
        for d in s.get("depends_on", []):
            if d not in ids:
                problems.append(f"step {s['n']}: depends on missing step {d}")
            elif d >= s["n"]:
                problems.append(f"step {s['n']}: depends on later step {d}")
        if "success" not in s and s["effect"] != "read":
            problems.append(f"step {s['n']}: write step with no success criterion")

    # every input a step reads must be produced by something upstream
    for s in sorted(plan["steps"], key=lambda x: x["n"]):
        for need in s.get("reads", []):
            if need not in produced and not exists_now(need):
                problems.append(f"step {s['n']}: reads {need}, nothing produces it")
        produced.update(s.get("writes", []))

    return problems

The three classes are worth naming, because they fail for different reasons. The first is a dependency that points at nothing, or points backwards in time: d not in ids catches a step waiting on a step that was never written, and d >= s["n"] catches a step waiting on one that comes after it, which cannot be satisfied by any execution order. The second is a write step with no success criterion, which is a step that will be declared done on the strength of its tool call returning. The third is the one that found the missing table: a step reading a resource that no earlier step writes and that does not already exist.

The produced set is what makes the third check work, and its ordering is the whole trick. The steps are walked in ascending n, and a step's writes are added to produced only after that step's reads have been checked against it. So the set always holds exactly what is available before the step under examination, which is the state the step will actually meet. Adding the writes first would let a step satisfy its own prerequisite, and every plan would pass.

exists_now is the part that does real work and the part that is easy to get wrong. It asks the world whether a resource is already there — the staging table might exist from yesterday's run, in which case step 0 is genuinely unnecessary. Asking the world at plan time is a snapshot, and it is a snapshot that can go stale before the step that relies on it runs. That is the crack the next two chapters widen.

Run it against the four-step plan and it emits one line:

step 2: reads staging.vendor_invoice_202603, nothing produces it

Which is the eighteen-minute failure, found in eleven milliseconds, before the ALTER TABLE ran.

The cost of that check and the cost of the failure it prevents are the whole argument for checking, and the gap between them is worth reading rather than skimming. The check costs about eleven milliseconds. The failure it prevents costs ten minutes of a production table held under a lock, a schema change already applied that somebody now has to reason about, and a person deciding at speed whether to wait or intervene. That is a ratio in the tens of thousands before you count the human, and it holds for every plan of this shape, not just this one. A check whose cost is measured in milliseconds and whose benefit is measured in minutes of locked production does not need a business case.

Restarting a half-finished migration#

Two ways the model gets the graph wrong#

The commonest is over-serialisation. Asked for dependencies, models tend to emit depends_on: [n-1] for every step, which is the flat list wearing a graph's clothes — it satisfies the schema, passes the checker, and encodes none of the actual constraints. The tell is structural and takes one line to detect: if no step has more than one dependency and no two steps share a dependency, the graph is a chain and the model has transcribed rather than analysed.

edges = [d for s in plan["steps"] for d in s.get("depends_on", [])]
is_chain = len(edges) == len(set(edges)) and all(
    len(s.get("depends_on", [])) <= 1 for s in plan["steps"])

The two conditions in that expression are testing two different ways a graph can fail to be one. len(edges) == len(set(edges)) says no step is depended on twice, which rules out anything joining back together. The all(...) says no step depends on two things, which rules out anything branching apart. A structure with neither joins nor branches has exactly one path through it, and one path through it is a list. The check does not read a single tool name or argument, and it does not need to.

A chain is not always wrong — the migration genuinely is mostly sequential — but on a task with independent branches it costs wall-clock, because steps that could run together are serialised for no reason. It also hides the missing prerequisite, since a chain has an answer for "what does step 2 wait on" that looks plausible and says nothing about the staging table.

The second failure is granularity. A step should be the largest unit of work with a single checkable success criterion and a single recovery probe. "Migrate the table" is too big, because no probe answers whether it happened. Splitting ALTER TABLE into "prepare the statement" and "execute the statement" is too small, because both halves share one probe and the first has no observable effect. Those two tests are usually enough to settle it: if you cannot write the probe, the step is too coarse; if two adjacent steps would use the same probe, they are one step.

Both tests are the same test seen from two sides, which is why they settle so many cases between them. A step is the right size when there is exactly one observable fact about the world that becomes true because of it. Too coarse and there are several such facts, so no single probe covers the step. Too fine and there are none, so the probe has to borrow its neighbour's.

A criterion is not a test, and a recovery point is not a transaction#

A success criterion is not a test. A test asserts that the system behaves correctly; a criterion asserts that one step reached the state it was aiming for. Step 2's criterion says every matched row now has a currency, and it says nothing about whether those currencies are right — a backfill that joined on the wrong key satisfies it perfectly. Criteria catch steps that did not complete. They do not catch steps that completed and were wrong, and the distinction matters because a plan full of green criteria reads like a successful run.

The wrong-key case shows why that gap cannot be closed by writing a stricter criterion. The criterion counts rows whose settlement_currency is null. A join on the wrong key still writes a currency into every matched row, so the null count still reaches zero, and it reaches zero by exactly the same route it would have taken had the join been right. The criterion is computed from inputs that do not include the join condition, which is the thing that was wrong. No threshold on the null count fixes that, because the null count is not a function of the property in question.

Recovery points are not transactions. A transaction gives you atomicity: either everything happened or nothing did, and the database enforces it. Recovery points give you idempotent re-entry: each step can be asked whether it already happened and skipped if so. Nothing rolls back. If the run dies after step 3, the column exists, the data is backfilled, the index is built, and the view still points at the old definition — a real, half-migrated state that somebody has to be willing to leave the system in. That is a design decision, not a free property, and for steps whose partial state is genuinely intolerable, a transaction inside a single tool call is the right tool and a recovery point is not.

Naming that as a design decision rather than a consequence is the point. Nothing about model-authored plans requires half-migrated states to be acceptable. What requires it is choosing to compose a task out of separately committed steps, which is a choice made for a reason — the steps are long, they belong to different systems, and holding one transaction across all of them is either impossible or ruinous. The half-migrated state is the price of that choice, and it should be paid deliberately by somebody who has looked at what the system looks like in each intermediate state, rather than discovered at 3 a.m. by whoever is on call.

And the graph is not a workflow definition. It has the shape of one — nodes, edges, guards — and it is checked before execution the way a workflow would be. What it does not have is any of the guarantees an engineered workflow has, because it was generated for this request by a model, and both the dependency edges and the success criteria are the model's claims about the world. The checker verifies that those claims are internally consistent. Nothing so far verifies that they are true.

That last sentence is worth holding onto in its exact form, because it is easy to hear as weaker than it is. Internal consistency is checkable without leaving the process: it is a property of the plan document. Truth is a property of the warehouse, the bucket and the vendor, and reaching it means issuing a request and reading what comes back. Everything in this chapter is the first kind. The exists_now call is the single exception, and even it only asks the world once, at plan time, about a world that keeps moving afterwards.

Everything in this chapter happens before the first tool call. The graph is validated, the missing prerequisite is found, the criteria are written down, the recovery probes are specified. Then the run begins, and the plan sits in context as a document while the executor walks the list.

The plan cannot answer, about itself, what consults any of this while the run is happening. A depends_on edge only means something if something waits on it. A success criterion only means something if something evaluates it. A recovery probe only means something if something calls it before re-entering a step. Write all three, hand the plan to an executor that iterates over plan["steps"] and calls each tool in order, and you have produced a very well-documented list that the machine is ignoring.