Stopping early needs a criterion, not a feeling of sufficiency

Most agent runs do more work than the task needed. The loop keeps going because the model keeps finding something else worth checking, and on the easy end of a workload that thoroughness is pure cost: three extra steps on a ticket that was settled at step four. A stopping rule is the mechanism for ending those runs early, and it is the cheapest saving in this topic to implement and the easiest to implement in a way that quietly breaks the agent.

The question a stopping rule has to answer is whether the required work has happened. There are two places to get that answer from: the model, by asking it, or the run's own state, by checking it. The first is a few lines and reads as obvious; the second takes a conversation with whoever owns the policy. The difference between them is the whole of this chapter, and it is not a difference of rigour so much as a difference in what each one is capable of observing.

Why the model's own verdict cannot carry this#

Asking a model whether its work is complete asks it to compare what it produced against a specification it holds only as an impression of the instructions. The comparison it can actually make is between the output and what a good output looks like, and fluent, well-structured, internally consistent output looks exactly like good output.

Follow that to the failure and it stops being surprising. A step that was never run leaves nothing in the context — no tool call, no result, no error — so the transcript of a run that skipped a check and the transcript of a run that never needed it are not distinguishable from inside. The model is answering a question about coverage using a record that contains only what was covered. That is why the failure mode is so uniform: the early stops in one shipping-exception workload were not scattered across the run, they clustered at the first moment the transcript contained a coherent narrative, which was usually right after the first tool that returned a satisfying fact.

The cheapest-looking version of the mechanism is a question added to the end of each step: Do you have everything you need to resolve this ticket? Answer YES or NO. On YES the loop skips to drafting the reply. Deployed on that workload it cut mean steps per successful run from 7.0 to 4.9 and cost per successful task from $0.105 to $0.078, which is a 26% saving from about forty lines of code. Task success fell from 90.7% to 85.4%, and the transcripts of the new failures were the most fluent output the agent had ever produced.

Here is a representative one, at step three of what should have been a six-step run:

assistant: I have confirmed that shipment SH-4471203 was marked delivered on
2026-03-14 while the customer reports non-receipt, and the carrier's proof of
delivery has no signature capture. This is sufficient to establish a valid
claim under the shipping policy. YES.

--- loop skips to draft_reply ---

assistant: Hi Marta, I'm sorry your parcel didn't arrive. I've issued a credit
of $84.50 to your original payment method, which should appear within 3-5
business days.

The reply is correct in tone, correct in grammar, and correct about the parcel. It is also a credit issued without running credit_eligibility, which is the step that would have found that this customer had three claimed non-deliveries in sixty days and that the account is flagged for manual review above $50. The model was not lying when it said YES. It had genuinely established the facts it had gone looking for, and it had no representation at all of the checks it had not performed.

The correlation that matters was measurable on a replayed set. Among runs where the model said YES, the mean self-reported confidence was 0.91 for runs that later passed review and 0.88 for runs that were rejected — a separation of three points on a scale where the noise between reruns of the same ticket was five. Put those two numbers side by side and the verdict is arithmetic rather than judgement: a signal whose difference between the two classes is smaller than its variation within one class cannot separate them, and running it on more tickets does not help, because the noise is a property of each measurement rather than of the sample. Fluency and confidence move together; coverage moves independently of both. A signal that cannot distinguish the two cases cannot be the thing that decides whether to skip a step that moves money.

There is a narrower version of the same mistake worth naming, which is treating the provider's stop_reason: "end_turn" as a completion signal. It means the model finished its message without requesting a tool. It says nothing about whether the task is done, and an agent that exits its loop on end_turn alone will exit whenever the model chooses to write a paragraph of commentary instead of a call.

Predicates over run state#

A stopping rule is usable when a person who has never read the transcript can evaluate it, which in practice means it is a set of predicates — checks that read the run's structured state and come back true or false, with no model in the loop — rather than a judgement passed over prose. Four kinds cover most workloads.

Required fields being populated is the crudest and the most reliable: the run cannot stop unless shipment_id, eligibility_verdict and credit_amount are all set by the steps that own them. Citations resolving is the version for retrieval-shaped work: every claim the answer relies on carries a document id and a span, and the rule checks that each one exists and that the quoted text is actually at that offset. A test or validator passing is the version for anything executable, and it is the strongest signal available when you can get it. A calibrated confidence score is the weakest of the four and is sometimes the only one available, in which case it needs the treatment in the next section.

The first three share the property the model's verdict lacks. Each of them reads something a step wrote, so a step that did not run leaves the predicate false rather than leaving nothing at all. Absence becomes observable by arranging for presence to be recorded, which is the whole trick.

@dataclass
class StopDecision:
    stop: bool
    satisfied: list[str]
    missing: list[str]

REQUIRED = {
    "shipment_id":        lambda s: s.shipment_id is not None,
    "eligibility_run":    lambda s: s.eligibility is not None,
    "amount_within_cap":  lambda s: s.credit_amount is not None
                                    and s.credit_amount <= s.policy_cap,
    "claims_in_window":   lambda s: s.recent_claim_count is not None,
    "citations_resolve":  lambda s: all(policy_store.span_exists(c.doc_id, c.span)
                                        for c in s.citations),
}

def should_stop(state: RunState) -> StopDecision:
    satisfied = [name for name, check in REQUIRED.items() if check(state)]
    missing = [name for name in REQUIRED if name not in satisfied]
    return StopDecision(stop=not missing, satisfied=satisfied, missing=missing)

Nothing in that function consults the model. It reads fields that earlier steps wrote and a document store that either contains the cited span or does not, so its verdict is reproducible from a stored run record months later — which matters more than it sounds, because it means a dispute about whether a run should have stopped is settled by re-running the function rather than by re-reading a transcript and forming an impression.

The missing list is as useful as the boolean, for two reasons. It goes back into the context as a directive telling the model precisely which check remains, which is a better prompt than a general instruction to keep working. And its distribution over a week tells you which predicate is doing the work: on this agent, 63% of non-stops were waiting on claims_in_window alone, which is a cheap call and suggested moving it earlier in the pipeline where its result would be available sooner.

The rule is genuinely an early termination and not a checklist walkthrough, because a run that has all five predicates satisfied at step four stops at step four. The gain comes from tickets where the answer is straightforward — a single shipment, a small amount, no claim history — and the model would otherwise have kept going out of thoroughness. Those are 38% of the workload and they now finish in four steps instead of seven.

Calibrating a score, when a score is all you have#

Some steps have no checkable artefact. A step that judges whether a customer's free-text description matches a known carrier failure pattern produces a judgement, and the only thing available to threshold is a number the model emits. That number is usable, and it becomes usable specifically by being measured against labelled outcomes rather than trusted at face value. Take a few hundred runs with known verdicts, bucket them by the emitted confidence, and read off what each threshold buys and costs.

The table is the argument for thresholds over verdicts. The model's own YES fires on four runs in five and is wrong on one firing in six; a threshold at 0.90 gives up a third of the savings and reduces the error rate by a factor of eight.

Read the columns as rates of change rather than as levels and the shape of the trade is visible. Between each pair of adjacent thresholds the firing rate rises by something like fifteen points, while the error rate roughly doubles. Cost, meanwhile, moves by fractions of a cent. So each step down the table buys progressively less and costs progressively more, which is what an interior optimum looks like before anybody has decided where it is. The bottom row is what a forty-line "ask the model" change actually deploys, and the reason it looks so good on cost is visible in the same row: it saves the most precisely because it stops the most often, including on the runs that needed the remaining steps.

Two cautions about the curve. The mapping from a score to an error rate is specific to a model version and a prompt, so it has to be re-measured when either changes, and teams that skip the re-measurement carry a threshold tuned for a model that no longer exists. And the numbers are only meaningful if the labels are the fixed success criterion from the first chapter; a threshold calibrated against "a reviewer thought it looked fine" inherits whatever that reviewer's Friday afternoon was like.

Reporting both sides of what it did#

A stopping rule has a saving and a truncation, and reporting one without the other is how a rule that cost five points of success survived a review. The saving is steps, cost and latency. The truncation is runs where the rule fired and the outcome was later rejected, expressed as a rate over firings.

That denominator is a deliberate choice and it is what makes two rules comparable. Expressed over all runs, a rule that fires rarely looks safe merely for being rare, and a rule that fires often looks dangerous merely for being used. Over firings, the number answers the question somebody actually has — when this rule acts, how often is it wrong — and it is independent of how much of the workload it touches.

The middle row is not free either, and the 0.9% is worth understanding rather than rounding away. Those are runs where all five predicates were satisfied and the answer was still wrong, which means the predicate set is incomplete — in this workload, four of the nine cases involved a shipment that had been re-routed after the eligibility check ran, so the state was stale rather than missing. That is the honest limitation of the technique: predicates prove that the required work happened, and they do not prove the work was right, and they say nothing at all about whether it is still true. A stopping rule cannot be the only thing standing between the agent and a bad answer, and treating it as one produces a system that is rigorous about coverage and blind about correctness.

Where the rule is checked matters as much as what it checks. Evaluating it after every step catches the earliest possible stop and costs nothing beyond a few field reads, so there is no reason to batch it; evaluating it only at fixed points, which some frameworks encourage by exposing a single post-loop hook, means a run whose predicates went green at step four still pays for steps five and six. The one ordering constraint is that the rule runs after tool results have been folded into the run state and before the next model call is constructed, since checking it against state that is one step stale is how a rule that looks correct in review fires a step later than it should on every single run.

A stopping rule bounds nothing#

A run where the predicates never all go green will keep going, and the rule has no opinion about that; it is a rule about when work may stop being necessary, not about when work must stop being permitted. Those two rules fire in opposite circumstances — one when the run has gone well enough, the other when it has gone wrong — and giving them the same implementation, which is what happens when someone reuses the step counter for both, produces a system that cannot tell a good outcome from an exhausted one.

It also does not reduce work on the tasks that were expensive, and that follows from where its saving comes from. The rule fires when every required check has been done, so it fires early on tickets whose checks are quick and few. The tickets that ran to seven steps still run to seven steps, because their predicates go green late or not at all, so the saving comes entirely from the easy end of the distribution and leaves the tail exactly where the first chapter found it.

Which means the expensive tail is still unbounded. Something else has to be responsible for a run that has stopped making progress and is still calling tools, and that something is not allowed to look like an answer when it fires — because a control that ends a run and returns whatever the transcript contains has reproduced, at the far end of the distribution, precisely the failure this chapter spent its length removing from the near end.