Showing a rescue means replaying the same context pressure

Harborlight runs a support agent for a subscription billing product, and in the sprint review someone demonstrated the new reconciliation subagent. They opened a session, pasted the customer's question — my March invoice looks wrong, I think I was charged twice — and the agent delegated, got back a clean verdict with three event IDs, and answered correctly in nineteen seconds. The room was convinced. Someone then asked what the main loop did on the same input, so they tried it, and the main loop also answered correctly, in eleven seconds, for two-thirds of the tokens.

The demo was not rigged and it did not show anything, because the delegation was never supposed to help at turn two. The problem it was built for happens at turn forty, when the agent has been discussing a refund policy, a failed card, and a plan change for twenty minutes, and reading three months of billing events into that window pushes the customer's actual question far enough back that the agent starts answering a different one. A claim about context pressure is a claim about a particular moment in a particular conversation, so any demonstration that starts from an empty window has removed the variable under test.

Replay the transcript, then branch#

The comparison that does demonstrate it starts from a session that failed. Harborlight had one in their logs: conversation c_8841, forty-one turns, in which the agent read 118 billing events, then told the customer that their plan change was the cause of the discrepancy, which was true of a different customer's ticket it had handled earlier in the same session.

The two branches are the arms of the experiment, in the sense the word carries in a controlled trial: identical conditions up to the branch point, one deliberate difference after it, and every number recorded on both sides. The word doing the work is replay. You are not re-running the loop from turn one, because a stochastic loop re-run diverges within a few turns and you end up comparing two arms that had different histories. You reconstruct the message array from the log and use it directly as the starting state for both arms, so the pressure at the branch point is byte-identical.

def branch_point(session_id: str, upto_turn: int) -> list[Message]:
    """Exact message array as it stood before the turn that failed."""
    msgs = load_transcript(session_id)          # includes tool results
    return msgs[:index_of_turn(msgs, upto_turn)]

BASE = branch_point("c_8841", upto_turn=41)
TASK = "Reconcile this customer's billing events for Jan-Mar and tell them what happened."

def arm_inline() -> Trial:
    msgs = BASE + [user(TASK)]
    return run(msgs, tools=ALL_TOOLS)           # events land in this window

def arm_delegated() -> Trial:
    msgs = BASE + [user(TASK)]
    return run(msgs, tools=ALL_TOOLS + [reconciler_subagent])

trials = [(arm_inline(), arm_delegated()) for _ in range(20)]

Twenty pairs rather than one, because a single trial of a sampling process is an anecdote and the failure being studied is intermittent. Harborlight had picked c_8841 precisely because the inline version failed there, so a single re-run would have been selected on the outcome it was meant to measure. Running both arms twenty times from the same base gives a failure rate rather than a story.

Everything above depends on having the failing session on disk in a replayable form, which is the part most teams discover they are missing at the moment they want it. A transcript that records only the user and assistant messages is not enough, because the tool results are most of the tokens and all of the pressure; the log has to contain every message the model saw, in order, including tool calls and their outputs, with the token counts as they were charged. Harborlight had this by accident, since they were already persisting full message arrays for a compliance requirement, and teams without it typically start the experiment by turning on logging and waiting two weeks for a failure to reappear. Sampling parameters belong in the log too, because a comparison run at a different temperature from the session it claims to reproduce is not a reproduction.

Four things get recorded per trial, and each of them answers a different objection.

Tokens arriving in the parent is the quantity the delegation exists to reduce, measured as the growth of the parent's window between the branch point and the final answer. Total tokens billed across both parent and child is the quantity finance cares about, and it moves in the opposite direction. Wall-clock from the branch point to the customer-visible answer is what the customer experiences. Correctness is a verdict graded against a human-labelled answer for that ticket, which for a reconciliation means the right event IDs and the right net figure, not a plausible paragraph.

Grading forty trials sounds like the expensive part and is not, because the output contract made it mechanical. The delegated arm returns event IDs and a net figure, so correctness is a set comparison against the labelled answer plus an integer equality, which is forty lines of Python. The inline arm returns prose, so its answers had to be graded by extracting the event IDs it cited and comparing those, and the four trials where it cited no identifiers at all were marked wrong on the grounds that an answer a human cannot check is not an answer that can be sent to a customer. Labelling the ground truth for five tickets took one support lead about forty minutes. That is the whole human cost of the experiment, and it is worth noticing that the arm with the tighter contract was also the arm that was cheaper to evaluate, which tends to be true generally.

Reading the result honestly#

The first four rows are the ones people quote and the last one is the one that decides it. Tokens are up 35% and latency is up twelve seconds, both real, and the arithmetic that matters divides by the correctness rate, because a wrong reconciliation is not a cheap answer. It is a support ticket that gets escalated, a human who re-does the work, and sometimes a refund issued against the wrong charge. At six correct out of twenty, the inline arm costs $0.14 for every answer you can actually send, against $0.063 for the delegated arm, so the delegation is a little over twice as cheap per unit of useful output while being 35% more expensive per call.

So Harborlight shipped it, and the honest version of that decision includes what would reverse it. If the inline arm's twelve failures had clustered on one shape — say, every failure involved a mid-conversation plan change — the fix might be a deterministic pre-filter that drops irrelevant event types before they enter the window, which costs no model call and would make the delegation redundant, and that is the first thing to check because it is the cheapest thing to check. If the twelve seconds sits in a synchronous chat path, it needs to be weighed against measured abandonment rather than against intuition; Harborlight's data said median human agent handoff was ninety seconds, so twenty-six was comfortable, and a team whose users abandon at fifteen seconds gets a different answer from the same table. And if fixing the inline arm's prompt — restating the customer's question immediately before the reconciliation step — recovers most of the correctness gap, the comparison has to be re-run against that improved baseline, because the delegated arm should be compared against the best cheap alternative rather than against the version that happened to be in production.

One proposal that came up in the review would have shown up immediately in the table, which is a good argument for building the table before the review. The suggestion was to have the parent re-read the billing events to confirm the child's verdict before answering the customer, as a safety measure. Run that variant through the same harness and the first row goes from 830 tokens back to roughly 42,000, the wall-clock adds the child's twenty-six seconds on top of the inline arm's fourteen, and the correctness rate drifts back toward the inline arm's because the parent is once again answering with the events in its window. Verification that re-reads the child's inputs is not a safety measure on top of the delegation; it is the inline arm with an extra bill attached. Where the child's output cannot be trusted, the contract is the thing to change.

The other thing worth doing before shipping is checking that the twenty pairs came from more than one conversation. A result established on c_8841 alone is a result about c_8841. Harborlight replayed four failing sessions and one session that had succeeded inline, and the fifth is the important one: on a conversation where the main loop had answered correctly, the delegated arm was correct 19/20 and cost 31% more, which is the honest cost of applying the delegation where it was not needed. That number is what makes the routing question in the next chapter concrete rather than theoretical.

What makes a task demonstrable at all#

Not every delegation can be shown to help by this method, and when the method shows nothing, that is usually a fact about the task rather than a flaw in the experiment. The demonstration works when the delegated unit's intermediate material genuinely competes with the parent's objective, which requires two properties at once: the bulk has to be large relative to the window, and it has to be superficially relevant enough that attention lands on it. Three months of billing events have both — 41,200 tokens of plausible, on-topic, numeric detail sitting between the customer's question and the answer.

A task that reads 41,200 tokens of an unrelated vendor PDF is large and not competing, so the inline arm mostly ignores it and both arms score the same. A task with twenty small tool calls totalling 3,000 tokens is competing and not large, so there is nothing to isolate. Either way the two arms come out level, and the correct conclusion is that this particular delegation is not earning its keep, which is a useful result to have measured rather than a failed experiment.

Two limits on what any of this proves. The replay fixes the history, so the comparison is valid only for the system as it was configured at that moment; change the system prompt, the model version, or the tool set, and the base transcript no longer corresponds to a session that system would have produced. And a paired comparison at one pressure point is not evidence that the delegation helps at turn five or turn eighty, which is why Harborlight's fifth replay — the one where the main loop was fine — was worth running.

That fifth result is the awkward one. The delegation is right at turn forty and wasteful at turn two, and the runtime has to decide which situation it is in every time, with no measurement available at the moment of the decision.