An 18% token saving measured only on requests that selected correctly

The slide has three rows on it. Two hundred real requests, replayed against two configurations of the same agent: arm A loads skill bodies on demand, arm B concatenates every body of every scoped-in skill into the system prompt upfront. Average prompt tokens fall 18% under deferral, from 14,200 to 11,640. Median latency rises 210 ms, because deferral spends round trips. Task success is 86.5% against 87.0%, a difference of one request in two hundred, which is inside the noise of a 200-request sample. The recommendation on the slide is to keep deferring, and the recommendation is probably right.

The number that is wrong is the third one, and it is wrong in a way that no amount of increasing the sample size repairs. Task success was computed over the requests where a grader could judge the answer against the skill that should have handled it — which in the harness that produced this slide meant the requests where a skill was selected and a run completed. Requests where the model looked at the resident descriptions, matched nothing, and answered from general knowledge produced a response with no skill attached, and the harness had no rubric to grade them against, so they were dropped before the mean was taken. There were 23 of them in arm A and 9 in arm B.

The denominator is the finding#

Dropping unmatched requests is not a coding error somebody made carelessly; it is what happens naturally when the evaluation is organised around skills. You have a labelled set that says request 61 should be handled by webhook-diagnostics, you run it, you compare the answer to what a correct application of that skill produces. When nothing is selected there is no skill to score against, and the row falls out of the join.

The population looks like this, and the shaded region is what the experiment can see:

Under deferral, 23 of 200 requests reached the leftmost leaf. Under upfront loading only 9 did, because arm B's prompt contained every body, so the model had the full text of each skill rather than a phrase, and the requests whose vocabulary the description missed were still recoverable from the body. The 18% token saving and the flat success rate are both computed on the 177 and the 191 respectively — different denominators, quietly, in the two arms being compared.

Redo the arithmetic over all 200 and the picture moves. Counting an unmatched request as a failure, arm A scores 132 + a few partial credits out of 200, call it 71%, and arm B scores 78%. That is a seven-point gap that the original comparison reported as one request of difference. Whether it is real depends on how you grade a general-knowledge answer, and some of those 23 were fine — a request about the deploy freeze does not need a skill. The point is that the metric as computed could not have detected the difference at any sample size, because the affected requests were removed before it was taken.

The reporting fix is to state the denominator every time the saving is quoted, and to publish the no-selection rate next to it. "Deferred loading cuts average prompt tokens 18% with no change in task success" is a claim that will be repeated in three meetings and a design doc. "Deferred loading cuts average prompt tokens 18%; task success is unchanged on the 88.5% of requests where a skill was selected, and the no-selection rate is 11.5% under deferral against 4.5% under upfront loading" is the same measurement with the part that matters attached to it.

Measuring the three numbers so they can be compared#

The comparison has to be paired — the same request through both arms — because request difficulty varies far more than the treatment effect does. Prompt tokens should be summed across every model call in a run rather than taken from the first, since counting only the initial prefill is the mistake that makes deferral look free; the round trips are where its cost lives. Latency is wall-clock from request to final token, including tool execution.

def compare(requests, run_deferred, run_upfront, grade):
    rows = []
    for req in requests:
        for arm, runner in (("deferred", run_deferred), ("upfront", run_upfront)):
            t = runner(req)
            rows.append({
                "request_id": req.id,
                "arm": arm,
                # every call in the run, not just the first prefill
                "prompt_tokens": sum(c.usage.input_tokens for c in t.calls),
                "calls": len(t.calls),
                "latency_ms": t.wall_ms,
                "selected": t.selected_skill,          # None is a result, not a gap
                "fetched": sorted(t.files_read),
                # graded against the request's intent, never against a skill,
                # so a no-selection run still receives a score
                "score": grade(req, t.final_text),
            })
    return rows

The comment on the last field is the whole correction. A grader that takes the request and the answer, and asks whether the answer is one a competent engineer would accept, produces a score for every run including the ones with no skill attached. A grader that takes the skill's expected procedure and checks conformance produces a score only where a skill was selected, and silently defines the population.

The upfront arm needs one caution of its own, because it is easy to treat it as the ideal that deferral approximates. Loading every body means the model answers with 40 skills' worth of instructions in its context, some of which contradict each other and most of which are irrelevant to the request, and that has a cost on answer quality separate from the token bill — the same dilution effect the context-selection material covered. On this comparison arm B's advantage came entirely from the requests where arm A selected nothing; on the requests where both selected the same skill, arm B was fractionally worse. Upfront loading is not a ceiling on quality that deferral falls short of, and reporting it as a control rather than as a target keeps that straight.

Two other things belong in that row and are cheap to record. The calls count exposes round trips directly, which is what you will want when the latency difference needs explaining. The fetched list is what makes the next section possible.

The result that justifies inlining one specific body#

An aggregate comparison tells you whether the architecture is defensible. It does not tell you which skill to change, and the decision that actually gets made is per skill. Group the deferred arm's runs by selected skill, split each group by whether the bundled file was fetched, and look at the two success rates:

The result that stops deferral for a particular body is the fourth column being materially worse than the third: answers are worse on the runs where that body was not fetched. That is a within-skill comparison against a within-skill control, so it is not confounded by which requests route to which skill, and a gap of 50 points on 41 runs is far outside anything a sample that size produces by chance. changelog-entry and deploy-preview-env both show it, for the two different reasons the previous chapters separated — one is a constraint the output must satisfy, the other is state the run must establish.

refund-partial-crossledger and log-search show the opposite and are the reason not to inline everything. Their deferred files are consulted rarely and the runs that skip them do fine, which is what a genuine lookup looks like in the data. Inlining log-search's 1,600-token reference would charge 45 runs for something 5 of them needed.

incident-timeline is the row that decides on latency instead of correctness. Its file is fetched on 88% of runs and the runs that skip it are no worse, so there is no correctness case either way; what there is, is 26 of 30 runs paying a 540 ms round trip to defer 400 tokens. Inlining that one is a pure latency win with a small token cost, and it is a decision the correctness columns alone would have called a tie.

One guideline, and the number it refuses to give#

Two teams arrive at this measurement with opposite priors — one capping bodies at 200 tokens and deferring everything else, one writing 2,000-token bodies with no bundled files — and each can show that their approach performs better on their own traffic, because their traffic has different fetch-rate distributions. A single guideline has to be true for both, which rules out any rule expressed in tokens.

What survives is this: content whose absence changes what the agent does belongs in the body, and content the agent consults for a subset of cases belongs in a file; where you are unsure, ship it deferred, measure the fetch rate and the success gap, and inline anything whose gap is material. The guideline deliberately does not name a size threshold, and it does not name a fetch-rate threshold either. It names two measurements and the direction they push, because the crossover depends on the round-trip cost and the token price of the deployment, both of which change when the model changes.

The asymmetry in how the two failures announce themselves is what makes the guideline lean the way it does. A body that is too fat shows up as a token line on a dashboard and a latency number, both of which somebody reviews monthly and can attribute to a specific skill. A body that is too thin shows up as an answer that was fluent and missing a constraint, on a fraction of runs, with no error and a slightly lower token count than a correct run. The first failure is expensive and visible; the second is cheap and invisible. Given equal uncertainty, the error to prefer is the one your monitoring can see.

What to do with this on Monday#

The library in front of you already has a resting cost, and the first chapter of this arithmetic is a number you can compute today from the repository without shipping anything. Do that, then run the labelled-request eval at your current catalogue size so the accuracy floor has a baseline rather than a target invented later. Add the static path check to CI, because it is deterministic and finds real breakage in an afternoon.

Then pick the three skills with the highest selection volume and instrument their fetch rates. That single number, per skill, is the input to almost every decision in this topic: it tells you whether a deferred file is a lookup or a prerequisite in disguise, whether a round trip is buying anything, and which body to inline first. Everything else here — the tiers, the resting-cost ceiling, the routing architectures, the trace assertions — is scaffolding around getting that number and acting on it.