Just-in-time retrieval and latency

Just-in-time retrieval is the design where the model fetches material during generation instead of being handed it up front: it reads the question, decides it needs something, calls a tool, and is called again with the result appended. The alternative, pre-loading, pastes the material into every request whether or not it is needed. Both are selection decisions, and the token counts make the choice look obvious in one direction.

The token counts are the wrong axis. What just-in-time actually buys is that questions needing none of the material stop carrying it; what it costs is that questions needing it now take two model calls, and a percentile budget is decided by the slow population rather than the fast one. The figures below come from an agent answering engineers' questions about an internal platform API with 300 endpoints, against a p95 latency budget of three seconds agreed with the platform team whose IDE plugin calls it.

Three costs, and which side pays each#

The reference that documents those endpoints is 30,000 tokens: a path, a method, a parameter table and two worked examples for each. Version A, the one in production, pastes the entire reference into every request. Version B gives the model a search_endpoints tool and lets it fetch the one endpoint it needs. On the median question, B sends 2,400 input tokens where A sends 32,400. A comes in at 2.8 seconds p95 and B at 3.5.

Version A pays two things that B does not. The first is prefill on every request. A question like "is the sandbox environment down again?" needs none of the 300 endpoints, and roughly 40% of the traffic is that kind of question — status, auth setup, rate-limit policy, "who owns this service" — yet every one of them prefills 30,000 tokens of endpoint documentation before the model produces a character. Money is not the issue, for the reason the cache arithmetic already covered: the reference is a stable prefix, it sits behind a cache_control breakpoint, and it bills at a fraction of the list rate. Time is a different account. A cache read is faster than a fresh prefill and it is not free.

The second cost A pays is dilution, which the crossover work measured in a different setting and which behaves identically here. Of the 300 endpoints, 41 are paginated list endpoints whose parameter tables are near-identical — limit, cursor, expand[], order — differing in one or two fields. Version A puts all 41 in front of the model at once. On a frozen 240-question eval, version A scores 0.81 overall and 0.62 on the subset of questions naming an endpoint that has a near-twin, because the model answers with a parameter that belongs to the sibling. Version B scores 0.79 overall and 0.86 on that subset, since the search returns one endpoint and there is nothing to confuse it with.

The cost B pays is the one that decides the latency question, and it is structural. Retrieving just in time means the model has to notice it needs something, emit a tool call, wait for the result, and then be called again with the result appended. One model call becomes at least two, and the second one prefills everything the first one did plus the tool call and its result.

"Structural" is carrying a mechanism there, and it is the reason a shorter prompt can be a slower one. A model call has two phases that scale differently. Prefill reads the whole input before any output appears, and it works over the input positions together, so doubling the input does not double the time it takes. Decode produces the answer one token at a time, each token conditioned on the ones already emitted, so it cannot be worked on in parallel and its duration is set by the length of the answer rather than of the input. Pre-loading buys its extra tokens in the phase that parallelises and adds nothing to the phase that does not. Just-in-time cuts the first phase and then adds an entire second call, which is a second prefill and a second decode. The second prefill is not a continuation of the first, because the model retains nothing between calls: it re-reads the original prompt, the tool call and the result from the beginning. The design sending fewer tokens is paying for a phase twice that the design sending more tokens pays for once.

The diagram is the whole argument for why B is slower on the questions it is supposed to be better at. Nothing in that sequence is wasteful; every hop does real work. There are simply more hops, and two of them are model calls with their own time-to-first-token and their own decode.

The prompt sizes in the diagram add up, and following them is worth thirty seconds because it shows what the second call is actually re-reading. The first call carries 2,400 tokens: the question, the instructions and the tool definitions, with none of the reference in it. The model replies with a tool call of 45 tokens. The index returns one endpoint at 700 tokens. The second call is the first call's prompt plus both of those, which is 3,145 — the figure in the diagram — and every one of those 3,145 tokens is prefilled from scratch, the original 2,400 included. Compare that with the pre-loaded arm, whose single call carries 32,400 tokens because it is the 30,000-token reference plus the same 2,400 of everything else. The token comparison everybody quotes, 2,400 against 32,400, is real: it is about 92% fewer. It is also a comparison of the first call in one design against the only call in the other.

Putting numbers on the two paths#

The measurements below come from 2,000 replayed production questions against both arms, at a decode rate of about 110 tokens per second, with answers of 150 tokens at the median and 260 at the 95th percentile. Two of the row labels are worth reading precisely, because the whole comparison is built from them. Time to first token is everything that happens before a single character comes back — the network hop, any queueing, and the prefill over the assembled prompt — and it grows with the length of the input. Decode is the generation that follows, one output token at a time, and it grows with the length of the answer instead.

Two of those rows are the decode rate applied to two different answers, and computing them is the check that the table is a measurement rather than a guess. At about 110 tokens a second, a 150-token answer takes a little over a second and a third — the 1,360 ms in both of the full-answer decode rows. The 45-token tool call at the same rate takes about 410 ms, which is the third column's first decode. Nothing about the model changed between those two cells; only the length of what it was writing. The p50 rows are then the sums of the cells above them: 330 plus 1,360 for A, 130 plus 1,360 for B when no endpoint is needed, and 130 plus 410 plus 55 plus 150 plus 1,360 for the two-call path, which is 2,105.

The gap between 330 ms and 130 ms in the first row is the pre-loaded reference being read, and it is the one cost the token argument gets right. Even from cache a 30,000-token prefix has to be brought in before the first output token can appear, and that shows up as two hundred milliseconds of extra time to first token on every single request, including the ones that need none of it. The just-in-time arm genuinely does not pay it. It then pays 410 ms of decode, 55 ms of index query and 150 ms of a second time to first token to get to the same place, which is why the arm that avoided the expensive prefix is behind by the time the answer starts.

Version B is 200 ms faster on the 40% of questions that need no endpoint, which is a genuine improvement and the thing the proposal was built on. It is 415 ms slower at the median and 730 ms slower at p95 on the other 60%, and that is where the budget is decided, because p95 is a percentile of the whole traffic mix rather than of a chosen slice. With 60% of requests on the two-call path, the 95th percentile of all requests falls inside that population — around its own 92nd percentile — so the fast 40% never touches the number being measured. Improving the median of a distribution by shortening its short half does not move its tail.

All three of those differences are subtractions between columns of the same table: 1,690 less 1,490, 2,105 less 1,690, and 3,490 less 2,760. The percentile claim takes one more step and is where the reasoning is easiest to skip. Sort every request in a hundred by how long it took. Forty of them took the single-call path and sixty the two-call path, and the two-call path is slower throughout, so in an idealised sort the forty fast ones occupy the first forty places. The request at the ninety-fifth place is then the fifty-fifth of the sixty slow ones, which is about where the ninety-second percentile of that population falls. The consequence is the part worth carrying: the number the team is held to is drawn entirely from the slow population, so any improvement confined to the fast one is invisible to it. Making 40% of requests two hundred milliseconds quicker changes the p95 by nothing at all.

A further 18% of the two-call questions make a second tool call, because the first search returned a related endpoint and the model went looking again. Those requests pay another retrieval hop and another prefill, landing at roughly 4.2 seconds. They are a small share of traffic and they are entirely inside the tail that the budget is about.

What the tool actually costs to have#

The tool definition itself is context, on every request, in both the first and the second call. This one is small, and reading it is a reminder that a just-in-time design does not reduce the prompt to nothing.

SEARCH_ENDPOINTS = {
    "name": "search_endpoints",
    "description": (
        "Find endpoints in the platform API reference. Returns the full "
        "specification for up to 3 matching endpoints: path, method, "
        "parameters, response schema and one example. Call this before "
        "answering any question that names or implies a specific endpoint."
    ),
    "input_schema": {
        "type": "object",
        "properties": {
            "query": {"type": "string", "description": "natural-language description"},
            "service": {
                "type": "string",
                "enum": ["deployments", "builds", "secrets", "audit", "billing"],
            },
        },
        "required": ["query"],
    },
}   # 148 tokens, present in every call

At 148 tokens this is not a budget item, and the fixed tax that tool definitions levy was the subject of the budget work rather than this one. It matters here for a different reason: the quality of that description field is what decides whether the model calls the tool on the questions that need it. An early version of B said only "search the API docs", and on the eval set the model skipped the call on 11% of endpoint questions and answered from its own priors — confidently, and often about a different product's API. Rewriting the description to state explicitly when to call it took the skip rate to 3%. The prompt engineering that a pre-loaded blob does not need is part of what B costs.

Why a vague description produces skips rather than bad searches is worth following, because it is not obvious that it should. The model's decision at that point is whether to call the tool at all, and the only evidence it has about the tool is the description. "Search the API docs" states a capability and leaves the trigger unstated, so the model has to supply the trigger itself, and the question it ends up answering is whether it already knows the answer. For an API endpoint it has never seen, a model's own sense of whether it knows something is exactly the judgement least worth trusting — the shape of endpoint documentation is extremely familiar even when one particular endpoint is not. Rewriting the description to say call this before answering any question that names or implies a specific endpoint replaces that judgement with a pattern match on the question, which is a far easier thing to get right and one the model can perform without knowing anything about the platform. The failure was never retrieval quality. It was that the model was never asked the question the tool needed it to answer.

Where tool definitions themselves grow large — dozens of tools, thousands of tokens of schema before the model has read the question — the same just-in-time logic applies one level up, and Anthropic's Code execution with MCP: Building more efficient agents is a good account of loading definitions on demand rather than dumping all of them up front.

Three designs, not two#

The argument is usually framed as pre-load against just-in-time, which collapses a distinction that decides the latency table. Retrieval performed by the service before it builds the prompt is not the same thing as retrieval the model decides to perform.

The middle column is the one that gets skipped. If a ticket's product area is already a field on the request, or the question's endpoint can be guessed from the file the plugin is open on, the service can query the index itself and paste the result into a single call — one round trip, one prefill, no dependence on the model choosing correctly. Version B's 700-token result could have been fetched that way for 55 ms and no second call, and the whole latency deficit would disappear. It works only when the request carries enough signal to build the query before the model has read anything, which for an IDE plugin sending the current file and cursor position is often true and for a bare chat box is usually not.

When pre-loading is simply right#

A block that is small, stable and needed by nearly every request should be pre-loaded, and the reasoning is not a preference. As a fixed prefix it caches, so after the first call it is close to free in money and cheap in prefill time; fetching it instead would add a round trip to every request in order to save tokens that were barely being paid for. The 4%-saving proposal from the cache chapter is the same shape of error seen from the token side rather than the money side.

This agent has such a block. The authentication and error-envelope section of the reference is 1,100 tokens, and 92% of the eval questions either ask about it directly or need it to make sense of an answer. Retrieving it on demand would mean a search hop on nine requests out of ten to fetch the same 1,100 tokens, plus a 3% chance the model does not think to search for it and answers with a wrong error shape. Pasting it costs 1,100 cached tokens and roughly 12 ms of read.

Setting the two costs side by side shows why the answer is not close, and the general form of the comparison is the useful part. Fetching on demand costs the round trip and the second call on every request that needs the block, and it still costs the prefill of the block once it arrives — the tokens are not avoided, only deferred. Pasting costs the prefill of the block on every request and nothing else. So the fetch design wins only when the share of requests needing the material is small enough that the requests skipping it outnumber the extra cost imposed on the requests taking it. At 92% there is almost nothing to skip: the hop is paid on nine questions in ten, to deliver material that would have cost 12 ms to have already. The 3% skip rate is then pure downside, because it is a way for the design to be wrong that the pasted version has no mechanism to produce. A block needed by nearly everything is not a candidate for just-in-time retrieval at any level of engineering quality.

The measurement that would change the answer#

Shipping the hybrid is the right call at a 3-second budget with this traffic mix, and it is right for a reason that is a property of the mix rather than of the design. The number to watch is the share of requests that take the tool path. At 60% it sets the p95; below roughly 20% the 95th percentile of overall traffic moves into the single-call population and the tool arm's tail becomes 2.5 seconds, at which point it wins on latency, quality on near-twin endpoints, and tokens at once. The plugin team is planning to route status and auth questions to a cheaper path, which would take the tool-path share to 78% and make the tool arm worse still.

The one measurement that flips the decision without any of that is the quality gap on near-twin endpoints. A latency budget is a constraint you can negotiate with a product owner; 0.62 against 0.86 on the questions engineers actually get stuck on is a difference in whether the agent is useful. If the near-twin population grows — and it does, because every new list endpoint added to the platform is another sibling — the argument stops being about three seconds.

The trade, then, is not tokens against tokens. It is a fixed prefill on every request against a round trip and a second prefill on the requests that need material, and which side wins is decided by two shares you can measure: how often the material is needed, and how much of the traffic mix sits inside the percentile you are held to. Both are properties of the workload rather than of the architecture, which is why the same two designs come out differently on two services with the same corpus.

Both arms have now been compared on latency, tokens and mean quality, and the hybrid won on all three. That comparison is still missing a column. Version A cannot fail to have the reference in context; version B can search and come back with the wrong endpoint or with nothing, and the model will answer anyway. Nothing in the eval numbers above distinguishes an answer grounded in the right specification from one grounded in a plausible-looking neighbour, which is the next thing to measure.