Predicted outputs and prefetches bill for every miss

The editor feature takes a 400-line file and a request like "make parseInvoice return a Result instead of throwing", and returns the whole file rewritten. Four hundred lines is about 4,800 tokens, the deployment sustains roughly 220 tokens per second, and the median request took 21.5 seconds, nearly all of it spent regenerating lines the model had no intention of changing.

Predicted outputs is the API feature aimed exactly at this. You send the current file alongside the prompt as the output you expect, and the decoder verifies the predicted tokens in parallel rather than generating them one at a time, accepting the run wherever the model agrees with the prediction.

const res = await client.chat.completions.create({
  model: "gpt-4.1",
  messages: [
    { role: "system", content: "Rewrite the file. Output the whole file, nothing else." },
    { role: "user", content: instruction },
    { role: "user", content: currentFile },
  ],
  prediction: { type: "content", content: currentFile },
});

console.log(res.usage.completion_tokens_details);
// { accepted_prediction_tokens: 4612, rejected_prediction_tokens: 218, ... }

On the fixed workload of 500 recorded edit requests, median latency fell from 21.5 seconds to 8.4, a 61% improvement. Spend rose 17%. And 10% of requests came back slower than they had been before the change, which is the part that does not fit the story the median tells.

Rejected tokens are output tokens#

The mechanism is the speculative decoding described in the previous chapter with the draft supplied by you instead of by a small model inside the deployment. That is why the accounting is visible in your bill rather than hidden in someone's throughput numbers: when the deployment supplies its own draft, rejected tokens are the provider's waste, and when you supply the draft, they are yours.

The two fields at the bottom of that response are the whole economics of the feature. Accepted prediction tokens are the ones the model agreed with, delivered fast. Rejected prediction tokens are the ones it did not, and they are billed at the output rate exactly as if the model had generated them, on top of the tokens it actually produced. A prediction is a bet placed at output prices, and the loser pays.

That produces two entirely different requests wearing the same code path. A localised edit — changing a return type, renaming a symbol, adding a guard clause — diverges from the prediction for a few dozen tokens and rejoins, so nearly the whole file is accepted:

The restructure column is the one that answers the puzzle. When the instruction is "split this into three modules", the output departs from the prediction near the top of the file and never comes back, so almost the entire 4,800-token prediction is rejected and billed, and the verification work buys nothing — the request is slightly slower than the unpredicted baseline because the decoder spent effort checking a prediction it kept throwing away. That request pays double and waits longer.

Weighting those columns by the workload mix gives 5,759 billed output tokens per request against a baseline of 4,850, which is the 17% rise in spend, and a median dominated by the 70% column, which is the 61% latency win. Both headline numbers are real. They describe different subsets of the traffic, and reporting only the median hides that a tenth of users got a worse product at twice the price.

The remedy is to stop treating prediction as a global setting. A cheap classifier on the instruction, or simply the ratio of the diff the model is being asked to produce, is enough to decide whether to send the prediction at all:

const RESTRUCTURE = /\b(split|extract|reorganis|reorganiz|rewrite (the )?(whole|entire)|convert .* to)\b/i;

const opts = RESTRUCTURE.test(instruction) || currentFile.length > 60_000
  ? {}                                        // no prediction: it will be rejected
  : { prediction: { type: "content", content: currentFile } };

That gate is crude and it recovers most of the loss, because the failure is not evenly spread across requests but concentrated in a category you can name in advance.

A prefetch is a bet with a hit rate#

The same shape appears one layer up in the product. The document viewer has a Summarise button, and summaries take 4.2 seconds at p95, which product wanted removed. The change was to start generating the summary as soon as a document is opened, so that by the time anyone clicks, the answer is already cached. On the fixed workload the button's p95 time-to-answer fell to 150 milliseconds, which is a cache read and a render.

Finance flagged that spend on the summariser was up 5.3x. The arithmetic is not subtle once the click-through is on the table: the product sees about 12,000 document opens per day and 2,280 Summarise clicks, so 19% of opens end in a click. That fraction is the prefetch's hit rate, meaning the share of speculatively generated work that somebody eventually asks for; the rest is generated, paid for, and thrown away. A summary costs $0.011 to generate.

Prefetch on open buys the full latency win and throws away 81% of what it spends, which is $39,000 a year for this one button. That may still be the right trade — $107 a day is cheap against the engineering time to do better, and the calculation changes entirely if summaries cost $0.30 instead of $0.011 — but it is a trade, and a comparison that reports only "p95 fell to 150ms" is not a comparison. Any prefetch has to be priced at its hit rate, because the hit rate is the exchange rate between the money and the milliseconds.

The third column is what happens when the trigger moves closer to the decision. Firing on hover over the button, or on a dwell of more than four seconds in the document body, generates for 3,800 opens instead of 12,000 and still catches 1,950 of the 2,280 clicks. The 330 clicks it misses fall back to on-demand generation and wait the original 4.2 seconds. That is 86% of the latency win at 32% of the prefetch bill, and it is available to anyone willing to instrument one more event.

There is also a correctness dimension that on-demand generation does not have. A prefetched summary is generated against the document as it stood at open time, and the viewer allows inline editing, so a user who opens a contract, edits three clauses, and then clicks Summarise gets an answer about a document that no longer exists. The cache key has to include a content hash rather than a document id, which means edits invalidate the prefetch and the click falls back to a 4.2-second generation — precisely on the sessions where the user has been most engaged. On this product 6% of clicks followed an edit, so the effective hit rate is 18% rather than 19%, and the invalidation logic is the part most likely to be got wrong, because a stale summary looks entirely plausible.

Prefetching also spends something other than money, which is capacity. Twelve thousand summariser calls a day are twelve thousand calls competing for the same rate limit as everything else in the account, and the fan-out arithmetic from earlier applies: raising baseline concurrency to serve speculative work degrades the latency of the non-speculative work sharing the deployment. Prefetch bills at the margin and can bite in the tail.

Four levers, one workload#

Putting the levers side by side on the same 500 recorded requests, so the numbers are comparable rather than four vendors' best cases:

Skeleton-then-fill is the one that has not appeared yet, and its cost is a shape it commits to before the content exists. The panel renders its headings — Summary, Parties, Risks — within 150 milliseconds and fills them as the model produces each section, which gives a genuine time-to-useful-content of 1.9 seconds for the first section rather than the 6.8 seconds it takes for all three to finish.

The commitment is what costs. The skeleton is drawn before any content exists, so it is drawn from an assumption about the answer's shape, and when the assumption is wrong the interface has to correct itself in front of the reader. A document with no risks leaves a heading over an empty region, which reads as a system that failed rather than as a document with no risks, unless the fill step is allowed to remove sections it did not use — and removing a section the reader has already started scrolling towards is its own kind of unpleasant. A document with nine parties in a skeleton that allotted three rows reflows everything below it. The version that avoids both requires knowing the section list before drawing, which is an extra structuring call of about 0.6 seconds against the completion time and roughly $0.0006, and that call is why the completion row is positive rather than zero.

Skeleton-then-fill therefore works well when the output schema is fixed by the product — a contract review panel that always has the same six fields — and badly when the structure is discovered from the content. The failure is not expensive in money and it is unusually visible, since layout shift — content already drawn on screen jumping to a new position as more of it arrives — is one of the few latency-adjacent regressions users describe unprompted.

Only two of those columns interact safely: streaming and skeleton-then-fill compose, since one fills the other. Predicted outputs and prefetch both stake money on a guess, and stacking them multiplies the waste rather than adding it — a prefetched rewrite that is never requested pays the full rejected-prediction bill for nothing.

What none of this is#

Neither prediction nor prefetch is caching, although both get described that way. A cache is populated by demand and its hit rate is a property of repetition in the traffic; prompt caching, specifically, reduces the input side of the bill on repeated prefixes and has no downside beyond a small write cost. Predicted outputs touches only the output side and adds to the bill when wrong. Prefetch is populated by speculation, so its hit rate is a property of your guess about user intent rather than of the traffic.

None of these levers reduces the work the model does on a hard request. The restructure column still takes 24 seconds because the model is genuinely writing 5,200 new tokens, and no amount of prediction changes that. Everything here shifts when the work happens or whether it can be skipped, which is why the honest way to present any of it is with the milestone it moved and the money it spent on the same line — and why a quarter of latency work that reports only one of those two numbers can look like a success while being a cost increase.