An inline suggestion must be dismissible; batch enrichment trades freshness for cost

A support platform shipped inline reply suggestions into its agent console and turned them off eleven days later. The mechanism worked: an agent opened a ticket, started typing a reply, and a greyed-out continuation appeared ahead of the cursor which Tab would accept. In a staged demo it looked like the future. In production, agents complained that the console had become "twitchy", three of them reported losing text they had typed, and the acceptance rate settled at 4 percent.

The post-mortem found two numbers. Median time from the last keystroke to the suggestion rendering was 910 milliseconds, and the suggestion box was rendered as a positioned element that pushed the composer's content down by one line when it appeared. A competent typist working at 55 words per minute produces roughly 275 characters a minute, which is one character about every 218 milliseconds. A suggestion computed from the text as it stood 910 milliseconds ago is proposing a continuation for a sentence that has moved on by four characters, and it arrived by shoving the layout. Nobody lost text; the composer reflowed under their cursor while they were mid-word, which feels identical.

A suggestion that arrives four characters late is proposing a continuation to a sentence that no longer exists#

The timing constraint on an inline suggestion is not "fast" in the general sense that engineers mean it. It is a specific budget set by the user's own typing speed, because a suggestion is only useful while the text it was computed from is still the text on screen. The working target is under 300 milliseconds from the last keystroke to a rendered suggestion, which has to cover a debounce — the short wait after the last keystroke that stops every character firing its own request — plus a network round trip and enough model output to be worth showing. That budget rules out a large model producing a long completion, and it rules out any design where the suggestion is computed from a full conversation history assembled per keystroke.

Three properties are non-negotiable, and each one has a concrete failure attached. The suggestion must be dismissible — a single keystroke that means no, with no dialogue and no state change. It must be non-blocking, which means it cannot occupy layout the user was already using, cannot steal focus, and cannot make the next keystroke wait on it. And it must be cancellable in flight, because if the user has typed since the request went out, the response is already wrong and rendering it is worse than rendering nothing.

let inFlight: AbortController | null = null;
let debounce: ReturnType<typeof setTimeout> | null = null;

function onKeystroke(editor: Editor) {
  inFlight?.abort();                        // the prefix changed; the answer is stale
  if (debounce) clearTimeout(debounce);

  debounce = setTimeout(async () => {
    const prefix = editor.textBefore(editor.cursor);
    const ctrl = new AbortController();
    inFlight = ctrl;
    const requestedAt = performance.now();

    try {
      const res = await suggest(prefix, { signal: ctrl.signal });
      if (ctrl.signal.aborted) return;      // user typed while we waited
      if (editor.textBefore(editor.cursor) !== prefix) return;  // belt and braces

      editor.showGhost(res.text);           // overlay, zero layout impact
      telemetry.emit("suggestion_shown", {
        id: res.id,
        latency_ms: performance.now() - requestedAt,
        prefix_len: prefix.length,
      });
    } catch (e) {
      if (!ctrl.signal.aborted) telemetry.emit("suggestion_failed", { reason: String(e) });
    }
  }, 120);                                  // debounce below one keystroke interval
}

The 120-millisecond debounce is chosen to sit under a single keystroke interval, so a typist in flow never triggers a request mid-word, and a typist who pauses gets one immediately. showGhost renders into an overlay layer rather than into document flow, which is the entire fix for the support platform's "twitchy" complaint. The two staleness checks look redundant and are not: the abort covers the request that is still open, and the prefix comparison covers the response that resolved in the same tick as a keystroke.

Acceptance rate is the rent the suggestion pays for its place on screen#

An inline suggestion occupies attention on every keystroke whether or not it is useful, so the surface owes a running measurement of whether that attention is being repaid. Acceptance rate — suggestions accepted divided by suggestions shown — is the primary one, and it is unusually well suited to the job because it is emitted by the user's ordinary behaviour rather than by a feedback widget nobody clicks.

Published figures for code completion have hovered around 30 percent accepted, which is a reasonable stake in the ground for a surface where the next token really is largely determined by the previous ones. Prose replies to support tickets are a harder case and a good result is lower. The number to react to is the floor rather than the target: below roughly 10 percent, the suggestion is being ignored nine times out of ten and is charging every agent a glance for the privilege. The support platform's 4 percent was the surface telling them what the latency trace confirmed.

The lever that moves the number is usually not the model. It is where the suggestion is allowed to fire at all. The support platform's relaunch triggered only after a sentence boundary, only when the ticket had a matched macro category, and never in the first three characters of a reply — three conditions that cut the number of suggestions shown by 62 percent and took acceptance from 4 percent to 23 percent. Narrowing the trigger looks like doing less, and it is the cheapest available improvement to a surface whose denominator is the problem. A suggestion that never appears when it would be wrong costs nothing and annoys nobody.

Acceptance alone will mislead you in one specific way, so pair it with a second metric. An agent under queue pressure will accept a mediocre suggestion and then rewrite it, which counts as acceptance and delivers nothing. Retention — the fraction of accepted suggestions still substantially present in the submitted text — separates the two. Emit suggestion_shown, suggestion_accepted and a suggestion_retained computed at submit time by diffing the accepted span against what was sent, and the pair tells you whether to tune, to narrow the trigger, or to remove the surface.

Batch enrichment buys unit cost by giving up freshness#

The other under-built surface has a trade at its centre that is easy to state and easy to get wrong in practice. A marketplace with 4.2 million live listings wanted each one classified into a 900-node product taxonomy — the tree of categories a listing has to be filed into, from "Home" down to "Home > Kitchen > Cookware > Stock pots" — with three attributes extracted from the description. At roughly 1,200 input and 250 output tokens per listing on a small model priced at $1 per million input and $5 per million output tokens, one listing costs about $0.0025 synchronously. A batch API takes a file of many requests at once and returns the results some hours later instead of answering each call as it arrives; the ones offered by the major providers, including Anthropic's Message Batches API, run at half the standard rate with results returned within 24 hours, which takes the same listing to about $0.0012.

Halving the unit price is the visible half of the trade. The invisible half is that a result computed on a schedule is stale between runs, and the size of that staleness is what actually decides the cadence.

The per-unit price stops falling after the first row, because the discount comes from the asynchronous queue rather than from the size of the job. What changes down the column is total spend, and that is a function of how many records you re-derive rather than of when you run. A nightly full sweep of 4.2 million listings costs about $5,200 a night and mostly recomputes values that did not change; a nightly sweep restricted to the 60,000 listings created or edited that day costs about $74. The marketplace ran the delta nightly and a full sweep monthly, and the full sweep existed for one reason: it is the mechanism by which a prompt change reaches records that were classified under the old one.

Batch is only viable where a wrong value can be corrected on the next run at no cost to anyone. A misclassified listing sits in the wrong category until tomorrow and then moves, which is annoying and recoverable. The same pipeline writing a tax category onto a filed invoice is not recoverable, because the invoice has left the building. That is the correctability discriminator doing real work rather than sitting in a table.

A batch surface without a re-run story is half a surface#

The failure that shows up three months in is not a bad classification. It is a prompt change: someone improves the taxonomy prompt, ships it, and now the catalogue holds 4.2 million values written by two different versions of the system with nothing on the record saying which. Every batch surface therefore needs a version stamp per record and a state machine that says what a re-run is allowed to touch. Sweeping back over records that were already written, so that values produced under the old prompt are re-derived under the new one, is a backfill, and it is the operation the states below exist to constrain.

The absorbing state — human_corrected, the one state a record can enter and never leave — is the part teams discover the hard way. Once a person has corrected a value, a later sweep that overwrites it destroys the most reliable data in the table and does it silently, at scale, overnight. Writing enriched_by_version and corrected_by_user_at onto every record, and filtering the sweep on both, is the whole mechanism — cheap to add on day one and expensive to add after a backfill has quietly reverted nine thousand manual corrections.

What neither surface will do for you#

An inline suggestion is not a small conversation. It has one turn, no memory of being refused, and no way for the user to say "not like that, more like this" — the only feedback channel is Tab or not-Tab. Teams who want the user to steer the suggestion end up adding a comment box, or a slash command, or a panel, and at that point they have built a conversation in a cramped space and should compare it honestly against building one properly.

Batch enrichment is not a cheaper background job either, and treating it as one is the common mistake. They differ in their correctness contract rather than in their schedule: a background job runs per record and can be retried per record when it fails, while a batch run either lands or does not, and a partial failure leaves a fraction of the table in an inconsistent state that you have to be able to name. Ask what happens when 3 percent of a sweep errors before you choose the cadence.

Both surfaces also share a limit that no amount of engineering removes: they are answers to requests whose shape you already know. The inline suggestion knows the user is continuing a sentence. The batch job knows every record needs the same three attributes. Neither has anywhere to put a request the designer did not anticipate, and that gap is the strongest argument anyone has for conversation — which the next chapter takes seriously enough to draw the line properly.