Shortening a prompt

Cutting a prompt is the one prompt-engineering task with an obvious external sponsor. Input tokens are billed on every call, they are visible in a finance dashboard, and a prompt that has accumulated instructions for two years always contains text that is genuinely doing nothing. The request to halve it is reasonable and the assumption underneath it usually is not: that a shorter prompt is a free win, verifiable by reading.

It is not verifiable by reading, because the value of an instruction is proportional to how often its condition fires and the cost of reading is proportional to how much text it occupies. Those two quantities are unrelated, and the instructions worth the most per token are the ones you will scroll past.

Both halves of that are worth walking, because the conclusion only follows if each step does. An instruction earns its keep by changing the answer on the inputs where its condition holds, so what it is worth is the share of traffic meeting that condition multiplied by how much the answer changes there. The effort of noticing it is a different quantity built out of different things: a reviewer forms a view of a prompt by scanning the text and by looking at a sample of outputs, and how much a clause contributes to either depends on how many lines it occupies and how often the sample happens to contain a case it governs. Nothing connects the two quantities. A condition can hold on one input in nine and be written in a single line; a paragraph can sit above every input and change no answer at all. Ranking clauses by how much room they take up therefore ranks them by a quantity with no relationship to what they do.

The belief this produces is worth stating in the form somebody would defend it in, because nobody arrives at it carelessly. A prompt is a document, deleting text from a document is exactly the kind of change a competent reader can check by reading, and every other artefact an engineer prunes behaves that way. Dead code, an unused import and a config key nothing reads all announce their own uselessness under inspection. The comparison is a real one and it breaks at a nameable place: each of those artefacts has consumers you can search for, and a prompt clause has no consumer any search will find. Its consumer is a distribution over inputs, most of which are not on your screen. The reading is honest and the conclusion is unavailable, because text you cannot see doing anything is not the same as text that does nothing, and the only instrument that separates them is an input that triggers the condition.

Redundant-sounding is not the same as redundant#

The property is worth naming directly: instructions covering rare conditions look identical to instructions covering nothing. A rule that fires on 11% of inputs is invisible on the 89% you read while forming an opinion, and its cost may be 25 tokens, which is under 2% of a 1,400-token prompt. Both of those figures repay a second look, because between them they make the trap symmetrical. Twenty-five tokens inside fourteen hundred is about one part in fifty-six, so deleting the clause buys back under a fiftieth of the input bill — a saving that is real and that will not be visible on an invoice. A condition holding on eleven inputs in a hundred, meanwhile, turns up about three times in the thirty outputs a reviewer reads before forming a view, and three cases scattered through thirty is not enough for anyone to work out which sentence produced them. Cheap to keep and almost impossible to observe are the two properties that put a rare-condition clause at the top of a cut list it should never appear on.

A paragraph restating that the assistant should be helpful and honest is the opposite case and is correctly deleted. Post-training already optimises heavily for that behaviour, so restating it in the prompt is buying a property you already have, and it survives in prompts largely because it was necessary in 2022 and got copied forward. Verbose framing has the same profile: "Please carefully read the following text and think about what it means before you begin" costs tokens and changes nothing measurable on a well-specified task. Those two — restatements of post-trained behaviour, and framing that only sounds like instruction — are the candidates a reader can rule on without running anything. The table below is mostly a list of the ones they cannot.

The categories sort cleanly enough to work from directly.

The last row is the one that produces the expensive surprises. "Nobody remembers why this is here" is a statement about your team's documentation, and treating it as evidence of uselessness is how prompts lose their edge-case handling one cleanup at a time. When you find such a clause, the cheap resolution is to construct three inputs that trigger the condition it names and run them with and without it. Fifteen minutes, and the answer is unambiguous.

The systematic version of that is leave-one-out ablation, meaning you remove one block of the prompt at a time, re-score the whole set, and record what each block was worth, which is worth running once on any prompt you intend to cut hard:

BLOCKS = split_into_blocks(PROMPT)          # 12 blocks, by paragraph

baseline = score(PROMPT, CASES, MODEL, seed=11)
for i, block in enumerate(BLOCKS):
    reduced = "\n\n".join(b for j, b in enumerate(BLOCKS) if j != i)
    s = score(reduced, CASES, MODEL, seed=11)
    print(f"{i:2d}  {len(tokenize(block)):4d} tok  {s - baseline:+5.1f}  {block[:48]!r}")

Twelve blocks over 400 cases is 4,800 completions, roughly $20 and half an hour of wall clock at eight-way concurrency, and it produces a ranked list of what each paragraph is worth. Blocks with a delta inside the noise band are your cut list; blocks with a large negative delta stay; blocks with a positive delta are interesting for a different reason, because a paragraph whose removal improves the score is either irrelevant context or one half of the kind of conflict the previous chapter but one described.

Ablation has one blind spot worth knowing about before you trust its output. It measures each block alone against the full prompt, so a rule stated twice in different words shows a delta of zero for each copy — remove either and the other still covers the behaviour — and a naive reading of the table says both are free to delete. Removing both costs you the rule. The same trap applies to two examples that jointly cover an edge case neither covers alone. The defence is to apply the cut list as one change and measure the result, rather than treating twelve independent measurements as a licence for one twelve-part edit, which is the same discipline chapter 5 arrived at from the other direction.

The reason for that blind spot sits in the question the loop asks rather than in any error in how it asks it. Each iteration scores a prompt with one block removed and every other block still present, so the number it reports is what the prompt loses when this block goes and everything else stays. Measured that way, each copy of a duplicated rule truthfully reports nothing, because the question put to it was already answered by the copy that remained. Twelve honest answers to twelve separate questions are not an answer to a thirteenth that nobody asked, and the thirteenth — what the prompt loses when several blocks go at once — is the only one a cut list actually poses. Ablation ranks candidates for deletion; only the combined run scores the deletion.

Where the ablation table is most useful is in settling arguments about examples, because example blocks are large enough that the deltas clear the noise. Four examples at 200 tokens each are 800 of a 1,400-token prompt, so they are always the first place a serious cut looks, and the question of which two to keep is otherwise decided by taste. If two of them both demonstrate a straightforward record with all fields present, one is paying 200 tokens to repeat a lesson, and the table will show it.

Cutting is one cost lever and usually not the first#

Take a production case: a 3,100-token prompt, 400,000 calls a day, p95 of 2.4 seconds — the figure 95% of calls come in under — an eval score of 88, and a mandate to reduce inference cost this quarter. The arithmetic is worth doing before choosing an approach, because it decides which lever is worth the risk.

Input volume is 3,100 × 400,000 = 1.24 billion prompt tokens a day. Multiplying is the right operation because every one of those calls carries the same prompt: the prompt is not shared between requests or sampled down, so the daily total is the per-call figure repeated once per call. Thirty-one hundred times four is twelve thousand four hundred, and the five remaining zeros in the call count carry that to a little over a billion and a fifth of tokens a day, which is where 1.24 billion comes from.

Turning tokens into money takes one more step, and that step is a division because the rate is quoted per million rather than per token. At $3 per million input tokens, the unit being priced is a million tokens, so divide the daily token count by a million and you have one thousand two hundred and forty of those units; multiply the units by the rate and the day costs $3,720. A year is that daily figure repeated three hundred and sixty-five times, which comes to about $1.36M. Halving the prompt halves the token count, and since every step after the token count is a multiplication by a fixed rate, halving the tokens halves the bill: roughly $1,860 a day, at the price of putting the eval score at risk on every rare slice in the system.

The asymmetry underneath that arithmetic is worth naming before any lever gets chosen, because it explains who turns up asking for a cut. What a prompt costs is a multiplication over three quantities somebody already reports — a token count, a call volume and a published rate — so anybody with a spreadsheet can produce the figure and nobody has to be persuaded it is real. What a prompt is worth is a score on a set of cases that has to be assembled, labelled and run, and it exists only where somebody has already built one. The pressure to cut therefore arrives with a number attached and the defence does not, which is a fact about the two measurements rather than a fact about the merits of cutting.

Prompt caching is the first move because it is the only row with no quality risk at all. If the first 2,700 tokens of the prompt are byte-identical across calls — which they are, if the placement discipline from topic 2.6, on where an instruction belongs, was followed — a provider cache serves them at roughly a tenth of the input price, and the saving exceeds what halving would deliver while leaving every constraint in place. This is also the reason the economics of prompt length changed in the last two years: the marginal cost of a long static prefix at high call volume is now closer to $0.30 per million than $3.00 per million, and a great deal of prompt-shortening work that was clearly worth doing in 2023 is now an expensive way to buy a small number.

Latency deserves the same scepticism, and the arithmetic that settles it is a division. Prefill, the pass over the input that a model makes before it produces a single output token, is roughly linear in how many tokens it has to read, so the time it takes is a length divided by a rate. At a typical serving throughput of five to twenty thousand tokens per second, 3,100 tokens of prompt costs 0.15 to 0.6 seconds: thirty-one hundred tokens against twenty thousand a second is a little over a seventh of a second at the fast end, and the same thirty-one hundred against five thousand a second is a little over three fifths of a second at the slow end. The rest of a 2.4-second p95 is output tokens generated one at a time, and that loop does not get shorter when the prompt does.

Halving the prompt therefore removes at most a few hundred milliseconds, which is what the table records as −0.1 to −0.3 s. A little over fifteen hundred fewer tokens to read is under a tenth of a second at the fast end and about three tenths at the slow one, and set against a p95 of two and four tenths of a second that is a change in the second decimal place of somebody's experience. If latency is the actual complaint then capping or restructuring the output is where the seconds are. Selling a prompt cut as a latency fix and delivering 0.2 seconds is a good way to be asked for a second cut.

An aggregate that barely moves can still be a regression#

Suppose a later shortened prompt comes back at 88.4 against a baseline of 88. The tempting reading is that the cut was free and mildly positive, and there are two independent reasons not to publish that.

The first is that 88 is a single number with no known spread. If two runs of the unchanged prompt land at 87.6 and 88.5, then 88.4 sits inside the range the prompt produces when nothing changes, and the honest claim is that no regression was measured. That is a perfectly good claim to ship on — a 46% reduction in input tokens with no measured regression is a real result — and it is a different claim from "quality improved", which invites someone to cut again next quarter on the same reasoning.

The second reason is the one that bites. An aggregate over 400 cases is a weighted average, and the slices most likely to have been damaged are the small ones. Suppose a missing-date slice is 4% of the set rather than 11%: a collapse from 81 to 12 on that slice moves the aggregate by 2.8 points, which a +3 point improvement elsewhere would hide completely. The slices at risk are precisely the rare conditions the deleted constraints existed for, so the correlation between "small enough to disappear in the average" and "likely to have broken" is close to one.

Run the same weighted-average step on those two figures and the concealment turns out to be arithmetic rather than rhetoric. Four parts in a hundred of the same sixty-nine-point collapse is a shade under two point eight points, so identical damage arrives at the aggregate reduced by nearly a factor of three against the eleven-per-cent version, purely because fewer cases carry it. Set a three-point gain across the rest of the set beside that and the two very nearly cancel: the aggregate moves by something like two tenths of a point, which is smaller than the gap between the two runs of the unchanged prompt quoted a few lines above, and therefore reports as nothing having happened. The slice did not become less important when it became smaller in the evaluation set. It became quieter, and the quantity that went quiet is its share, which is a property of how the set was sampled rather than a property of the system.

The remedy is to score by slice and to define the slices before you cut, then report a table rather than a number:

slice                 n     before   after    delta
all                 400     88.0     88.4     +0.4
missing_date         17     81.2     11.8    -69.4
multi_entity         44     84.1     83.0     -1.1
non_english          23     79.0     80.4     +1.4
long_record          31     86.2     85.8     -0.4
standard            285     89.5     92.6     +3.1

That table answers the ship question by itself, and it also explains the mechanism: the model got better at the common case once the noise was gone and lost an edge case entirely, which is a real and repeatable effect of cutting. Restore the clause, keep the rest of the savings, re-run, and ship on the second table.

The improvement on standard deserves an account of its own, since a cut that raises a score sounds as though it ought to be impossible. A prompt is one string, and the model is scoring what comes next given the whole of it. Nothing in the text is marked as applying only to records of a particular kind, and there is no step at which a clause is set aside on the grounds that its condition does not hold today. A sentence about how to handle a missing date is therefore still material sitting in front of a record whose date is present, still available to be read as relevant to it, and deleting it removes something the model could have taken as a cue. The scores over the next fragment are conditioned on all of the text, and the model continues accordingly. How much weight that carries is worth being careful about: the standard row moves by three points and the missing_date row moves by nearly seventy, so the effect is real, repeatable and by a wide margin the smaller of the two the table records. What the table does that no aggregate can is hold the two apart, which is why one commit reads as a mild improvement on one instrument and as a broken edge case on the other.

The rule that comes out of this is not "do not cut". Prompts do accumulate text that is worth nothing, the savings at high call volume are real, and an ablation run will find the dead weight in half an hour. The rule is that the aggregate is the wrong instrument for the decision, because cutting damages exactly the population the aggregate is least sensitive to. A shortened prompt is shippable on a slice table and is never shippable on a single number, however large the improvement in that number looks.

None of this apparatus helps if the change does not exist as a diff. Ablation compares two versions, bisection reverts to a scored one, and the slice table has a "before" column because there was a before to run. A prompt that lives in a text box in a provider console, edited in place by whoever is on support that week, supports none of those operations — and the failure that produces is not a lower score so much as an inability to say anything about the score at all.