Pricing the example block

An example block is the only part of a prompt that is paid for on every request and never varies between them. The instructions are paid for too, but they are usually a few hundred tokens; the user's input is what the request is for. A block of five or six worked demonstrations is commonly the largest thing in the payload, and because it sits in a text file that nobody looks at after it is written, its cost is the one nobody has a figure for.

What makes the block worth pricing on its own, separately from the rest of the prompt, is a property it has that the other parts do not. Its token count is settled once, on the day somebody saves the file, and it does not vary with the request. The user's document is different on every call, and paying for it is paying for the work being asked for. The block is the same tokens re-sent, so its contribution to the invoice is a single multiplication — tokens in the block, times calls in the month, times the rate — and every term in that multiplication is a number the team already has. What is missing is not data. It is that nobody has performed the multiplication.

Three separate costs come out of the same tokens. The first is the invoice, which is arithmetic on the token count and the call volume. The second is prefill — the phase in which the model reads the prompt before it can emit anything — which is roughly linear in input length and so shows up as time to first token and in the tail of the latency distribution. The third has no invoice at all and is the room the block occupies in the context window, which is invisible until the day something else needs that room.

Decomposing the bill#

Pricing a block starts by splitting the request into its parts and multiplying each by the volume. Take a document extraction service: six few-shot examples, each a real source document paired with the JSON it should produce, averaging 600 tokens apiece. Six of those at six hundred tokens each is 3,600 tokens, and multiplying is the allowed step because every example is sent on every call, whether or not the input in front of it resembles that example at all. Add roughly 200 tokens of instructions and a user document averaging 900 tokens, and a typical request carries 4,700 input tokens. Of those, 3,600 are demonstrations, and three thousand six hundred out of four thousand seven hundred is a shade over three quarters — the 77% you would otherwise have had to take on trust. The output is about 150 tokens of JSON. At 40 cents per million input tokens and $2 per million output tokens, and 5 million calls a month, the month decomposes like this:

Every cell in that table is one of three operations, and each is worth doing rather than reading. The monthly-token column is tokens per call multiplied by calls per month: the block's 3,600 tokens against 5 million calls is eighteen billion tokens, the instructions at 200 tokens are one billion, the documents at 900 tokens are four and a half billion, and the 150 tokens of output are three quarters of a billion. Those four add to the 24.25 B in the total row, and the per-call figures add to 4,850 the same way.

The cost column is the second operation, and it carries the step people skip. A rate quoted per million tokens applies to units of a million, so the monthly token count has to be divided by a million before the rate touches it. Eighteen billion tokens is eighteen thousand of those units; at 40 cents a unit the block costs $7,200. One billion is a thousand units, which at the same rate is $400. Four and a half billion is four and a half thousand units, which is $1,800. The output is billed on a different rate, so three quarters of a billion is seven hundred and fifty units at $2 each, which is $1,500. Adding those four costs gives $10,900.

The share column is the third operation, a division of each cost by that total. $7,200 out of $10,900 is almost exactly two thirds, which is the 66% in the table.

Two thirds of the bill is six examples. The ratio is what generalises, rather than these particular figures, and the reason lies in the shape of the multiplication rather than in the numbers it was performed on. Rates change, volumes change, and a document extractor is not a classifier. What survives all three is that raising the volume or the rate multiplies every row of the table by the same number, and multiplying every row by the same number leaves the proportions between them exactly where they were. A block that is three quarters of the input will be roughly three quarters of the input bill at any volume and at any price.

Take a smaller service: five examples of about 500 tokens each, an instruction header of 200, a user input averaging 400 tokens, an output of 200, and 2 million calls a month. Five examples at five hundred tokens apiece is 2,500 tokens, and adding the header and the user's input brings the request to 3,100. The example block is therefore 2,500 tokens of a 3,100-token input, which is 81% of everything you send, and the user's actual input — the only part of the request that differs between calls — is 400 tokens of the same 3,100, or 13%. Anyone reading that prompt in an editor sees a file where the examples are most of the screen and thinks of them as context. The billing system sees a service that spends four fifths of its input budget re-sending the same five tickets 2 million times a month, which is 2,500 tokens multiplied by two million calls and comes to 5 billion tokens, and it will do that every month until someone opens the file with the intention of deleting something.

The same tokens, charged again as time#

Every one of those 3,600 example tokens is also work the model does before it can emit a single character of output. Prefill runs the whole input through the network to build the key-value cache, the per-token intermediate state the model then consults for every token it generates, and it is compute-bound and roughly linear in input length, so a longer prompt is a longer wait before the first token appears.

Why the wait is proportional to the length, rather than merely correlated with it, is worth having as a chain. The model scores the next fragment of text conditioned on everything before it, and producing that score requires the per-token intermediate state for every token already present. None of that state exists when a request arrives; the request is bytes. Building it is one pass over the input, the work per token is roughly the same for the first token of the prompt as for the last, and no output fragment can be scored until the pass has finished. So the delay before the first character appears grows with the number of tokens that had to be passed over, and an example block is tokens like any other. A long prompt is not free even when the answer is one word.

Turning that into a number is a division, and dividing is the allowed operation precisely because the work is proportional to length — time is length over rate. On a shared endpoint delivering something like 8,000 tokens per second of effective prefill throughput for a single request, 4,700 input tokens divided by eight thousand a second is a little under six tenths of a second, or about 590 milliseconds of prefill. The block's share of that wait is its share of the tokens, so dividing 3,600 by the same eight thousand gives 450 milliseconds as the part of the wait the examples are responsible for.

Whether that 450 milliseconds matters depends entirely on the shape of the response, and the way to find out is to price the other half of the request the same way. An extraction service emitting 150 tokens at roughly 60 tokens per second spends 150 divided by sixty, or about 2.5 seconds, in decode. Adding the prefill to the decode gives a little over three seconds, which is the 3.1-second median this service actually measures — the arithmetic and the histogram agreeing is the check worth doing before trusting either one. The examples are therefore adding 450 milliseconds to a 3.1-second total, and four hundred and fifty milliseconds out of three thousand one hundred is about a seventh — 15%, noticeable but survivable.

A classification service returning a single-word label is a different story, and running the same two divisions shows why rather than asserting it. Its output is one or two tokens, so the decode term is one or two divided by sixty — a few tens of milliseconds, small enough to vanish into the noise — and essentially all of the latency is prefill. Once the decode term is gone, the block's share of the response time is nothing but its share of the input, so an example block that is 80% of the input is 80% of the response time. The general rule is that few-shot examples are paid in time in proportion to how short your outputs are, which inverts the intuition people bring from thinking about latency, where long outputs feel expensive.

Prefill also interacts badly with the tail. The p95 of a service — the response time that 95% of requests come in under, so the experience of the unluckiest one request in twenty — is usually set by requests that arrive when the endpoint is busy, and a busy endpoint batches prefill work across concurrent requests, so a queue of requests each carrying 3,600 tokens of identical examples saturates the compute budget five or six times faster than the same queue carrying only the documents. That multiplier is the same ratio a third time: a request carrying only its 900-token document asks the endpoint for roughly a fifth of the prefill work a 4,700-token request asks for, so a fixed amount of compute clears five or six times as many of them a second. The unluckiest request in twenty is overwhelmingly a request that arrived behind a queue, which is why anything that shortens the queue moves the tail much further than it moves the median. When this service was measured, p50 was 3.1 seconds and p95 was 6.8; cutting the block to two examples moved p50 to 2.8 and p95 to 4.9 — three tenths of a second at the median against 1.9 seconds at the tail, from one edit to a text file, for exactly this reason.

The third cost is the room the examples take up, and it is the one that arrives as a surprise, because nothing anywhere reports it. A model with a 128,000-token context has 128,000 tokens for everything, and 3,600 of them are permanently committed before the user's document arrives. Nothing bills for that, and the reason is that a context window is a limit rather than a metered resource: you are not charged for the room you occupy in it, you are refused when the room runs out. So the commitment appears in no column of any invoice and in no latency histogram, and it stays invisible until the day a customer uploads a 60-page contract, or until someone wants to add retrieved context, or until the team wants to raise the number of documents processed per call from one to three. At that point the example block is not a cost line, it is a hard constraint on the product, and the argument about removing it happens under deadline pressure with no measurements available.

What caching actually buys#

The obvious response is prompt caching, and it is the right first move. The example block is byte-identical on every request and sits at the front of the prompt, which is exactly the shape caching is designed for: the provider keeps the computed key-value cache for a prefix and, on a subsequent request beginning with the same bytes, reuses it instead of recomputing.

The property that makes the reuse legitimate is worth stating, because everything else in this section is a consequence of it rather than a separate fact to remember. The intermediate state for a stretch of input depends on that stretch and on nothing after it: the model builds the state for the first token before it has seen the second, and seeing the second never revises the first. So the state computed for a prefix is a function of the prefix alone. Two requests beginning with the same bytes must therefore have the same state across those bytes, and a provider holding the state from the first request can hand it to the second without repeating the work. Marking the boundary is a small change:

{
  "model": "extraction-model-v3",
  "system": [
    { "type": "text", "text": "You extract structured fields from contracts..." },
    { "type": "text",
      "text": "<examples>\n… 3,600 tokens of demonstrations …\n</examples>",
      "cache_control": { "type": "ephemeral" } }
  ],
  "messages": [
    { "role": "user", "content": "<document>\n… 900 tokens …\n</document>" }
  ],
  "max_tokens": 400,
  "temperature": 0
}

usage: {
  "cache_creation_input_tokens": 3800,   # first call: written to cache
  "cache_read_input_tokens": 0,
  "input_tokens": 902
}
usage: {                                  # second call, 30s later
  "cache_creation_input_tokens": 0,
  "cache_read_input_tokens": 3800,        # billed at a fraction of base rate
  "input_tokens": 902
}

What that buys is a discount and a latency win, both substantial. A cache read is typically billed at around a tenth of the base input rate, and a cache write at around 1.25 times, and the prefill for a cached prefix is skipped almost entirely, so the 450 milliseconds of example prefill largely disappears on a hit. The part that gets overstated is the hit rate. Cache entries have a short time to live — five minutes is a common default — so any gap in traffic longer than that costs a write on the next request. Traffic is bursty rather than uniform, requests are load-balanced across replicas and regions that may not share a cache, and every deploy that touches a single character of the example block invalidates everything. A well-behaved high-volume service lands somewhere around 75–85% hits; a service with overnight quiet periods and three regions lands lower.

Run the arithmetic at 78% hits, where each miss also pays the write premium:

The middle column is that arithmetic performed. At 78 hits in every hundred calls, 78 of them pay a tenth of the base rate and the remaining 22 pay one and a quarter times it. Seventy-eight hundredths of a tenth is a bit under eight hundredths; twenty-two hundredths of one and a quarter is a bit over a quarter; and those two add to roughly thirty-five hundredths, which is the 0.35 in the table. Adding rather than averaging is the allowed step, because the two shares are shares of the same hundred calls, so each rate has already been weighted by how often it applies.

Running the same sum at 95 hits in a hundred moves both terms. Ninety-five hundredths of a tenth is a bit under a tenth, five hundredths of one and a quarter is about six hundredths, and the two come to roughly sixteen hundredths — the 0.16 row. Why that improvement is so large, and why it stops, are both visible in the two sums. A miss costs more than twelve times what a hit costs, so the effective rate is set almost entirely by the misses, and removing misses is the only lever with any leverage in it. But every hit still pays a tenth of base, so a service with a perfect hit rate would pay a tenth and no less. Caching has a floor, and the floor is a tenth of where you started.

Applying the rate to the block gives the last column. Thirty-five hundredths of $7,200 is roughly $2,540. The two-example row is the same rate against a block a third the size — two examples of 600 tokens where there were six — so twenty-four hundred dollars of base cost rather than $7,200, which lands at about $850. And the headline saving is a division: $7,200 falling to $2,540 is a drop of a little over four and a half thousand dollars, and four and a half thousand out of seven thousand two hundred is close to two thirds.

That 65% saving is worth shipping on its own. It does not take the number to zero, it does nothing for the context budget, and it leaves the block's contribution to the bill larger than the user's own documents. The more expensive consequence of declaring the problem solved at that point is that nobody ever asks the other question, which is whether six examples were worth more than two in the first place.

Caching also has a design consequence that is easy to miss and worth building around from the start. It only helps a prefix that is identical byte for byte, so anything that varies per request must sit after everything that does not. That ordering rule follows from the property above rather than being a convention somebody chose: the reusable state is the state of a prefix, and a prefix ends at the first byte that differs. A varying value near the top of a prompt does not make the cache slightly worse; it ends the cacheable prefix before the examples begin, and the examples were the whole of the saving. A prompt that interpolates the current date, the tenant name, or a request identifier ahead of the example block destroys the cache on every call, and a service that A/B tests two example sets halves its own hit rate by splitting traffic across two prefixes. Ordering the prompt so that the stable material comes first is free at design time and expensive to retrofit, because by then the interpolated values have consumers.

Tokens are not the only thing the block costs#

Money and milliseconds are the costs that can be put in a table, and they are not the only ones the block carries. Six examples of 600 tokens is 3,600 tokens of hand-written specification that nobody reviews on a schedule, that no test asserts anything about, and that any engineer can edit without a second pair of eyes noticing what the edit implies. That maintenance surface is a real liability, and it grows with the block in a way that is worse than linear, because the failure modes come from interactions between examples rather than from examples individually. Two later chapters — one on what a single mislabelled demonstration does, one on what an unaudited example set turns into — are about blocks that accumulated that way, and the reason they belong in a topic about counting examples is that the cheapest example set to keep correct is the smallest one that scores well.

That framing also puts a bound on what else is worth trying. Fine-tuning a smaller model on the same labelled examples eliminates the block entirely and usually beats a six-example prompt on accuracy, at the cost of a training pipeline, a versioning story, and a much slower loop for fixing a new failure mode. It is the right answer at 50 million calls a month and the wrong answer at 50,000, and the crossover is roughly where the annual saving on the example block exceeds the engineering cost of owning a fine-tune. Having the cost table above is what lets you find that crossover instead of arguing about it.

Priced properly, an example block stops being part of the prompt and starts being a line item with an owner. The three costs are all computable from numbers you already have — token counts, call volume, the provider's rates, and a latency histogram — and the whole exercise is an afternoon. What it produces is not a recommendation to cut examples. It is the second column that makes the accuracy column mean something, and in the extraction service's case it argued for keeping every one of the six.

There is one thing wrong with the decision as just made, and it is not the arithmetic. Both columns are single aggregate numbers, and 1.5 points of aggregate accuracy can be spread thinly across every kind of input or concentrated entirely on one — the Portuguese tickets, the multi-issue documents, the customers onboarded last quarter. A trade that looks like a rounding error on the mean can be a catastrophe on the slice that generates every escalation, and the aggregate cannot tell you which of those you are buying.