Prompt block ordering

A prompt sent to a hosted model is assembled from blocks — a system policy, a set of tool definitions, some retrieved documents, a user profile, the question — and the assembler that concatenates them usually grew one block at a time without anyone treating the order as a decision. Once prompt caching is switched on it becomes one, because the provider matches a request against a stored entry by walking its serialized bytes forward from the first token and stopping the moment they disagree with the entry's. The position where they disagree is the divergence point, and everything after it is fresh input at the ordinary rate.

That makes block ordering the cheapest lever in this topic and the one with the widest range of outcomes. One set of blocks, byte-for-byte unchanged in content, can produce a shared prefix of forty tokens in one arrangement and 23,000 in another.

The divergence point is the only number that matters#

Two requests share a cache entry for exactly as long as their serialized bytes agree, starting at the first one. The moment they differ, the match stops. There is no mechanism for resuming a match after a gap, no diffing and no alignment — the comparison is a forward walk that ends at the first mismatch and cannot restart. So the useful measurement of any prompt layout is a single number: for two requests drawn from your traffic, how many leading tokens do they have in common?

None of that is a limitation somebody chose to leave in, and the reason it cannot be relaxed is worth carrying forward from chapter 1. The value a lookup is keyed on is computed run of tokens by run of tokens, each one from the bytes of that run and from the value computed for the run before it. Every value therefore depends on every byte in front of it. Change one byte and every value from that point on becomes a value nothing was ever stored under, so there is no entry to resume from — and no state that would be correct even if there were, since the keys and values at those positions were computed while attending to the bytes ahead of them and those bytes have moved.

Read that dependency in the direction an assembler cares about and the ordering rule falls out of it. A block's contents feed into the lookup value of every block placed after it, and into none of the values before it. A block that varies between requests therefore has no effect whatever on the reusability of what sits in front of it, and destroys the reusability of everything behind it — however stable that material is, and however byte-identical it happens to be between the two requests. The cost of a volatile block is not measured by its own length. It is measured by the length of what follows it.

That is the sentence that turns ordering into a design decision rather than a matter of taste. What has to be in a prompt has to be in it, and no amount of sorting removes a block that the product needs. What is not fixed is which blocks end up sitting behind the volatile one, and that is decided entirely by where things are put. The same block is cheap at the end of a prompt and ruinous at the front.

Take a shopping assistant that assembles every request from four blocks. There is a product catalogue of about 20,000 tokens, regenerated by a batch job every Monday morning. There is a tool schema block of roughly 3,000 tokens, changed only when someone ships a new tool. There is a per-user preference profile of about 500 tokens — sizes, dietary flags, favourite brands, the last three orders. And there is the shopper's question, around 200 tokens. Written out with the divergence marked, an arrangement that puts the profile first stops being mysterious and becomes arithmetic.

Request from shopper A          Request from shopper B
────────────────────────        ────────────────────────
[profile: A, 500 tok]     ✗ diverges here
[catalogue: 20,000 tok]         [catalogue: 20,000 tok]   ← identical, unreachable
[tools: 3,000 tok]              [tools: 3,000 tok]        ← identical, unreachable
[question: 200 tok]             [question: 200 tok]

common prefix: ~40 tokens (the profile header, before the name)

The catalogue and the tool schemas are byte-identical between the two requests, and both sit behind a block that is not, so both are worth nothing. Reordering the same four blocks without changing a character of their contents moves the divergence point from token 40 to token 23,000.

Put in the terms of the paragraphs above, the profile is a 500-token block whose cost is 23,000 tokens. That is the arithmetic people get wrong when they first look at a layout like this one, and the reason they get it wrong is that every other cost in a prompt really is proportional to the size of the thing. Adding a hundred tokens of instructions costs a hundred tokens. Moving five hundred tokens of personalisation to the front costs whatever happens to be standing behind it, which is a quantity the block itself gives no hint of.

The second column is the one that decides the bill for a storefront, because such a service serves far more distinct shoppers in a five-minute window than it serves repeat questions from a single shopper. Under the first order the cacheable span shared across shoppers is forty tokens, which is below every provider's minimum and therefore zero. Under the last two it is 23,000 tokens shared by every request in the system.

Therefore zero is doing something specific there and it is not a rounding convention. Providers refuse to cache a span below a published minimum, in the region of a thousand tokens, because an entry carries bookkeeping of its own and a short one never repays it. So forty shared tokens and zero shared tokens bill identically. There is no partial credit between them, which is why the top row of that table is not a worse outcome than the bottom two but a different kind of outcome altogether.

Why the last two rows tie, and why no other arrangement beats them, is worth settling rather than accepting from the table. Which blocks two requests share is decided by the contents and not by the order — shopper A and shopper B share the catalogue and the tools whatever sequence they are written in. What the order decides is how many of those shared blocks form an unbroken run starting at the first token, because that run is the whole of what a lookup can reach. Placing every shared block before every unshared one makes the run the entire shared set, which is the largest it could possibly be. Any other arrangement puts at least one unshared block in front of at least one shared block, and truncates the run there.

For a single pair of requests that argument is complete. A population is harder, because different pairs share different sets and one order has to serve all of them at once. The rule in the next section — sort by how many requests carry an identical copy, most first — is exactly optimal when the audiences are nested, meaning each block's audience contains the audience of every block placed below it. The shoppers who share the catalogue include the shoppers who share any one profile, who include the single shopper asking any one question. Nesting is the condition that lets one sort be right for every pair at once. Where two blocks have audiences that overlap without one containing the other, no order is best for every pair, and the honest resolution is to sort by the larger audience and accept that the smaller one is served worse. That case is rare enough to be worth naming rather than designing around.

Sorting blocks by how many requests share them#

The rule that produces the right order is to place blocks in descending order of the number of requests that carry an identical copy. A catalogue and a tool schema shared by every request from every shopper go first. A block shared by everyone in one tenant — one customer organisation on a platform, whose users all get the same configuration — goes after those, because it is common to thousands of requests and not to all of them. A per-user profile is shared by that user's requests only, which in a one-shot storefront is often exactly one request. The question is shared by nothing.

Change cadence is the same ordering in a different vocabulary, and it is usually the easier one to apply in review, because you can ask of each block: what makes this change? A tool schema changes on deploy. A catalogue changes weekly. A tenant playbook changes when an account manager edits it. A profile changes when the shopper updates their preferences, which from the endpoint's point of view is per request, since a request is where you discover the current value. Anything that changes per request belongs after everything that does not.

# Blocks carry the breadth of their audience, and the assembler sorts on it.
BLOCKS = [
    ("tools",     0, render_tool_schemas),      # every request, every tenant
    ("catalogue", 0, render_catalogue),         # every request, refreshed weekly
    ("playbook",  1, render_tenant_playbook),   # every request from one tenant
    ("profile",   2, render_user_profile),      # every request from one user
    ("question",  3, render_question),          # this request only
]

def build_prompt(ctx):
    ordered = sorted(BLOCKS, key=lambda b: b[1])
    return [{"type": "text", "text": render(ctx)} for _, _, render in ordered]

Sorting inside the assembler rather than relying on the order of the literal list is worth the four lines, because the next person to add a block appends it to the bottom of the list and gives it a tier, instead of appending it to the bottom of the prompt and hoping that was right. The tier is also the thing a reviewer can check: a new block declared tier 0 had better be genuinely identical for every request in the fleet.

Breadth and cadence disagree occasionally, and the disagreement is usually resolved in favour of breadth. A catalogue shared by every request and regenerated weekly competes with tool schemas shared by every request and changed monthly. Putting the catalogue first means every Monday's regeneration invalidates the tool schemas behind it as well, which costs one extra fill per node per week — a few tens of thousands of tokens, against nothing measurable. Putting the tool schemas first costs nothing and avoids that, so it is the better default, but the margin is small enough that it should never hold up a review. The disagreement that does matter is between a block shared by everyone and a block shared by one tenant, because there the ratio is thousands of requests to one and the wrong order forfeits the shared span entirely.

Two details change the ranking in ways that are easy to miss. The first is that serialization order counts as content — a tool block assembled by iterating a dictionary in insertion order is stable only while the insertion order is, and a JSON encoder that sorts keys on Tuesday and does not on Wednesday produces two different byte sequences from the same logical schema. The second is that a block being logically constant is not enough; it has to be rendered identically. A catalogue regenerated from the database on every request, with the same rows in an unspecified order, is a per-request block wearing a weekly block's costume.

Those are the same failure seen twice, and the chain is why neither is forgiving. A run's lookup value is computed from its bytes, so two byte sequences a human would call the same schema are two unrelated inputs producing two unrelated values. Swapping two keys in a JSON object is not a small difference costing a small amount of reuse. It is the first differing byte, and everything behind it goes. What a reviewer checks when they see a tier-0 declaration is therefore not whether the block means the same thing on every request. It is whether the block serialises to the same bytes, which is a question about encoders and iteration order rather than about content.

Reading the shared span out of request logs#

The ordering argument is easy to have in the abstract and easy to settle with data, because the divergence point is computable from request logs without calling the model at all. Log the fully rendered prompt string for a sample of a few hundred requests, sort them, and compute the length of the common prefix between each adjacent pair; the median of those lengths is the span your traffic actually shares. Sorted order puts strings with long common prefixes next to each other, so the pairwise maximum is a good estimate of the best available sharing and the median is a good estimate of the typical sharing.

Sorting is the right instrument here for the same reason the cache works, which is what makes it a measurement rather than a heuristic. Lexicographic order is decided by the first position at which two strings differ, which is the same forward walk the lookup performs. Sorting therefore arranges the sample by precisely the quantity being measured. Two strings that agree for 23,000 characters cannot be pushed apart in that order by a third string that diverges at character 40, because the third string sorts either before both of them or after both of them. Adjacent pairs are consequently the pairs with the longest common prefixes available anywhere in the sample, and no pairing chosen some other way would have found longer ones.

Two things the measurement does not capture are worth stating before its output gets treated as a forecast. It counts characters, because a logged prompt is a string, while the provider matches tokens — so the figure is a proxy, dependable for telling forty apart from twenty-three thousand and not for deciding whether a span clears a minimum by a hair. And it says nothing about whether entries survive between requests. A median common prefix of 23,000 over an hour of logs is entirely compatible with a hit rate of zero, if the requests carrying that prefix arrive further apart than an entry lives.

Run it per tenant as well as globally, since a global median of 23,000 hides the case where one large tenant injects a header and gets forty. The same measurement after a change is the only honest way to confirm a reorder worked, because the quantity you care about is the shared span across a population and no single request can report what that is.

That measurement also settles the question chapter 1 left open about a one-shot endpoint. Fifty different users a minute sharing a 15,000-token system prompt is the best available caching population, and the only thing that turns it into the worst is a block ordering — or a single interpolated field — that pushes the divergence point to the front. When someone says a one-shot endpoint cannot benefit from caching, the check is not an argument about conversation structure; it is the median common prefix over an hour of logged requests, and it comes back either near 15,000 or near zero.

What the ordering buys, in tokens#

Put the shopping assistant at 30 requests a minute across 900 distinct shoppers an hour, on a provider that writes at 1.25× and reads at 0.1× the base input rate. Thirty a minute for sixty minutes is 1,800 requests an hour, so those 900 shoppers ask two questions each on average. Roughly half the traffic is therefore somebody's first request, which is why the arithmetic below is built on what two different shoppers share rather than on what one shopper shares with themselves.

Under the profile-first layout every request bills 23,500 tokens of prefix at the write rate, which is 29,375 effective tokens, plus the question. The prefix is 23,500 because the profile's 500 tokens sit in front of the catalogue's 20,000 and the tools' 3,000, and all of it is a write rather than a read: the profile makes the first block unique to one shopper, so nothing that gets stored is ever found again. Adding the 200-token question at full rate gives the 29,575 in the first row of the table below.

Under tools → catalogue → profile → question, one request per window pays 23,000 × 1.25 = 28,750 to fill the entry and every other request pays 23,000 × 0.1 = 2,300 plus 700 tokens of profile and question at full rate. The 23,000 is the tools and the catalogue with nothing volatile in between them; the 700 is the profile's 500 and the question's 200, neither of which is shared with anybody. So a reading request costs 2,300 + 700 = 3,000 effective tokens, which is the last row.

The fill is left out of that 3,000, and it is worth checking that leaving it out is fair rather than convenient. Thirty requests a minute against a five-minute entry puts a hundred and fifty requests behind each fill. The fill costs 28,750 where a read costs 2,300, an excess of a little over twenty-six thousand effective tokens, and spreading that across the window adds under two hundred to each request. Against 3,000 that is visible; against the 23,700 the comparison is really being made against, it is not. So the table quotes the steady-state read, and the relative column still reads 0.13×.

The middle row is the one worth pausing on. Turning caching on without fixing the order is strictly worse than leaving it off, because the endpoint pays the write premium 1,800 times an hour to create entries that nothing ever reads. A hit rate of zero on a cached endpoint is never a neutral outcome; it is a 25% surcharge on the prefix.

The relative column is measured against that middle row rather than the top one, because 23,700 is what the endpoint costs with the feature switched off and switching it off is the decision actually available. Read that way, 29,575 against 23,700 is the 1.25× surcharge — a quarter more on the prefix, and the prefix is nearly the whole request. And 3,000 against 23,700 is a little over an eighth.

Two beliefs make that outcome hard to see coming. The first is that reordering blocks is cosmetic. It is cosmetic as far as the content goes and it is not cosmetic on the invoice: not a character changes, the model receives the same instructions in a different sequence, and the input cost moves by nearly a factor of eight. The second is that a hit rate of zero is the same as not caching, which is exactly where a team lands after switching the feature on and doing nothing else. On a provider that charges to write, the writes are all still happening and all still billed, so zero hits puts the endpoint on the top row rather than the middle one.

Per-request blocks and forced suffixes#

Ordering cannot make a per-request block cacheable. If a shopper's profile genuinely has to be in the prompt, those 500 tokens are prefilled and billed at full rate on every call regardless of where they sit; the ordering decides what happens to the 23,000 tokens around them, not to the 500 themselves. Teams sometimes chase that last block into the prefix by bucketing users — rounding a profile down to one of twelve segment templates so that segment 7's block is shared by everyone in segment 7. That is a real technique with a real cost in personalisation quality, and it is a product decision rather than a caching one.

Ordering also does not override the reasons a block might need to be somewhere specific. Retrieved documents placed at the very top for attention reasons were covered in topic 1.10, and the resolution there — keep quality levers in the suffix, where they are already cheap — is the same resolution here, which the ranking above assumes. What the sort cannot do is negotiate with a requirement that some instruction must appear after the user's message, which occasionally happens for jailbreak resistance, meaning the practice of keeping a user's own text from talking the model out of the instructions it was given. Those instructions live in the suffix and are paid for at full rate, which is fine when they are eighty tokens and a problem when someone proposes moving the whole policy down there.

Block order is therefore the first thing to fix and the cheapest, because it changes no content, breaks no evaluation and can be verified from logs before a single request is sent. The failure it prevents is the one where a prompt is entirely stable, entirely shared, and entirely unreachable because five hundred tokens of personalisation sit at the front of it.

The ordering also only pays if the provider is actually caching the span you arranged. Everything above assumes that a long stable prefix at the front of a prompt becomes an entry that later requests can read, and whether that happens without being asked is a property of the provider rather than of the prompt. Under one common provider design it happens on its own with no markers of any kind; under another, nothing is cached until you explicitly say where the reusable region ends, and a perfectly sorted 23,000-token prefix earns exactly nothing because nobody asked for it. Which of those two worlds an endpoint is in changes what else has to be done, so it is the next thing to establish.