Header tokens recur on every embed, query and re-ingest

A contextual header is a short string naming what a chunk is and where it came from — a document title and heading path, or a generated sentence saying the same thing — stored with the chunk and prepended to its text before that text is embedded. Costing one is harder than it looks, because a header is a handful of tokens and tokens are charged wherever they pass through something metered. The header passes through several such places, at different rates, on different occasions, and a costing that counts only the first of them will be wrong by more than an order of magnitude in the direction that gets a project funded.

The figures below work through one concrete proposal: one model call per chunk across 2.4 million chunks — 80,000 documents averaging 30 chunks each, spanning product docs, support tickets and PDFs of signed contracts — producing a 60-token contextual header. A costing deck for that proposal quoted the generation bill for one full pass, arrived at by multiplying tokens by a published price, and it came to about eleven thousand dollars. That figure is correct arithmetic on the wrong question, and the rest of this chapter is the right one.

Writing down everything that recurs#

Every term in this accounting has the same shape: a price per token, times a number of tokens per occasion, times a number of occasions. Multiplication is the right operation because the price does not depend on how many occasions there are and the occasion count does not depend on the price — they are independent, so the total is their product rather than anything more complicated. What separates the terms from each other is the third factor. The occasions are counted per chunk for one term, per query for another, and per night for a third, and those three counts are governed by variables that have nothing to do with one another.

Five terms, of which generation is one. Prices below assume Claude Haiku 4.5 for the generation calls at $1 per million input tokens and $5 per million output, cache writes at 1.25× input and cache reads at 0.1×; an embedding model at $0.10 per million tokens; and a generator at $5 per million input tokens. Substitute your own and the shape of the conclusion does not move.

Generation, initial pass. With the document in the cached prefix, as in chapter 3, the first chunk of each document writes the cache and the remaining 29 read it.

cache writes    80,000 docs x 24,000 tok x $1.25/M            =  $2,400
cache reads     2,320,000 chunks x 24,000 tok x $0.10/M       =  $5,570
chunk input     2,400,000 chunks x 800 tok x $1.00/M          =  $1,920
output          2,400,000 chunks x 60 tok x $5.00/M           =    $720
                                                                -------
                                                                $10,610

Two things in that block close exactly, and both are worth checking by hand before trusting anything built on top of it. The first is the chunk count. One chunk per document writes the cache, so the write row counts 80,000 chunks; the other 29 chunks of each document read it, and 80,000 documents times 29 is the 2,320,000 in the read row. Those two counts add to 2,400,000, which is the total chunk count used by the last two rows. Every chunk appears in exactly one of the two caching rows — none is charged both prices, and none is missed — so the caching arithmetic is complete rather than approximately complete.

The second closure is the column itself. The four row totals add to $10,610 with nothing left over and no rounding bucket at the bottom. What that licenses is division: because the bill is fully decomposed, dividing it by the 2,400,000 chunks gives a genuine unit price of about $0.0044 per chunk rather than an average with unaccounted costs hiding behind it, and that unit price is what every later estimate in this chapter is built from.

Note where the money is, because the intuition points the wrong way. Just over half of the bill is the cache-read row, and adding the cache-write row alongside it puts about three quarters of the total on the document prefix rather than on anything to do with the header. The mechanism is worth following: a cache read is priced at a tenth of the input rate, which sounds like the cheap line, but the thing being read is the whole 24,000-token document, which is thirty times the length of a chunk. A small unit price multiplied by a large token count still dominates a large unit price multiplied by a small one. So the generation bill is driven by document length rather than by header length. A corpus of 200-page contracts moves that line by an order of magnitude, and a corpus of short tickets makes generation nearly free per chunk while removing the caching benefit entirely, since a 900-token ticket sits below the model's cacheable minimum.

Embedding. Sixty extra tokens per chunk is 144 million extra tokens per full rebuild, or about $14. This term is real and negligible, and saying so plainly is more useful than omitting it, because a cost table with a tiny row in it is more credible than one where every row is alarming.

Index growth. Worth being precise here, because the intuition is usually wrong. Prepending to the embed input does not change the vector — a 1,536-dimension vector is 6 KB whether it came from 800 tokens or 860. Growth comes from storing the header string in the payload and from any lexical index built over it: 2.4 million headers at roughly 250 bytes is about 600 MB of payload, plus the postings a keyword index has to keep — one list per term, naming the chunks that contain it — over 144 million additional terms. The team in chapter 7 measured 38% index growth, and that figure was dominated by their hybrid lexical index and a second stored copy of the header; on a dense-only index the same change would have shown under 5%. Read your own number rather than inheriting anyone else's.

That vector-size point is the one place in this accounting where a longer input buys no extra storage, and the reason is worth one sentence. An embedding model maps an input of any length within its window onto a fixed-width vector, so the output has the same number of dimensions whatever went in. Everything that does grow — the payload copy, the postings list — grows because the header was stored as text somewhere in addition to being embedded, which is a storage decision rather than a consequence of embedding it.

Query-time context. Every retrieved chunk carries its header into the generator's prompt. At k=10, that is 600 tokens per query before a single word of chunk body:

600 tok/query x 40,000 queries/day x $5.00/M   =  $120/day  =  $43,800/year

The chain behind that line is short and each link is checkable. The header is stored with the chunk, so what is retrieved is header-plus-body rather than body alone. What is retrieved is what gets pasted into the generator's prompt, so the header is charged at the generator's input rate. Ten chunks are retrieved per query, so the per-query header load is ten headers of sixty tokens each, which is the 600 above. That per-query figure then multiplies by traffic rather than by anything about the documents.

This is four times the entire initial generation bill, it recurs forever, it scales with traffic rather than with corpus size, and it is the term that appears in no proposal. It also has a cost that is not denominated in dollars: 600 tokens of the generator's context budget spent on provenance rather than on evidence, on every turn of every conversation that keeps its retrieved context.

The same figure can be reached from the other end, and the agreement is a useful check rather than a coincidence. Six hundred tokens at the generator's input rate is three tenths of a cent per query. A year of traffic at forty thousand queries a day is a little under fifteen million queries, and the $43,800 divided by that count comes back to the same three tenths of a cent. Because the two routes are the same statement scaled differently, quoting the annual figure and the per-query figure is one argument rather than two — which matters when somebody asks for both and takes their agreement as corroboration.

Re-ingestion. Nightly, at 4% of documents changed. The naive implementation regenerates every chunk of every changed document: 96,000 chunks a night, 35 million a year, at the initial pass's $0.0044 per chunk — about $154,000 a year, fifteen times the number in the deck. With the content-hash cache from chapter 3, only chunks whose text actually changed regenerate, and a document edit typically touches a small fraction of its chunks; at 8% of chunks within a changed document, that is 7,700 chunks a night, 2.8 million a year, about $12,400. The cache is worth thirteen times its implementation cost in the first year.

Follow the naive figure through, because it is the one that decides whether this project has an ongoing bill or a one-off one. Four per cent of 80,000 documents is a little over three thousand documents a night. Each averages 30 chunks, and the naive path regenerates all of them, which gives the 96,000 chunks a night. Multiplying by a year of nights gives the 35 million, and multiplying that by the $0.0044 unit price derived from the closed column above gives the $154,000. The hash-cached path changes exactly one factor in that chain — the fraction of a changed document's chunks that actually need regenerating — and every other term stays where it was, which is why the saving is close to the ratio between those two fractions and nothing subtler.

The last column is the one to read, and reading the table by gaps down that column rather than by levels down the dollar column is what turns it into a decision. Three different variables appear there, and no two of them move together. The gap that matters is between the generation row and the generator row: one is a one-off governed by how many documents exist, the other recurs and is governed by how many questions get asked, and a project that doubles its traffic without adding a document doubles the second while leaving the first untouched. A gap between two dollar figures that scale with different variables is not a ratio you can carry into next year.

That column also names the stumble this chapter exists to prevent, and the stumble is a belief that a defence works rather than a slip in the arithmetic. "We costed it" feels like a finished defence: a real price list was used, the multiplication was done correctly, and the answer was a specific number of dollars. What that defence cannot do is discriminate. A one-off generation bill is constant with respect to query volume, with respect to k, and with respect to churn — so it returns about eleven thousand dollars for a system serving a hundred queries a day and about eleven thousand dollars for one serving forty thousand, which are the two cases the costing was supposed to tell apart. A figure that comes out the same across the situations it must distinguish has not answered the question conservatively; it has not answered it.

It also helps to separate what the design forces from what somebody chose, because only the second kind is negotiable. Charging the header at the generator's input rate on every retrieved chunk is a requirement of storing the header with the chunk and serving stored text to the model; nothing about prompt construction avoids it while keeping the header. A model call per chunk is a requirement of choosing generated headers rather than templated ones. Against that, k=10 is a design decision, the nightly refresh cadence is a design decision, and regenerating every chunk of a changed document rather than hashing its chunks is a design decision — which is precisely why the content-hash cache can convert a term scaled by document churn into one scaled by true chunk churn, and why no amount of engineering can do the same to the query-time row.

The honest answer to finance is a per-query figure and a per-refresh figure — roughly $0.003 of header overhead per query, and about $34 a night to keep headers current — with the mitigation stated alongside: template the header from the heading path wherever the path is informative, and spend model calls only on the population from chapter 3 where it is not. Applied to a corpus that is two thirds structured, that removes about two thirds of both generation lines and none of the query-time line, which stays whatever the header's provenance.

One column is missing from that table because it is not denominated in tokens. Generated headers add a model dependency to the ingestion path, which means ingestion now has a failure mode it did not have — rate limits, timeouts, a provider incident — and needs retry, backoff, partial-batch resumption, and a decision about what to do with a chunk whose header generation failed after three attempts. It adds the routing predicate from chapter 3 as versioned configuration, the content-hash cache as a store to operate, and a sampling check on generated headers as a recurring quality task. On a team where one engineer owns ingestion, that is a meaningful fraction of a person indefinitely, and it is the cost most likely to be discovered rather than budgeted. The templated path adds none of it.

The experiment that decides the design#

A before/after test answers a question nobody is asking. Headers against no headers will show a gain, and that gain will be used to authorise per-chunk model generation, when the comparison that actually decides the design is generated headers against a template that costs nothing per chunk. Three arms, not two.

The reason the two-arm test cannot decide it is structural rather than statistical. Both a template and a generated sentence put a subject line above a chunk that lacked one, so both move the same failures. A test whose control is "no header at all" therefore credits the entire gain to whichever arm was run, including the part a free template would have delivered on its own. The templated arm is the control that separates the effect of having a header from the effect of having paid for one.

Running it at full scale costs the whole initial pass, and it does not have to. Stratify a subcorpus, which means drawing a fixed quota of chunks from each source rather than sampling the corpus at random, so that the two sources you most need to hear from are not represented by a handful of chunks each:

SAMPLE_PER_SOURCE = {"product_docs": 120_000, "tickets": 50_000, "contracts": 30_000}

sub = stratified_sample(all_chunks, SAMPLE_PER_SOURCE, seed=17)
queries = [q for q in FROZEN_QUERIES if q.gold_chunk_ids & {c.id for c in sub}]

arms = {
    "none":      [c.text for c in sub],
    "template":  [f"{c.header}\n\n{c.text}" for c in sub],
    "generated": [f"{gen_header(c)}\n\n{c.text}" for c in sub],
}
results = {name: recall_at_k(build_index(texts), queries, k=10, filter_fn=tenant_filter)
           for name, texts in arms.items()}

Two hundred thousand generations is about $880 and a few hours. That price is not a fresh estimate: the three quotas in SAMPLE_PER_SOURCE add to two hundred thousand chunks, and two hundred thousand at the $0.0044 unit price divided out of the closed column above gives the $880. The sample bill and the full bill are therefore the same accounting at two scales, which is what makes the experiment safe to authorise on the strength of the earlier table — if the unit price is wrong, both are wrong by the same factor and the ratio between them still holds.

The absolute recall numbers from a 200k-chunk index are optimistic, because a smaller index holds fewer distractors — wrong but plausible chunks competing for a place in the top k — so do not compare them against the production 0.58; the differences between the three arms are what the experiment is for, and those hold up because all three arms face the same distractor population. Report per source type as well as pooled, since the entire question is whether the sources where the template fails are numerous enough to justify generation everywhere else.

The decision rule follows from the numbers already on the table. If generated beats template by less than a couple of points, template everything and stop. If it beats template only on tickets and contracts, route by the predicate and spend generation on a third of the corpus. If it beats template everywhere by a wide margin, you have earned the full bill and can now defend it with the annual figure rather than the one-off.

Whether a 0.58 is a header problem at all#

Chunking was set in an earlier pass: fixed 800-token windows with 100-token overlap, applied uniformly to product docs, support tickets and scanned contracts. That single sentence is the strongest signal in the whole brief, and it points away from headers.

Before funding anything, read the failures. Take the queries that miss at k=10, pull the gold chunks, and classify by hand — a hundred of them is an afternoon and is the highest-value afternoon in this project:

misses = [q for q in FROZEN_QUERIES
          if not (q.gold_chunk_ids & {p.id for p in idx.search(q.text, k=10)})]

for q in random.sample(misses, 100):
    for cid in q.gold_chunk_ids:
        print(q.text, "\n  GOLD:", store.retrieve([cid])[0].payload["text"][:400], "\n")

A representative classification of a hundred such failures on a corpus like this one:

That table closes too, and the closure is what makes its third column usable. The four counts add to exactly the hundred failures that were read, with no residual class and no bucket labelled "other", so every failure in the sample has been assigned. That in turn means the third column partitions the sample rather than sampling it: the two rows marked "No" account for three fifths of the failures between them, the header-fixable row is a little over a fifth, and the "Partly" row is the rest. A funding argument for headers is therefore an argument about a fifth of the observed failures, and that is a statement the table supports outright rather than one somebody has to be persuaded of.

Forty-one of a hundred failures are boundary failures, and a header describes a chunk without moving a boundary. Uniform 800-token windows are the cause: a support ticket is a conversation whose exchanges are the natural unit and whose whole thread is often shorter than one window, so unrelated exchanges get packed together while a long thread gets cut mid-diagnosis; a signed contract has clauses and defined terms whose meaning depends on a definitions section 40 pages away; product docs are the one source the setting was probably chosen for. Per-source chunking — thread-aware for tickets, clause-aware for contracts — addresses the largest class, and it changes chunk ids, which means it invalidates the header cache and any evaluation labels tied to those ids, so it wants to happen before the header work rather than after.

Nineteen more are coverage failures, where no amount of retrieval engineering helps because the answer is not in the corpus. Those belong in a different conversation, with whoever owns the documents.

Small corpora, and a distribution that would reverse this#

It is worth naming when none of this applies. On a corpus of fifty thousand chunks with a hundred queries a day, the entire annual bill above is a rounding error and so is the alternative of simply raising k, and the right move is usually to spend the effort on the documents instead. These numbers become decisions somewhere in the low millions of chunks, or as soon as query volume is high enough that the per-query line dominates — which, as the table shows, happens sooner than corpus size does.

What would change the conclusion about headers is the failure distribution itself. If subject-loss is the largest class rather than the second-largest — which is what a ticket-heavy or heavily-numbered corpus produces, since those are exactly the sources where nothing in the chunk names its own subject — then headers are the right lever and the generated arm is likely to win, because those are also the sources the template cannot serve. The classification is cheap and it is the only thing that tells you which of those worlds you are in.

The prices are the other thing that would move it, and they move unevenly. Every term here is a rate times a count, so halving a rate halves its term and leaves the others alone. Generation rates have historically fallen fastest, which shrinks the two lines a proposal worries about most; the generator-context line falls only as fast as the rate charged for the model actually answering questions, and that model is chosen for answer quality rather than for price. A future in which generation is nearly free and the query-time line is unchanged is the one this accounting is most exposed to, and it makes the per-query figure more load-bearing rather than less.

The pieces are all in hand. Using them well is mostly a matter of order. Carry the heading path through the splitter and prepend it to the embed input; keep the header short enough that it discriminates chunks from their siblings rather than binding them together; route to model generation only for the population where the path is empty, numbered or shared by a long run of chunks, and cache those generations on a content hash so the nightly refresh pays for what changed. Store doc_id, revision_id, section_path, effective_date, source_type and the ACL reference as typed payload fields, and resolve permissions in a filter built from live group membership inside a single search entry point that fails closed. Freeze a query set and a label file before touching any of it, and hold k fixed when you compare.

Then measure the thing you are about to spend money on against the free alternative, on a sample, and read the failures before you believe any of it. The number to carry out of this chapter is not the eleven thousand dollars or the $43,800; it is the habit of asking, of every term, how many times it is charged and what makes that count go up. A header bought once and charged on every query is priced by traffic, a header regenerated nightly is priced by churn, and only the first pass is priced by the corpus — which is why a costing that reports a single total has, whatever its arithmetic, answered a question nobody will be asked again after launch week.