Selecting by marginal contribution

Everything a model has to work with when it answers is the assembled prompt sitting in its context window, which is finite. An evidence block retrieved for the question shares that space with the system instructions, the conversation history, the tool definitions and the tokens held back for the answer itself, and the material competing for it is not finite at all. Deciding what occupies it is context selection, the branch of context engineering this topic is about: choosing, block by block and request by request, which material is actually sent, and on what evidence. It exists because the assembler is permanently over-subscribed — there is always more plausibly useful material than room, and every block admitted displaces either another block or part of the answer. The budget work in the previous topic fixed how many tokens each component may claim; selection is the question of what to spend that allocation on, and it cannot be settled once and filed away, because the right contents differ per request while the budget does not.

You meet selection wherever a value in a config decides how much material a request carries. top_k in a retrieval config is the clearest instance — how many passages a retriever pastes into the prompt after searching a pre-indexed copy of a document set, cut into passages, or chunks, of roughly 400 tokens each. The others in this topic are the same decision in different clothes: whether a 12,000-token policy manual is pasted into every prompt or fetched when a rule applies, whether six worked examples stay in a classifier prompt, which tool definitions are exposed to the model, whether a library of product notes lives in the system prompt or in an index. Those values are usually set once by whoever built the feature and then inherited by everyone after, which is how a dial ends up at a number nobody has measured. The test people reach for when they do finally turn one is relevance, and relevance is the wrong test.

Twenty on-topic passages, six distinct facts#

Here is one regressed question from a support bot built on retrieval-augmented generation, or RAG — the model was never trained on the company's help centre and cannot see it, so the service searches an index of those articles on every question and pastes the best-matching passages into the prompt above the customer's question. The customer asked why they had been charged twice for the same order. The last column is the only one that matters: what this passage tells the model that the passages above it had not already told it.

Twenty passages, 8,235 tokens, six distinct facts. Ranks 1 through 6 carry five of the six in 2,420 tokens. Rank 13 carries the sixth. The remaining thirteen passages carry nothing that was not already present, and they cost 5,390 tokens to say so. A k=8 arm gets five facts for roughly 3,200 tokens; the k=20 arm gets six for 8,235.

Those are additions rather than assertions, and the table holds everything needed to reproduce them. The token counts at ranks 1 through 6 are 410, 395, 380, 430, 360 and 445, and they add to 2,420. Rank 13 costs 425. Take both of those away from the 8,235 the twenty passages occupy and 5,390 is left, which is what the thirteen passages at ranks 7 through 12 and 14 through 20 charge for repeating material that is already in the prompt. The two grouped rows in the table, 2,410 and 2,980, add to that same 5,390 from the other direction, and the two routes agreeing is the check that the figures are internally consistent rather than quoted.

The k=8 number is the one that has to be estimated rather than read off, and it is worth seeing why it is approximate. The table groups ranks 7 through 12 into a single row of 2,410 tokens covering six passages, so the individual sizes of ranks 7 and 8 are not recorded; six passages sharing 2,410 tokens run to roughly 400 each, which puts the pair at about eight hundred between them. Add that to the 2,420 that ranks 1 through 6 cost and an eight-passage prompt carries about 3,200 tokens of evidence. It carries five facts rather than six, and the reason is positional rather than statistical: the sixth fact is the 30-day issuer exception, it sits at rank 13, and a cut at eight never reaches it.

That trade — one extra fact for 5,000 extra tokens — is not obviously bad. It became bad for three reasons that are worth separating, because they have different fixes.

The first is what the restatement does to the model's sense of what is important. The 3–5 day window appears in seven of the twenty passages. The 30-day issuer exception appears in one. A model reading that prompt is reading something that looks like a consensus and a footnote, and it answers accordingly: it says three to five business days and omits the exception. Adding rank 13 did not help, because ranks 14 through 20 buried it.

"The model reads that as a consensus" is itself an assertion, and the step under it can be stated. A model receives the prompt as one sequence of tokens. It holds no record of which passage each sentence arrived in, and it has no operation that recognises two identical sentences as copies and counts them once. Seven statements of the 3–5 day window are therefore seven parts of the prompt agreeing with each other, which is exactly the form that a genuine weight of evidence would take. The 30-day exception is one part disagreeing with all of them. Nothing anywhere in the assembled request marks the seven as duplicates of one source, so the shape a retrieval quirk produced is indistinguishable from the shape a settled policy would produce — and an answer that follows the shape is the model doing the sensible thing with a prompt that misrepresents its own evidence.

The second is distance. The instruction that governs the answer — answer only from the passages, cite the article ID, and say so explicitly if the passages disagree — sits above the evidence block, and the customer's question sits below it. At k=8 there are about 3,200 tokens between the instruction and the question. At k=20 there are 8,235. Instruction placement was the subject of the previous topic, on the order in which blocks are assembled, and the effect is the one you would expect: the further an instruction sits from the material it governs, the less reliably it governs it. On the regressed questions, citation compliance fell from 94% to 81% between the two arms, which is a formatting failure caused entirely by adding correct evidence.

Distance does that for a reason that can be stated without appealing to anything mysterious about long contexts. The instruction is one constraint on what gets written. Every passage between it and the question is also material the model is weighing when it decides what to write, and 5,000 extra tokens of passages do not weaken the instruction's wording by a single word. What they change is the ratio: one short block of instruction against a much larger block of evidence, with the question at the far end of it. The eval measured the outcome — citation compliance at 94% in one arm and 81% in the other — rather than the mechanism, and it is worth being honest that a drop of that shape is consistent with more than one account of how attention is spent. What it is not consistent with is the instruction having got worse, because it is the same bytes in the same place in both arms.

The third is arithmetic. The allocation built in the budget work gives retrieval a fixed slice, and 8,235 tokens of passages against a slice sized for eight chunks does not fit. Something gets evicted, and it is not the passages — they are what you just asked for. Raising k is never a local change, because the assembler has to find those tokens somewhere.

The size of that miss is worth putting next to the slice rather than leaving as a word. The greedy pass below is given 4,000 tokens to spend, which is the order of a slice sized for eight chunks. Eight passages at about 3,200 tokens fit inside it with room left over. Twenty passages at 8,235 miss it by more than the whole slice is worth, so the shortfall cannot be found by trimming retrieval at the margin; it has to come from another component entirely. The assembler is not permitted to send fewer passages than were requested, because passages are precisely what the request asked for, so the tokens come out of whichever block the eviction policy is willing to shrink. Raising k is therefore two changes wearing one name: a change to retrieval, and a silent change to something else that is specified in a different file.

Scoring a block by what it adds#

Before the fix, the stumble is worth naming, because it is not carelessness and it survives being told the answer. Relevance feels like the right test because it is the right test for the question a retriever is actually asked: given this query and this one passage, how well do they match. Nothing in that computation is broken, and the support lead who found all 800 passages on topic was confirming that it worked. The false step is the one taken immediately after — that a set whose every member passes the test is a set that passes the test. Relevance is a property of a pair. What gets sent is a set, and a set has properties that no member of it has, redundancy among them. Twenty passages that are individually the best available answer to "is this relevant" can jointly be a poor answer to "what should the model read", and no improvement to the pairwise score touches that, because the pairwise score was never asked the second question.

The fix is to stop scoring passages independently. A retriever hands you a ranked list where every score is a function of one passage and the query; nothing in that computation knows what the passages above it already said. Selection is the step where you introduce that knowledge, and it is a straightforward greedy pass: walk the ranked list, and admit a passage only if it contributes something the already-admitted set does not have.

from dataclasses import dataclass

@dataclass
class Passage:
    id: str
    tokens: int
    score: float          # the retriever's relevance score
    claims: frozenset[str] # atomic facts, extracted once at index time

def select_by_marginal_gain(
    ranked: list[Passage],
    budget_tokens: int,
    min_new_claims: int = 1,
) -> list[Passage]:
    """Admit a passage only if it adds claims the selection does not have."""
    chosen: list[Passage] = []
    covered: set[str] = set()
    spent = 0

    for p in ranked:
        new = p.claims - covered
        if len(new) < min_new_claims:
            continue                      # relevant, but redundant
        if spent + p.tokens > budget_tokens:
            continue                      # try the next one; it may be smaller
        chosen.append(p)
        covered |= new
        spent += p.tokens

    return chosen

Run that over the twenty passages above with a 4,000-token budget and it returns ranks 1, 3, 4, 6 and 13 — five passages, 2,090 tokens, all six facts. The 30-day exception survives instead of being drowned, and there are 1,900 tokens of the retrieval slice left over that the assembler can give back to conversation history.

Walking the loop by hand over the ranked list is what turns that result from a claim into something predictable. Rank 1 arrives with covered empty, so both of its claims are new, and it is admitted; covered now holds them and spent stands at 410. Rank 2 restates the 3–5 day window, so p.claims - covered is the empty set, len(new) is zero, and the continue skips it without touching the budget — a passage the retriever ranked second, that a support lead would confirm as on topic, discarded on the only ground that matters. Rank 3 adds how the two lines render on a statement and goes in. Rank 4 adds the automatic refund. Rank 5 is skipped for the same reason as rank 2. Rank 6 adds capture timing. Ranks 7 through 12 add nothing at all. Rank 13 adds the 30-day issuer exception, and by that point well under half the slice has been spent, so the budget test passes comfortably and it is admitted. Ranks 14 through 20 are localised copies and contribute nothing.

The admitted set is ranks 1, 3, 4, 6 and 13, and its cost is the sum of their token counts — 410, 380, 430, 445 and 425 — which is 2,090. Against a 4,000-token slice that leaves a little over 1,900 tokens unspent, and the loop simply returns rather than hunting for something to fill them with. That last part is the behaviour worth noticing, because it is the one a top_k control cannot produce. A count-based control fills the slice by construction: ask for twenty passages and twenty passages is what you are charged for. A marginal-contribution pass leaves the slice under-filled exactly when the corpus has nothing further to say, which is why handing the surplus back to conversation history is available as a decision at all.

The order of the two guards inside the loop is deliberate and easy to get backwards. The claim test runs first and the budget test second, so a passage that is redundant is rejected before its size is ever considered, and a passage that is genuinely new but too large for the remaining room is skipped with continue rather than ending the loop. That second choice is what lets a smaller passage further down the list still be admitted after a larger one has been passed over, which is the difference between filling a budget greedily and filling it well.

The obvious objection is the claims field: where does a frozen set of atomic facts per passage come from? Three answers, in ascending order of cost. You can approximate claims with normalised sentence hashes, which catches literal reuse across localised copies and section splits and costs nothing at query time. You can extract claims offline with a cheap model at index time, once per document version, and store them alongside the embedding — for a 40,000-passage corpus on Claude Haiku 4.5 at roughly 500 input and 120 output tokens per passage that is about $0.044 per full re-index, which is not a number that needs a meeting. Or you can skip claim sets entirely and use a coverage proxy: admit a passage only if its maximum cosine similarity to any already-admitted passage — how closely their embedding vectors point in the same direction, which is the same score the retriever ranked on — is below a threshold, which is cheap and approximate and discussed below for what it misses.

What makes the middle option affordable is not its per-passage price but where the work sits. Claim extraction runs offline, once per document version, against the index rather than against the traffic: a document is re-processed when somebody edits it, not when somebody asks a question. That puts the bill on the size of the corpus and the rate at which it changes, both of which a team already knows, and takes it off the request path, where the same work would appear as latency on every question. A cost that scales with documents rather than with queries is one that shrinks in importance as the service grows, which is the opposite of how a query-time reranker behaves.

The threshold that matters is not the similarity cutoff but min_new_claims. Set it to 1 and you admit a passage that contributes a single incidental detail; set it to 3 and you will drop passages whose one contribution is the one the question needed. There is no universal value, and the next chapter is largely about how to find yours.

The tempting reading of that parameter is that it is a strictness dial, so turning it up gives a cleaner prompt. It does give a cleaner prompt, and the reason turning it up is dangerous is specific rather than general. new is the claims a passage has that the already-admitted set does not, so its size falls as the selection grows, and a passage arriving late whose whole contribution is one narrow exception has a new of exactly one. Rank 13 in the table above is that passage: against the claims already covered it offers a single item, the 30-day issuer window. Set min_new_claims to 3 and rank 13 fails the test, the pass returns four passages instead of five, and the arm looks like an improvement — four hundred fewer tokens, no visible loss — right up to the questions where that exception is the answer. The parameter is not a strictness dial but a bet about how many facts a useful passage carries, and passages carrying one fact are disproportionately the ones carrying the unusual fact.

Deduplication, reranking and recall are different problems#

Marginal contribution gets confused with deduplication, and the confusion is expensive because near-duplicate removal deletes the wrong things. The corpus above has four localised copies of the refunds policy that are 96% identical by embedding similarity. A dedup pass at any threshold that catches them also catches the pair of passages that differ only in a number — the EU copy says 14 days and the US copy says 7 — and it keeps whichever one happens to rank higher. For a question from a customer in Berlin, that is a wrong answer produced by a correctness measure. Claim-level selection keeps both, because the second one contributes a claim the first does not have. Text-level dedup cannot tell the difference between a paraphrase and a policy variant, because at the level it operates on, they look the same.

It is also not reranking. A reranker is a second, slower model that re-sorts the retriever's shortlist by reading the question and a passage together in one pass — a cross-encoder — where the retriever itself compared vectors computed separately for each side, which is what makes it a bi-encoder and what makes it fast enough to run over the whole index. The better ordering that buys you is genuinely valuable, but a reranker scores (query, passage) pairs — one passage at a time, with no knowledge of the set. Rerank the twenty passages above and you get the same twenty passages in a better order, with the same seven copies of the 3–5 day window. Reranking improves the input to selection. It is not a substitute for it.

And it does not fix recall. If the article containing the answer is not in the retrieved set, no selection policy will conjure it; marginal contribution can only choose well among what the retriever handed over. This matters for diagnosis, because a question the system gets wrong has at least two possible causes — the evidence was never retrieved, or it was retrieved and drowned — and they call for opposite responses. The audit distinguished them by checking, for each regressed question, whether the answer-bearing passage appeared anywhere in the 20. It did, in 63 of the 71. Those are selection failures. The other 8 are recall failures and belong to a different investigation.

Those two counts are worth reading as arithmetic rather than as a verdict. Sixty-three and eight add to seventy-one, so every regressed question falls into one diagnosis or the other and none is left in a residual bucket. That completeness is what makes the split usable. Had a third of the regressions belonged to neither, "redundancy explains this" would have been a claim about some of the evidence presented as a claim about all of it. The audit behind it divides just as plainly: forty questions with twenty retrieved passages printed for each is 800 passages, which is the number the support lead read, and is why "all 800 passages were on topic" is a statement about a countable sample rather than a figure of speech.

One more boundary: none of this says small prompts are better. A block earns its place by what it adds, and a large block that adds a great deal earns a great deal of space. The reason the k=20 arm lost is not that it was big. It is that thirteen of its twenty passages added nothing, and the seven that did were harder to find as a result.

Replacing relevance with marginal contribution is therefore a change to where the decision is made rather than to how well anything is scored. The retriever keeps doing what it is good at, which is finding material about the question. The selector decides admission, and it decides it against the set already admitted, which is knowledge the retriever structurally does not have and no amount of reranking will give it.

What the audit established is that redundancy explains this regression on this corpus at these two values of k. It did not establish where the turn happens. Somewhere between eight passages and twenty the curve stops rising and starts falling, and 8 and 20 are two points on a curve nobody has drawn — so nobody can say whether 8 was already past the peak, whether the peak is at 6 or at 12, or how flat the region around it is, and a flat peak and a sharp one call for different amounts of caution when the corpus changes.

Drawing it means designing a sweep that varies k while holding everything else honest, which is harder than it sounds, because at higher k the assembler starts evicting other components and the arm you think you are measuring is not the arm you are running. It also means being ready for the first explanation anyone will offer for a result like this one, which is that the model got lost in the middle of a long context and the fix is to reorder. That explanation is checkable with arithmetic, and it is where the next chapter starts.