How much of a parent is relevant decides small-to-big

A retrieval system has to decide what a stored unit of text is, and the decision is pulled in two directions at once. A unit small enough to be found reliably is often too small to answer from, and a unit large enough to answer from is often too blurred to be found. Small-to-big is the arrangement that stops making one choice serve both jobs: index small pieces, and when one of them is the best match for a query, send the enclosing piece to the language model instead. The small piece is conventionally called the child and the enclosing piece the parent, which on documentation is usually the section under a heading, or the page.

That arrangement is not uniformly good or uniformly bad, and the property that decides which it will be on a given body of documents can be stated in one sentence: given that one child chunk matched a query, how much of its parent is relevant to that query. Where a section is tightly scoped, most of it is, and the expansion restores the definitions and tables the fragment referred to without containing. Where a page covers nine unrelated subjects, a tenth of it is, and the expansion hands a generator nine subsections of material about other questions and asks it to find the answer inside. This chapter is about measuring that share rather than guessing it, because guessing it is what makes the same change look like a good idea on two corpora where it has opposite effects.

Matching on a fragment, answering from a section#

The mechanism is a split between two roles that a plain chunked index collapses into one. A retrievable unit wants to be small. A 200-token chunk about one thing produces an embedding — the list of numbers standing in for its meaning, which is what a search actually compares — that is about one thing, and an embedding averaged over 2,000 tokens covering nine topics is a blur that matches everything weakly. A generatable unit wants to be large enough to be self-contained, because a 200-token fragment that begins "This field is ignored unless the account is in sandbox mode" is useless without the paragraph naming the field.

Small-to-big satisfies both by storing the two units separately and linking them. The vector index holds child embeddings. A document store, keyed by ID, holds the parents and is never searched. Retrieval matches children, then dereferences the parent link and returns parent text.

from dataclasses import dataclass

@dataclass
class Child:
    id: str
    parent_id: str
    text: str          # ~200 tokens, embedded
    n_tokens: int

def index_corpus(sections, split_child, embed, vectors, docstore):
    for section in sections:                     # section = the "parent"
        docstore.put(section.id, section.text)   # never embedded
        for child in split_child(section.text, target_tokens=200):
            c = Child(id=child.id, parent_id=section.id,
                      text=child.text, n_tokens=child.n_tokens)
            vectors.add(c.id, embed(c.text), metadata={"parent_id": c.parent_id})

def retrieve_small_to_big(query, embed, vectors, docstore, k_children=12, max_parents=4):
    hits = vectors.search(embed(query), top_k=k_children)
    parents, seen = [], set()
    for h in hits:                               # hits are in score order
        pid = h.metadata["parent_id"]
        if pid in seen:                          # several children, one parent
            continue
        seen.add(pid)
        parents.append(docstore.get(pid))
        if len(parents) == max_parents:
            break
    return parents

The deduplication step in that loop is where most of the behaviour lives, and it is the part that gets skipped. Twelve child hits do not become twelve parents. On an API reference they collapse to three or four, because several children of the same endpoint page match a query about that endpoint. That collapse is a feature — it is the system noticing that the evidence is concentrated — and it also means k no longer denotes the amount of context you are sending.

Follow that consequence one step at a time, because it is the step that turns a retrieval parameter into a budgeting problem. Twelve children at 200 tokens each is a product of two fixed numbers, so it is 2,400 tokens on every query this pipeline will ever serve. Twelve children collapsing to four parents is a sum of four numbers none of which the pipeline chose: whatever those four sections happen to weigh. Four short sections come to something like 1,600 tokens and four long ones to 20,000, and both are the same configuration running normally.

That gap is why max_parents in the function above is the wrong kind of guard, and it is worth naming the shape of the error because it recurs throughout this topic. The cap is a count. The quantity that can hurt you is a token total. Between a result set of four small sections and one of four enormous ones, the cap reads exactly four in both cases: it is constant in precisely the property it was put there to bound. A limit that cannot move when the danger moves is not a limit, and the repair is to accumulate parents until a token budget is reached rather than until a counter is.

The pattern here is a pointer indirection, in the sense a database uses the word: a secondary index stores small keys that are cheap to search along with a pointer to a row, and you search the small thing in order to read the big one. That is an analogy rather than a claim about how any vector store is built, and it is worth saying where it stops. The property that makes indirection safe in a database is that the row is the unit somebody wanted, and its width is fixed by a schema that somebody designed. Small-to-big inherits the indirection and inherits none of the schema. Nothing constrains a parent's size or its internal coherence, so a corpus where the "row" is a six-year-old wiki page covering nine unrelated topics is a corpus where the pointer leads somewhere nobody asked to go, and there is no declaration anywhere that says so.

Measuring the property instead of guessing it#

The parent relevance ratio is measurable on any corpus that already has relevance judgments, and it takes an afternoon. For each judged query, find the child chunks marked relevant, expand them to their parents, and compute the fraction of the expanded parent's tokens that come from relevant children. Averaged across the judged set, that fraction predicts whether small-to-big will help before you deploy it.

Read the first four rows as gaps rather than as levels, and two of them turn out to be the same gap counted in different units. The children-per-parent row is not an independent measurement: children are cut to a 200-token target, so the number of them a parent holds is roughly the parent's token count divided by 200. A 640-token parent divided by 200 is a little over three, and the table reports four. A 1,950-token parent divided by 200 is a little under ten, and the table reports eleven. Both land one child above the quotient, which is what you expect when the last child of a section is a short remainder rather than a full 200 tokens. The two rows are one fact, and the agreement is worth checking precisely because it licenses the arithmetic in the next paragraph.

With that established, the ratio can be converted into a count of children, which is the form in which it means something. A ratio of 0.62 across four children is about two and a half children of relevant material per expansion, so when a child matched, most of what came back with it was about the same question — the surrounding parameter table, the error codes, the definitions the fragment referenced without containing. A ratio of 0.11 across eleven children is a little over one child of relevant material out of eleven, which is the same statement as "a tenth of the page" arrived at from a different direction: the ratio is measured in tokens and the child count in units, and they agree because children are near-uniform in size. Nine tenths of every expanded wiki parent was about something else — the VPN section of a page whose expenses section matched, the 2021 deprecation notice at the bottom of a page whose top described the current behaviour.

The consequence for the generator is the row above the accuracy row. It receives 9,800 tokens, has to locate the relevant tenth, and does so slightly worse than it did when handed 2,400 tokens of pre-located material. The cost went up by a factor of four for the privilege.

There is a second measurement worth taking on the same pass, because it distinguishes two different reasons a corpus might do badly. Count, on the judged set, how often the relevant evidence spans a child boundary — how often the answer requires two adjacent children rather than one. That number is the size of the problem small-to-big is trying to solve. On the API reference it was 34% of queries, which is why there was that much room to gain. On a corpus where it is 6%, expansion has almost nothing to fix and every token it adds is pure dilution, whatever the parent relevance ratio says.

The two measurements are worth keeping distinct, because a reader who has only the ratio will over-predict. A high ratio says an expansion will not hurt. It does not say there is anything for the expansion to do, and the second number is the one that says that. Both being favourable is what the API reference result actually rested on.

Shrinking the parent without abandoning the idea#

A ratio of 0.11 does not mean the technique is unusable on that corpus, only that the unit it was pointed at is wrong. Three adjustments cover most of the recoverable cases, and they are worth trying before the whole approach is written off.

The first is a token cap with a centred crop: keep expanding to the parent, but if the parent exceeds a budget — 800 tokens is a reasonable starting value — return only the window of the parent centred on the matched child. That preserves the property that made expansion work on well-behaved documents and bounds the damage on the badly-behaved ones, at the cost of occasionally cropping away the definition that lived at the top of the page. On the wiki above, a cap of 800 tokens moved accuracy from 62% back to 67%, roughly recovering the baseline rather than beating it, which is the honest result for a corpus this technique does not suit.

The second is to expand to siblings rather than to the parent: return the matched child plus the child before and after it. That is a structurally anchored expansion with a bounded size, and it behaves like the parent when the parent is small and like a fixed-width window when the parent is enormous. It is also the point where small-to-big begins converging on a different technique entirely, which the next chapter takes up.

The third is to make the parents better by re-authoring the documents, which sounds like a non-answer and is frequently the highest-value change available. A wiki page that covers nine topics is a documentation problem before it is a retrieval problem, and splitting it into nine pages improves search, navigation and human reading at once. Retrieval engineers are reluctant to file that ticket because it belongs to another team. It is still the fix with the best ratio of effort to result on internal corpora, and the parent relevance ratio is the number that makes the case concretely rather than as an aesthetic complaint.

One implementation detail decides whether any of this is measurable: expansion has to happen after scoring. If the reranker sees expanded parents, it is scoring 2,000-token blobs whose relevance is diluted by the same nine subsections that dilute the prompt, and cross-encoders score long mixed passages unreliably. You lose precision at the ranking stage as well as at generation, and the two losses become impossible to separate afterwards. Retrieve children, rerank children, then dereference the survivors.

Where it is the wrong tool#

Small-to-big does not fix bad chunk boundaries; it routes around them. If the child splitter is cutting mid-table so that a row's meaning is destroyed, expanding to the parent will usually restore the table and the answer will improve. The retrieval that got you there still matched against a mangled child embedding, so queries whose only good evidence lives in the mangled part still miss. The boundary decisions from topic 9.5, on where a chunker should cut, remain load-bearing on the matching side, and expansion only repairs the generation side.

It also does not help when the parents are the wrong shape and cannot be changed. The tempting response to a 0.11 ratio is to keep the technique and shrink the parents — expand to the subsection rather than the page — which is reasonable when the documents have real subsections and pointless when they do not. Six-year-old wiki pages frequently have one ## heading and 1,800 tokens under it, so the structural unit available to you is the page, and the technique has no smaller parent to offer. In that situation the honest options are to re-author the documents, to fall back to flat chunks with generous overlap, or to expand by something other than structure.

A third case where it misfires is a corpus of short independent units. Forty thousand support tickets averaging 120 tokens have no parent worth expanding to, because the ticket is already the whole thought: it has a symptom, a diagnosis and a resolution. The container above a ticket is the queue it was filed in, which groups by routing rules rather than by meaning, and wrapping ten tickets in a synthetic parent produces a container whose only property is that its members were filed on the same day. Every expansion strategy assumes that proximity in a document implies relatedness, and that assumption is what a corpus of independent short records does not satisfy.

What the API reference result actually depended on was that its parents were uniform. Every endpoint page had roughly the same size and the same internal coherence, so expansion behaved the same way on every query and a median figure described every query. The wiki's parents ranged from 200 tokens to 6,400 and from one topic to nine, so the same rule produced a different amount of context every time it fired, and no single figure described any particular query. That variance is the real hazard rather than the average, and stating it that way points at an obvious alternative: expand by a fixed amount of surrounding text rather than by whatever structural unit happens to enclose the hit, so that the expansion is the same size everywhere regardless of how the document was written.