A starting configuration needs a rule that can disprove it

A starting configuration is the set of chunking, embedding and retrieval settings a new corpus is indexed with on day one, before any results from that corpus exist. It is a prediction: these boundaries, this token ceiling, this retrieval depth will suit documents of this shape. Predictions of that kind are unavoidable, because an ingest job will not run without settings in a file. The second half of the sentence is the part that goes missing. A retirement rule — sometimes called a falsification or kill criterion — is a condition written down at the same time, naming which measurement, on which data, at which moment, would establish that the prediction has stopped holding.

The two halves are not equally likely to get written, and the asymmetry is structural rather than a lapse of discipline. A configuration is forced; a retirement rule is optional, and an optional artefact attached to a system that currently works is one nobody has an occasion to produce. So the settings ship and the condition does not, and six months later a measured number and a guessed number sit in the same file looking identical, because the evidence for the measured one lived in a conversation and the file records only the digit.

A configuration is a claim about a corpus#

A configuration encodes propositions about documents: that sections are short enough to fit under a ceiling, that headings are real markup rather than inferred from font size, that answers live in spans of roughly a certain size. Those are properties of a corpus, not properties of the software. Because they are properties of a corpus, a change in the corpus can make them false without anybody touching a line of code — a new source format arrives, a parser is upgraded, an embedding model is swapped. Because nothing in an ingest pipeline measures corpus properties on a schedule, that falsification emits no signal; the job still runs, the index still builds, the latency graph is unchanged. And because there is no signal, nobody re-examines the setting. That is the whole mechanism: the settings are claims about something that moves, and the system watches only the parts that do not move.

So the useful question is not "are these numbers good" but "what would show them to be bad, and who would see it". A rule can disprove a configuration only when three things hold of it at once. There must be a measurement capable of coming back against the configuration rather than merely describing it. There must be a threshold fixed before that measurement is taken. And there must be a decision that genuinely changes depending on which side of the threshold the number lands.

The middle condition carries more weight than it looks like it should. A threshold chosen after the numbers are in is chosen to be passed, because the numbers are visible while the threshold is being written and a threshold that condemns the current system is uncomfortable to propose. Nothing about the arithmetic differs between the two orders — same metric, same data, same comparison — and what the result is evidence of differs completely. Fixing the threshold first is what converts a measurement into a test.

It also matters that the settings in a file are not all the same kind of thing. Some are requirements: forced by something outside the author's control, so getting one wrong is an error. Others are decisions: several values would work, one was chosen, and a reason should exist. Requirements move when the thing that forced them moves; decisions move when evidence arrives. Confusing the two is how a corpus-shaped choice gets defended as though it were a law, and how a genuine constraint gets overwritten by somebody who thought it was a preference.

The configuration for a corpus of API reference pages#

Take a new corpus: 6,000 API reference pages, generated from the documentation repository as markdown, each page documenting one endpoint with sections for the signature, parameters, request examples, response schema, error codes and migration notes. The structure signal is ## and ### in files a human wrote. Those are ATX headings, which is what markdown's own specification calls the hash-prefixed form, and what the markdown_atx provenance value in the configuration below refers to. Chapter 5, where heading reliability was scored per source format as a detection problem, put that signal at precision 1.00 and recall 1.00 — the strongest structure signal available anywhere.

splitter:
  strategy: heading_aware        # verified markdown ATX headings; ch.5 routing
  levels: ["##", "###"]
  token_ceiling: 420             # 512 encoder limit - 61 longest breadcrumb - 31 margin
  on_overrun:
    strategy: recursive          # subdivide, never clip; ch.3
    separators: ["\n\n", "\n", ". "]
    size: 420
    overlap: 60                  # only within a subdivided section
    inherit_breadcrumb: true
  merge_floor: 90                # merge consecutive short sections up to this
embedding:
  model: bge-base-en-v1.5
  measured_input_limit: 512      # probed, not read off the model card; ch.3
  prefix: "From {doc_title}, section {breadcrumb}:"   # embed_text only; ch.8
index:
  metric: cosine
  hnsw: {m: 32, ef_construction: 200}
  retrieve_k: 20
  rerank_to: 8
  per_document_cap: 3            # prefix raises intra-document similarity; ch.8
control_arm:
  strategy: fixed
  size: 420
  overlap: 60
  maintained: true               # rebuilt on every ingest run

Every line of that has a reason from an earlier chapter, and the one that people leave out is the last block. The control arm is a second index over the same corpus, chunked with plain fixed-size windows, rebuilt whenever the main index is rebuilt. It costs about 1.4 GB of vectors and adds roughly four minutes to a 25-minute ingest run for a corpus this size, and it exists so that the question "is the clever configuration still beating the dumb one" has an answer available at any moment rather than requiring a project.

Without a maintained control, the retirement rule below is unenforceable in practice, because firing it would require someone to volunteer a week of work to build the comparison — and the moment when a configuration has quietly stopped fitting is precisely the moment when nobody suspects anything and nobody volunteers.

Where each number in the file comes from#

The token ceiling is the clearest case of a number that is derived rather than picked, and its comment is the derivation rather than a note about it. Three quantities go in. The encoder accepts 512 tokens, established by probing the model rather than reading its card (chapter 3, where a model that silently discards everything past its input limit was shown to index only a chunk's opening while still serving the whole text). What reaches the encoder is not the chunk alone: the prefix line pastes From {doc_title}, section {breadcrumb}: in front of the chunk before embedding (chapter 8, where prefixing a chunk with its document title and section heading recovered most of what a late-chunking migration promised). Rendered through that template, the longest breadcrumb this corpus produces comes to 61 tokens. Subtracting that prefix budget from the encoder's limit leaves a little over four hundred and fifty tokens of room, and a 31-token margin is then held back, which lands the ceiling on 420.

The ceiling, the breadcrumb budget and the margin sum back to the encoder limit exactly, with nothing left over, and the absence of a residual is what makes the line auditable. Anyone can recompute it: subtract the two smaller figures from 512 and the answer either is the ceiling in the file or it is not. Written as a constant, token_ceiling: 420 would be a number whose story is kept somewhere else; written as a subtraction it carries its own audit, and a reader who changes the prefix template can see which term has to move with it.

The exact closure licenses one further thing. Because no slack is hidden inside the identity, the 31-token margin is the only free number in that line. The 512 is a requirement, forced by the encoder. The 61 is a requirement too, of a different sort: it is measured off this corpus under this prefix template, so it is forced by the data rather than by the software. The margin is the decision — the one place where somebody chose how much room to leave against a breadcrumb slightly longer than the longest one sampled. Anyone arguing about the ceiling is arguing about that margin.

Two further choices in the file are worth defending explicitly, because each has a plausible-looking alternative. Overlap appears only inside subdivided sections because a section boundary is a real boundary, and bleeding 60 tokens of the parameters section into the response-schema section adds duplicated text for no gain; overlap is insurance against arbitrary cuts, and heading cuts are not arbitrary. And there is no semantic chunking anywhere in the file, despite the corpus being a reasonable size for it, because the condition set out in chapter 6 — where cutting at drops in consecutive sentence similarity was shown to hinge entirely on a corpus-specific threshold — is not met: these pages have real paragraphs and real headings, so the segmentation signal that strategy reconstructs is already present for free.

The merge_floor deserves a note too, since API reference pages are exactly the shape that needs it. An endpoint page often has a ### Errors section of one line and a ### Rate limits section of two, and thirty-token chunks of that kind retrieve on their headings and answer nothing. Merging consecutive short sections up to 90 tokens under their common parent's breadcrumb turns six fragments into one usable chunk, and 90 is a starting number chosen from the corpus's section-length histogram rather than from anywhere else.

How many fragments actually collapse into one chunk is not fixed by the floor, and it is worth seeing why. The merge accumulates consecutive sections while the running total sits under the floor, so the count depends on how short the sections are: shorter fragments mean more of them per merged chunk, and the same floor produces a different grouping on a corpus whose short sections run longer. The floor sets a size, and the size sets the count.

What the control arm holds fixed#

The control arm block names three settings and nothing else, and the omissions are the point. It says strategy: fixed, size: 420 and overlap: 60. Its size and overlap are the same values the main arm uses when it subdivides an overrunning section, and every other line of the file — the embedding model, the prefix, the metric, the HNSW parameters, the retrieval depth, the rerank depth, the per-document cap — applies to both indexes because it sits outside the block. Two indexes built that way differ in where their boundaries fall and in nothing else, which is the pinning discipline of chapter 2, where a nine-point win was shown to be uninterpretable once three variables had moved in the same run.

There is a residual in that pinning, and hiding it would be dishonest. The two arms do not share a chunk-length distribution: every control chunk is 420 tokens, while the main arm's chunks are as long as their sections happen to be. That difference cannot be pinned away, because chunk length is downstream of boundary placement — you cannot hold length fixed and vary where the cuts go. The comparison therefore measures boundary placement together with the length distribution it produces, and that compound is the right thing to measure, since it is what a corpus would actually be served by.

The condition that retires it#

The rule has to name a metric, a threshold, a data set, a cadence and an owner. Anything missing one of those is a sentiment.

# runs after every full re-index, and monthly regardless
def review_chunking_config(main, control, heldout) -> Verdict:
    a = span_containment_at_k(main, heldout, k=8)
    b = span_containment_at_k(control, heldout, k=8)
    lo, hi = paired_bootstrap_diff(main, control, heldout, n=10_000)  # 95% CI

    if hi < 0:                       # control beats main, interval excludes zero
        return Verdict.RETIRE(f"control ahead by {b - a:.3f}, CI [{lo:.3f},{hi:.3f}]")
    if a < 0.55:                      # absolute floor, independent of the control
        return Verdict.INVESTIGATE(f"span containment {a:.3f} below floor")
    if cited_span_rate(main, heldout) < 0.80:
        return Verdict.INVESTIGATE("cited chunks often lack the answer span")
    return Verdict.KEEP

The depth in the first two lines is not a free choice. Both arms are scored at k=8, and 8 is what the configuration sets as rerank_to — the number of chunks that survive reranking and reach the generator. Scoring at any other depth would measure a shortlist nobody is served. That is why the last row of the trigger table below lists a change in the generator's context window as something that invalidates rerank_to: 8: move that setting and the metric in the rule has to move with it, or the rule quietly starts measuring a system that does not exist.

The comparison's sign convention repays a moment, because getting it backwards inverts the verdict. paired_bootstrap_diff(main, control, ...) returns an interval on the difference in the order its arguments are given, main minus control. So hi < 0 says the upper end of that interval is still below zero, which means every plausible value of the difference has main behind control: the control is ahead, with the interval excluding zero. The message the branch prints computes b - a, control minus main, the same gap written as a positive number. Testing the upper end rather than the lower is what points the test the right way — a main arm merely possibly behind its control does not fire; one behind across the whole interval does.

Two triggers, stated in advance. The first is relative: if heading-aware splitting stops beating plain fixed-size windows on a held-out query set, with a paired bootstrap interval that excludes zero, the configuration has lost its justification and goes back to the simplest thing that works. The second is absolute and is the one that catches a slow slide where both arms degrade together: if the chunks the generator cited stop containing the answer span at least 80 percent of the time, something upstream has changed — a new document class, a parser regression, a shift in what people are asking — and the configuration is no longer describing the corpus it runs on.

Each trigger is blind exactly where the other one sees, and that is why both are there rather than one. A relative test compares two arms, so anything that damages both arms equally leaves the difference untouched and the test silent. An absolute test watches one arm against a fixed line, so it says nothing when the main arm holds its level while the control quietly overtakes it. Between them they cover both ways the configuration can stop being justified: by getting worse, and by ceasing to be better.

The absolute floor is a number about this corpus and does not travel. Chapter 2's three-way comparison on a support corpus scored its heading-aware arm on this same metric and came in a little under this floor, which would make the same rule, applied there, fire on its first run. That corpus's heading-aware sections were long, and their answer sentences frequently sat past the encoder's input limit — a property of those documents rather than a verdict on the metric or on the strategy. A floor set from one corpus's section-length histogram is a claim about that corpus.

The two figures were measured at different retrieval depths, and the mismatch does not rescue the comparison. Chunks retrieved at a depth of eight are a subset of those retrieved at a depth of ten, and containment asks whether any retrieved chunk holds the answer span, so adding chunks can only turn a miss into a hit. Containment at the shallower depth is therefore never higher than at the deeper one. The support corpus figure was taken at the deeper depth and already sits under the floor, so the shallower measurement this rule would take must sit under it too.

The held-out set has to be large enough that the rule can fire. This is where most well-intentioned versions of this fail: a 40-query held-out set measuring a proportion near 0.65 has a paired-bootstrap interval wide enough that a realistic ten-point regression will not exclude zero, so the rule is written, the check runs green forever, and everyone believes the configuration is being monitored. Three hundred queries with marked answer spans is the working minimum for a rule with a two-arm comparison in it; below that, drop the interval requirement and use a plain threshold, which is noisier but at least capable of firing.

The reason those two query counts sit so far apart is the square root in the standard error of a proportion. Uncertainty on a measured rate shrinks with the square root of the number of queries, not with the number itself, so cutting a held-out set from three hundred queries to 40 does not widen the interval by that ratio — it widens it by the ratio's square root, which is somewhere between two and three. Widen an interval that was just able to resolve a ten-point difference by that much and a ten-point difference now sits comfortably inside it, which is to say the interval no longer excludes zero. The rule is intact, the arithmetic is correct, and the check cannot report the regression it was written to catch.

That is the stumble this chapter is really about, and it generalises past chunking. A defence that stays constant while the thing it must discriminate varies is not a defence. A frozen held-out set is exactly that: the corpus moves, the query mix moves, the model is replaced, and the check runs unchanged against a fixed set of questions, returning the same reassuring verdict throughout. Believing it is tempting because a green check is evidence of something, and reads as evidence of health. It is wrong because a check that cannot go red under any change to the system it monitors is reporting on itself. A set too small to resolve the difference and a set too old to contain the failure emit the identical green, and no inspection of the output separates either from a healthy one.

Events that break an assumption without a measurement#

Alongside the metric triggers there are structural ones that do not need a measurement to justify acting on, because they invalidate an assumption the configuration was built on.

The table is more informative read as gaps than as a list. What separates the rows is how much of the configuration each event invalidates, and the remedy column tracks that width precisely. A new source format touches one decision — whether this document class routes to heading-aware splitting at all — so the remedy is one measurement on sampled pages. A change of median section length touches two numbers, the ceiling and the merge floor, and the remedy names exactly those two. A parser upgrade is the only row whose remedy is total, because a re-parse changes the text that everything else was derived from: the breadcrumbs, the section lengths, the token counts, the chunk boundaries. There is no smaller check available for it, and that is why it sits alone.

The second row is the one that fires most often and gets handled least well, because an embedding-model upgrade arrives framed as an improvement and gets shipped on the strength of its benchmark numbers. A model with an 8,192-token window replacing one with 512 does not just improve scores; it makes the 420-token ceiling arbitrary and possibly harmful, and the configuration that was correct on Monday is a configuration nobody has evaluated on Tuesday.

Following that through shows why "possibly harmful" rather than merely redundant. With a wider window the ceiling stops protecting anything, since no realistic section would overrun the encoder any more. What it still does is cut sections that would now embed whole, and each of those cuts costs the thing heading-aware splitting exists to buy — a chunk whose boundaries are the author's. The setting has not become inert. It has changed from a guard into a constraint, and nothing in the file records the difference.

What a fake rule looks like#

The difference between a rule and a preference is whether an outcome exists that would change the configuration, and whether that outcome would reach someone with the authority to act on it.

That table is a ladder, and the rungs are the five components named above it. The first row carries none of them. The second supplies a metric and nothing else. The third supplies a cadence and nothing else, which is why its right-hand cell concedes that the trigger is real while the rule is not. The fourth supplies three — metric, threshold and data set — and is still missing a cadence and an owner. The last row is the only one carrying all five, and the number of components a row is missing is exactly the number of clauses its right-hand cell needs to explain the failure. Nothing in the table falls outside that accounting, which is what licenses treating the five as a checklist to be run against a proposed rule rather than as a description of one.

The fourth row is worth dwelling on because it is the version that looks rigorous. It has a metric, a threshold and a data set, and it will produce alerts. What it cannot do is tell you that the segmentation is wrong, because recall@10 falling is consistent with a model change, a corpus shift, a query-mix change or a reranker regression, and without the control arm there is nothing to attribute it against. A rule that fires and cannot be acted on trains people to silence it.

The first and fourth rows therefore fail in opposite directions and land in the same place: one never fires and so never reaches anyone, the other fires often and hands someone a number with four possible causes and no way to separate them. Both leave the configuration unfalsified.

Where a retirement rule stops helping#

A retirement condition tells you when a configuration has stopped being justified. It does not tell you what to replace it with, and it will not search the space for you: the verdict is RETIRE, and what follows is the same pinned three-way comparison from chapter 2 — the same corpus, the same queries, the same model and depth, with only the boundaries differing — run again with the corpus as it now stands. Expecting the rule to produce an answer rather than a prompt is how it ends up ignored the first time it fires.

It also cannot protect a configuration from being wrong in a way the held-out set does not represent. A query set assembled from last year's logs measures last year's questions, and a corpus that has acquired a new document class will be failing on questions nobody has asked yet in a form the evaluation can see. Refreshing the held-out set is its own cadence — sample new queries from production logs quarterly, mark their answer spans, retire queries whose source documents have been deleted — and a stale evaluation set is a slower and quieter version of the same failure this chapter is about.

That cadence has a cost worth naming in advance. Marking answer spans is annotation work, so a held-out set is a recurring expense rather than a fixture, and the size that makes the rule able to fire is the same size that makes refreshing it expensive. A team that will not fund the refresh has a retirement rule with a shelf life, and it is better to know that when the rule is written than to learn it from a check that has been green ever since.

And none of it substitutes for reading. The decomposition exercise from chapter 8 — taking twenty failing queries and finding by hand where each answer actually lives and what was retrieved instead — is what tells you whether a RETIRE verdict means the boundaries are wrong or means something upstream broke and dragged the chunking arm down with it. The metrics tell you when to look; they do not do the looking.

What you have now is a configuration, a control to measure it against, and a stated condition for abandoning it — which is enough to start indexing. Each number in that file is either forced by something outside it or chosen for a reason that can be written down, and the retirement rule names the measurement, the threshold, the data set, the cadence and the owner that between them make the whole arrangement capable of losing. A configuration that cannot lose is not being trusted; it is merely being left alone.

The first real corpus will produce a length distribution that surprises you, a document class the routing did not anticipate, and a query pattern the held-out set has three examples of, and all three of those are found by running it rather than by planning it. Which is the argument for building the control arm on day one instead of on the day it is wanted: the day it is wanted is the day nobody yet knows it is.