A reportable result carries the values, the baseline and the interval

A sweep report is the document that goes to the people who did not run the sweep, and it is the step at which most of this work either lands or quietly stops. This one arrives in the review channel and it is two sentences long: "After sweeping the retrieval pipeline we improved quality by 6%. Recommend we adopt the new settings." Three people react to it approvingly. Nobody can act on it.

Four words in that sentence carry technical weight and none of them are pinned down. A sweep is a search over configurations of a retrieval system, each configuration being one setting of every knob — chunk size, overlap, k, rerank depth, whether the query is rewritten — scored end to end. A metric turns each configuration's answers into one number; nDCG@10 is the one used throughout this topic, and it rewards a configuration for placing the passages an annotator marked as relevant near the top of its first ten results. A delta is the difference between two configurations' scores. A baseline is the configuration the delta was taken against. The report names none of the four, and each omission costs the reader something different.

Six percent of what#

Take the first number, since the arithmetic settles it and no judgement is required. Six percent of what — nDCG@10 moving from 0.512 to 0.543 is a 6% relative gain and 3.1 absolute points, while 0.512 to 0.572 is 6.0 absolute points and an 11.7% relative gain. Both readings check out from the same starting score, which is what makes the sentence useless rather than merely vague: subtract 0.512 from 0.543 and the difference is 3.1 points, and dividing that difference by the starting score gives 6% of it; subtract 0.512 from 0.572 and the difference is 6.0 points, which divided by the same starting score is 11.7%. One phrase, two arithmetically valid readings, differing by a factor of about two in how much quality moved.

Those are different results with different implications for whether the change is worth its serving cost, and the sentence is compatible with both. A relative percentage and an absolute point difference are a ratio and a difference; they coincide only when the baseline sits at one, and a metric bounded between zero and one guarantees that never happens. The fix is to publish the two scores, from which either form can be recomputed and neither can be mistaken for the other.

Then take "the new settings", which does not name a single parameter value, so the reader cannot apply them, cannot tell whether they differ from the settings adopted in February, and cannot check whether the winning chunk size forces the reindex from chapter 2 — where re-embedding a whole document collection turned out to be the cost separating the cheap axes from the expensive ones. And "improved" is a comparison against an unnamed baseline: the configuration currently in production, the sweep's own grid origin, and the February champion are three different reference points that were all live in this codebase this year. A delta without a named baseline is not a small omission in a comparison; it is not a comparison.

Three readers, three sets of missing fields#

The reason a sweep report needs so many fields is that it has three distinct audiences, and each of them fails differently when a field is absent.

Those three requirement columns barely overlap. The adopter needs values and operational consequences and never needs a run id; the reproducer needs identifiers and pinned versions and does not care about latency; the reviewer needs the statistical apparatus neither of the others will read. A report written for any one of them fails the other two silently, which is why the field list is longer than any single reader would ask for.

A report that satisfies all three is not long. It is one page with the numbers in it, and it is worth writing as a structured artifact rather than prose so that the next sweep's report is diffable against this one.

sweep_id: sweep-2026Q3
metric: ndcg@10           # pooled labels, gain=graded, 4-level relevance
judge: sonnet-4-5-20250929, prompt sha 88fd…, 3-sample majority

baseline:
  run_id: prod-champion-2026-02-12
  config: {chunk: 512, overlap: 64, k: 8, rerank_depth: 50, rewrite: true}
  score: 0.5120

winner:
  run_id: sweep-2026Q3/trial-0212
  config: {chunk: 768, overlap: 128, k: 20, rerank_depth: 100, rewrite: true}
  score: 0.5271
  delta: +1.51 points
  ci95: [-0.42, +3.44]      # paired bootstrap, 10k resamples, over queries
  queries_better: 96
  queries_worse: 71
  queries_tied: 333

measurement:
  selection_split: golden-v2-500@sha256:3ab1…   (selections_against: 1)
  confirmation_split: prod-2026Q3-240@sha256:71ce… (sealed, unsealed once)
  confirmation_delta: +0.9 points, ci95 [-1.8, +3.6]
  noise_floor_sd: 0.0028 over 5 identical reruns
  minimum_detectable_difference: 1.5 points at 500 queries
  null_arm_delta: +0.1 points, ci95 [-1.0, +1.2]

serving_impact:
  p50_latency: 1340 ms  (baseline 640 ms)
  cost_per_1k_queries: $49.70  (baseline $16.80)
  reindex_required: true  (5.0 h, $9.07)

not_tested: [embedding model, hybrid BM25 weighting, chunk boundary strategy,
             follow-up and out-of-scope query strata]

Everything in that file has been earned by an earlier chapter, and the fields that look like bureaucracy are the ones that stop specific failures: selections_against is how a reader knows whether the split is still a measurement instrument, null_arm_delta is the A/A check — chapter 4's null comparison, in which the same configuration is entered twice under different run ids and had better come out indistinguishable from itself — confirming the harness was not broken, and not_tested is what stops the next team from spending a quarter rediscovering that nobody has looked at hybrid retrieval.

Two of those fields close against each other. The winner's score minus the baseline's score is the delta printed below it, so the file states its own arithmetic and a transcription slip in any of the three would show immediately. That is a small thing on one report and a large one across a directory of them: a reader never has to reconstruct a delta by hand, and a script can verify every archived report at once.

The reindex_required line is the more interesting closure, because its two figures were not produced by this sweep at all. The winning configuration sets chunk 768, with overlap 128, and chapter 2 costed exactly that pair from the other end — counting the collection's tokens, inflating the count for overlap, and dividing by an embedding throughput — arriving at 5.0 hours and $9.07 for one build. This report reads the same two figures off a trial id. The two routes share no step, and landing on the same pair of numbers is what licenses a reviewer to price the adoption without rerunning anything. It is the difference between a report that restates its own output and one that joins a measurement to an independently computed cost.

The three per-query counts deserve a note, because they are the field most often dropped for being unglamorous. They also account for the query set exactly: 96 improved plus 71 degraded plus 333 unchanged is the 500 queries in the split, with nothing left over. That closure matters more than it looks. It says no query was dropped for erroring, none was counted twice, and there is no fourth category quietly absorbing the cases that did not fit — so the delta is a mean over precisely those queries, and the set a human could go and inspect is the one hundred and sixty-seven that moved, all of them, rather than an unbounded pile.

A delta of +1.51 built from 96 improved and 71 degraded queries means the change is close to a lateral move that shuffles which questions work, and somebody should look at the 71 before adopting it — regressions are not distributed randomly, and if 40 of them are the API reference queries from chapter 6 — where a query-level split left a chunk size fitted to one document layout unchecked — then the winner is trading away a stratum that matters. A delta of the same size built from 34 improved and 3 degraded is a different proposition entirely, safe to ship on much weaker evidence, because the downside is bounded by inspection rather than by statistics. The mean is identical in the two cases and the decision is not, which is the whole argument for publishing the counts.

Flat surface or blind instrument#

The report above describes an uncomfortable outcome, and it is the common one. The best of 360 configurations beats the current champion by 1.51 points, the 95% interval runs from −0.42 to +3.44, and adopting it means a reindex, 700 ms of extra latency and triple the serving cost. The reflexive readings are both wrong: "1.5 points is a win, ship it" ignores the interval, and "the interval spans zero so the sweep found nothing and the money was wasted" treats a null result as a failed experiment.

Start with whether the study could have seen the effect. Chapter 4's power table says a 500-query set resolves 1.5 points and no better, so this measurement sits on the boundary of what the instrument can distinguish. The quantity doing the work is the minimum detectable difference, written MDE in the table below, meaning the smallest true gap this evaluation could resolve at all given its size and its noise floor.

That boundary deserves more care than it usually gets, and it is genuinely hard rather than merely fiddly. An observed delta that lands exactly on the MDE is not the same as an observed delta of zero, and it is not the same as a comfortable win either. It is the value at which the two diagnoses stop being separable: a grid over which quality really is constant, and a study too small to see a real gap of that size, produce readings that look alike. Concluding "nothing was found" from a boundary reading is as much an over-claim as concluding "we found something", and the honest move is to say which of the two situations you are in is undetermined, then go and change the instrument if the answer matters.

The distinction between the first row and the last is the one that gets skipped, and skipping it is how "the response surface is flat" — the response surface being the metric read as a function of the parameter values, across the whole box of combinations you searched — becomes a comfortable way of saying "we could not measure anything". They are separable, and the operation that separates them is worth deriving rather than quoting, because a reader who only remembers the rule will apply it to the wrong pair of numbers.

Start with two configurations rather than all 360 of them, because the whole argument is visible at that size. Score two settings that are in truth exactly as good as each other, on the same 500 queries. The two numbers that come back will not be equal, because each carries the accident of which queries landed in the set and how the judge happened to grade the answers on them. The typical size of that gap is not free to be anything: it is the paired standard error, the quantity obtained by differencing the two configurations query by query and resampling the differences. So two scores separated by about one standard error are what a pair of identical configurations looks like, and two separated by several are not. The rest of this is that observation scaled up to a grid.

The comparison is between two spreads. Write s for the standard deviation actually observed across the 360 measured scores, t for the standard deviation of the true qualities of those same 360 configurations, and e for the spread that measurement noise alone contributes to a score. A measured score is a true quality plus a noise term, and the noise in a score has nothing to do with how good the configuration really is, which is the condition that lets the two sources combine as variances rather than as standard deviations. So s² = t² + e². Variances add and standard deviations do not, because variance is the quantity defined as an average of squared deviations and averages of independent contributions add; adding the standard deviations instead would count every configuration's noise as though it always pushed in the same direction as its true quality.

Rearranged, that identity is a measurement of the thing nobody can observe: t² = s² − e². The true spread across configurations is whatever is left of the observed spread once the noise spread has been taken out of it. If the parameters in the box genuinely move quality, t is positive, so s must come out strictly larger than e. If they do not, s and e are estimates of the same quantity and should land on top of each other. That is why the first row of the table compares a standard deviation across configurations against a standard error, and why the comparison is a subtraction of squares rather than an eyeball of the range.

Which standard error goes into e is the step most likely to be got wrong, and getting it wrong flips the answer. All 360 configurations were scored on the same 500 queries. The component of a score that comes from those particular queries being easy or hard is therefore identical for every configuration, so it moves all 360 scores up or down together and contributes nothing whatever to the spread across them. The paired standard error — the one from differencing two configurations query by query before averaging — is the quantity with that shared component already removed, which makes it the only one that is measuring the same thing s measures. Compare s against the standard error of a single configuration's absolute score instead, which still contains the query-difficulty term and is several times larger, and every grid ever run comes out flat.

The report carries one more instrument pointed at the same scale, and it was built for a different purpose. null_arm_delta is the same configuration entered twice under two run ids: two arms that differ in nothing, whose measured delta is therefore pure noise by construction. It comes back at +0.1 points with an interval comfortably containing zero. That is a direct, empirical reading of what this harness does when there is provably nothing to find, obtained without any resampling and without any assumption about the shape of a distribution, and it agrees with what the bootstrap says. Two routes to the same scale, sharing no step, are what turn a flat readout from an absence of evidence into a measurement.

One limit of the report belongs here rather than in the reader's head. It states its minimum detectable difference as a single number for the whole sweep, and the quantity depends on two things: the size of the query set, which is fixed across this grid, and the per-query spread of the difference between the particular pair of configurations being compared, which is not fixed at all. Two settings differing in one knob move a handful of queries and produce a tight distribution of differences; the winner and the baseline here differ in four knobs and move a third of the set between them. No single field can be right for every pair in a 360-configuration grid, and a reviewer who reads it as a universal threshold will over-trust it on exactly the widest comparisons the report contains.

Once the first row is established, the rule for choosing within the grid follows from it, and the obvious alternative is wrong. Do not pick the cheapest configuration scoring within some tolerance of the winner; pick the cheapest one whose interval overlaps the winner's. Where the true qualities are indistinguishable, "close to the top score" is ranking configurations by their noise, and the top score is itself the largest of 360 noisy draws and the most contaminated number in the grid. Interval overlap asks the answerable question instead — whether the data exclude this configuration from being as good as the winner — and that is the only one of the two an evaluation can settle.

Searched ranges, and the sentence the evidence supports#

An interval and a conclusion are different kinds of statement, and the gap between them is where a careful sweep still turns into an over-claim. An interval describes what this measurement, on this data, could and could not resolve. A conclusion describes the world. Nothing in the arithmetic promotes the first into the second, and the promotion happens in the writing rather than in the analysis, which is why it survives review by people who checked every number.

The belief to name here is the same species as chapter 6's belief that holding out queries protects a chunking result: it is the belief that publishing an interval makes a result honest. That is tempting because an interval is exactly the right instrument for the question it answers, and because a report carrying one looks, at a glance, like a report that did the work. It is wrong because it answers only one of the reviewer's two questions. An interval says how far this estimate would move if the query sample were redrawn. It says nothing about whether the study was capable of resolving the gap anybody cared about. A sweep too small to see anything publishes a wide interval, and a sweep over a genuinely flat grid publishes a wide interval, so the interval alone is a check that returns the same reassuring shape in both worlds. What separates them is the minimum detectable difference standing beside it, which is why the two fields travel together or not at all.

A flat surface is therefore not proof that these parameters do not matter. The honest sentence is that within the searched ranges, on this corpus, measured by nDCG@10 on 500 queries with a minimum detectable difference of 1.5 points, no configuration was distinguishable from another. Every clause in that sentence is a place it could fail, and each names a specific thing a future reader could go and change. A grid that spanned 128 to 4,096 tokens instead of 256 to 1,024 might well have found something, and so might a 4,500-query set. Writing "these parameters do not affect quality" instead of the qualified version is how a team ends up refusing to revisit chunking two years later when the corpus has changed underneath them.

It is also not a licence to skip the confirmation step. The +0.9 on the sealed production split is a weaker number than the +1.51 on the selection split, exactly as chapter 4's account of selection bias predicts, and both belong in the report. Publishing only the flattering one is the habit that produces the +9/+1 surprise two quarters later. The claim that belongs with those two numbers needs a condition attached, though, because their intervals are wide and they overlap heavily: this sweep's confirmation is consistent with a selection bias of about that size and it is also consistent with none, so the drop is what selection bias predicts rather than a measurement of it. Reporting the pair and saying which reading is not yet excluded is stronger than reporting a subtraction as if it were a quantity.

One residual the report does not close belongs in the same paragraph as the qualified sentence. not_tested lists the axes left out of the search, and nothing lists the values left out on the axes that were in it. The evidence covers a rectangle of parameter settings, flatness belongs to that rectangle alone, and every well-formed sentence about this sweep has to carry its rectangle with it.

Where this goes next is out of your hands and into the process. The report format above is worth putting in the repository as a template so the next sweep fills it in rather than reinventing it, and the approval conversation from chapter 2 is worth making a standing gate: reindex count, judge spend, wall clock, production latency, and which split will confirm the winner, agreed before anyone starts embedding. If you want to see the whole loop mechanised, AutoRAG's documentation shows a search space declared in YAML and benchmarked end to end, which is a useful thing to read against your own harness. For the discipline around it rather than the machinery, Jason Liu's Systematically Improving RAG Applications and Hamel Husain's How to debug Retrieval for RAG are the two treatments that stay closest to what actually goes wrong in a team.

The fields in that one-page file are not paperwork, and each of them is the residue of a specific way a sweep gets wasted: values so it can be applied, a baseline so the delta means something, an interval so the delta can be sized against noise, a split id and a selection count so the number is known to be a measurement rather than an artefact of having been chosen. The first sweep you run with this in place will be slower to start than the one you would have run last year and considerably harder to argue with when it finishes.