Sequencing component evals

An eval's worth is not its coverage but its ability to settle an argument, and settling an argument requires the number to be trusted by the people arguing. Two weeks spent thinly across every boundary and two weeks spent making one boundary credible produce very different meetings.

That kind of credibility is not a soft property, and it is worth defining before it is used as a criterion. A number is credible in this sense when the two obvious objections to it have answers on the day they are raised: how far would this move if nothing changed, and where did the labels come from. The first has a numeric answer that follows from the size of the eval set. The second has a procedural answer that follows from how the set was built. An eval that cannot answer either does not lose the argument on the merits; it simply fails to participate, and the decision reverts to whoever is most confident in the room.

This chapter is about which component eval to build first and why: build cost and decision value for five candidates on one table, the defence of building one properly rather than all of them, how the day and a half of labelling is spent, how to predict what fixing the chosen stage will do to the end-to-end score and what would prove the prediction wrong, and the pipelines where the whole apparatus is the wrong investment.

Cost and decision value, side by side#

The last row carries forward the result of chapter 4, on which stage boundaries can carry a probe: a boundary with no definable correct output consumes the largest share of the two weeks and returns a constant. The row above it is more interesting, because groundedness is genuinely useful and still not first — it requires generation outputs to exist for every eval question, which means the full pipeline runs before you can score anything, and its cost scales with the answers rather than with the questions.

Retrieval recall is first because of a combination of properties rather than any single one. It runs without a generation call, so a full pass is seconds and cents and can go in a pre-merge check rather than a nightly job. It needs labels, and those labels are the same labels every other retrieval-side metric in the topic reuses, so the expensive part of the build is an investment the rerank eval inherits for free. It sits at the top of the chain, so its value bounds what every downstream stage can achieve — a question whose evidence never arrives cannot be rescued later, which is the argument of chapter 6, on prompt fixes and missing evidence, in metric form. And on the pipeline at hand it has the highest prior — the strongest expectation held before any new measurement is taken — of being where the fault lives, given that fifty-seven percent of attributed failures in the nightly build were gold chunks that never appeared among the candidates.

Why five thin evals is the worse plan#

The objection to building all five is not that breadth is bad. It is that a two-day retrieval eval built on forty hastily labelled questions produces a recall figure with a standard error near 0.08, which cannot distinguish 0.58 from 0.72, so when someone says "I don't think the labels are right", the number cannot defend itself and the discussion returns to opinion. The same two days spent on 300 well-sourced labels produces a number that survives the meeting.

The word "cannot" there is doing arithmetic rather than rhetoric. A standard error of 0.08 means the plausible range around a measurement of 0.58 runs roughly two of those either side, so it reaches from the low forties to the mid-seventies. The figure 0.72 is inside that range. A measurement whose plausible range contains the value you are trying to rule out has not ruled it out, whatever the point estimate says. With 300 labels the standard error is 0.025, the plausible range is about five points either side, and 0.72 sits well outside it. Nothing about the labelling technique changed between those two cases. The only thing that changed is how many questions the average was taken over, and that alone decided whether the number could speak.

Building all five also front-loads the expensive stages before you know whether they matter. Groundedness and any answer-correctness score require full pipeline runs per iteration; if retrieval turns out to own most of the failures, that budget bought a precise measurement of a stage that was not the constraint. And the plan quietly assumes that the five boxes in the diagram are the five real boundaries, which the seam test disproved for at least one of them and made conditional for another.

Both objections share a root, which is that instrumenting a boundary is not the same activity as producing a number at it. Producing a number takes a day. Making that number survive the first person who doubts it takes labels somebody stands behind, a sample size that reaches the movement you care about, and a record of how both were chosen. A plan that budgets for the first and calls it the second will finish on time and change nothing.

The ordering that follows from this is: build the retrieval eval properly, use it plus traces to get the attribution distribution, then build the second eval where that distribution points. In practice the second is usually the rerank eval on frozen candidates, because it is cheap and it splits the largest bucket into "never found" and "found and lost", which are owned by different people and fixed by different work.

How to spend the day and a half#

Almost all of the retrieval eval's build cost is the labelled set, so that is where the decisions are. Size follows from the smallest change you need to detect: the standard error of a proportion near 0.75 is about 0.043 at n = 100, 0.025 at n = 300, and 0.022 at n = 400, so a set of 100 cannot see a five-point movement and a set of 300 can. Beyond 400 the curve flattens and the marginal question buys very little, which is why 300 to 400 is the range most of these sets settle at regardless of how many documents are indexed.

Why it flattens is the one fact about sample size worth carrying around, and the three figures above demonstrate it. The number of questions enters the standard error underneath a square root, so precision improves with the square root of the count rather than with the count. Quadrupling the set therefore halves the error and nothing better: going from 100 questions to 400 takes 0.043 to 0.022, which is four times the labelling for twice the precision. The step from 300 to 400 is a third more labelling for a movement in the third decimal place. That is the shape of every argument about eval set size, and it is why the useful question is not "how many can we afford" but "what is the smallest change we need to see", which picks a point on the curve and stops.

Which 300 matters more than the exact count. Sampling uniformly from traffic overweights the handful of intents that dominate volume — on this product, four intents covered 38% of questions — and leaves the long tail, where retrieval usually fails, represented by two or three items. Stratifying by intent, with a floor of fifteen questions per intent and the remainder allocated by volume, produces a set where per-intent recall is readable rather than a single average that moves only when the head moves.

The floor of fifteen is where the sampling argument meets the sample-size argument, and it is the reason stratification is not merely tidier. A recall figure computed over three questions can take four values — none, one, two or all three — so the finest movement it can report is a third, and no change worth shipping is that large. Fifteen questions per intent is still a coarse instrument and it is coarse in a way that is at least readable: it can distinguish an intent where retrieval mostly works from one where it mostly does not, which is the distinction the slice exists to draw. Uniform sampling does not fail to measure the tail because the tail is unimportant. It fails because a handful of questions cannot resolve anything, and a set that allocates by volume alone will always leave the tail with a handful. Reserve about a fifth of the set for questions drawn from the last two weeks of traffic and refresh that fifth monthly, because documentation and product surfaces change and a frozen eval set slowly becomes a measurement of last quarter's product.

Labelling itself goes faster than teams expect once it is framed as verification rather than search. Run the current retriever with k = 20, show an annotator the question and the twenty candidates, and ask which of them contain the answer; that is about 45 seconds per question rather than the several minutes it takes to find passages unaided.

That difference is what makes the day and a half in the cost table arithmetically possible rather than optimistic. Three hundred questions at three quarters of a minute each is under four hours of annotator time, which fits inside a working day with the sampling and the audit pass around it. The same three hundred at several minutes each runs to two or three full days of somebody's attention before any of the checking happens, and a task of that size gets scheduled rather than done. The saving comes from replacing an open-ended search over a whole corpus with a closed-ended verdict on twenty candidates. The annotator is answering "which of these" rather than "where is it", and only the first of those has a fixed amount of reading in it.

The bias that verification introduces is real and bounded — chunks the current retriever never surfaces will never be labelled gold, so recall is overstated — and it is corrected by having the annotator search the indexed documents directly on a random 15% of items, which on this set found unlabelled gold chunks for 9% of questions and put a number on the overstatement rather than leaving it as a worry.

Predicting what a retrieval fix is worth#

Before committing a quarter to retrieval, it is worth writing down what you expect the end-to-end score to do, because a prediction that turns out wrong is more informative than a vague expectation that turns out ambiguous. The decomposition is simple: the end-to-end pass rate is the pass rate on questions whose gold evidence reached the model, weighted by how often that happens, plus the pass rate on the rest.

def project(recall, p_pass_given_gold, p_pass_without_gold):
    return recall * p_pass_given_gold + (1 - recall) * p_pass_without_gold

# measured on the current build, 400 questions
# recall@5 = 0.58, end-to-end pass = 0.71
# slicing the same run by whether gold reached the context:
#   with gold    : 204/232 pass = 0.88
#   without gold : 79/168  pass = 0.47

project(0.58, 0.88, 0.47)   # -> 0.708, matches the observed 0.71
project(0.85, 0.88, 0.47)   # -> 0.819

That one-line function is the whole model, and it is worth deriving rather than accepting, because every prediction in the rest of this topic is made with it. Start from the definition of the end-to-end pass rate: the number of questions that passed, divided by the number of questions asked. Now split the questions into two groups by a property that has nothing to do with passing — whether the gold evidence reached the model. Every question is in exactly one group, so the passes in the whole set are the passes in the first group plus the passes in the second, and dividing that sum by the total is the same as taking each group's own pass rate and weighting it by that group's share of the total. Recall is precisely that share for the first group, so the second group's share is one minus recall. Substituting those two shares gives the expression in the function, and no assumption was made anywhere: the identity holds for any system, any two groups that partition the set, and any pass rates.

The comments under the function are the measured values, and putting them through the identity by hand is what shows it is not a model of anything. Of 400 questions, 232 had their gold evidence in context, and 232 out of 400 is 0.58 — which is recall, arriving from a different direction. Those 232 passed 204 times, and 204 out of 232 is 0.88. The other 168 passed 79 times, and 79 out of 168 is 0.47. Now weight them: the 0.58 share passing at 0.88 contributes a little over half the whole set, and the remaining share passing at 0.47 contributes just under a fifth. Those two contributions add to 0.708, and the observed end-to-end pass rate is 0.71. The identity did not predict that. It reproduced it, which is the check that the two conditional rates were measured on the same run as the thing they are being used to explain.

The prediction is the same expression with one input changed. Raising recall to 0.85 leaves both conditional rates where they are and moves weight from the weaker term to the stronger one: the first term grows to nearly three quarters of the set and the second shrinks to well under a tenth, which is where 0.819 comes from. Everything the estimate assumes is in that sentence — that fixing retrieval moves questions between the groups and changes neither group's internal pass rate — and the falsifier table below is a list of the ways that assumption fails.

Lifting recall from 0.58 to 0.85 while touching nothing else predicts an end-to-end pass rate of about 82%, an improvement of eleven points, against the 0.4 points that three weeks of prompt work produced. Both numbers come out of the same 400-question run, and the two conditional rates are the entire content of the estimate — they are measurable today, from a single sliced eval, and they are what makes this a projection rather than a hope.

The 0.47 in the second slice deserves attention, since a naive reading would say questions without evidence should fail. They do not, for three reasons that are each worth checking: some questions are answerable from the model's parametric knowledge, particularly generic ones about widely documented protocols; some labels are incomplete, so a chunk that answers the question was retrieved and simply was not marked gold; and a pass/fail judge with a lenient rubric will pass a partially correct answer. A high p_pass_without_gold is the single largest determinant of how little a retrieval fix buys you, so if it comes back above 0.5, audit a sample of those passes before making the argument to your team.

Four things would make the projection wrong, and naming them in advance is what makes the estimate arguable rather than a forecast to be defended after the fact. The first row is the one worth understanding in full, because the same shape appears whenever a rate measured on one group is applied to another. The figure 0.88 was measured on the questions retrieval already handles. The questions a retrieval fix would recover are, by definition, the ones it currently fails on, and there is no reason those two populations should be alike — a question whose evidence is hard to retrieve is often one whose evidence is scattered, unusually worded or spread across documents, which are the same properties that make it hard to answer once retrieved. Applying 0.88 to them assumes an exchangeability nobody has checked, which is why the check in the right-hand column is to recompute the rate on the recovered questions alone rather than on the set as a whole.

When to argue against building component evals at all#

There are pipelines where this whole apparatus is the wrong investment, and being able to say which is part of knowing the technique.

A single-stage system has no interior boundary, so there is nothing to attribute; the end-to-end score already is the component score. A pipeline whose stages are being added, removed and reordered weekly will invalidate its labels faster than it can collect them, and a prototype at that stage is better served by a small end-to-end set and reading failures directly. A system whose end-to-end score already meets the product bar does not have an attribution problem to solve, and building attribution for it optimises a number that nobody is going to act on. Where traffic and eval sets are small — a set of forty questions cannot resolve a five-point change — the labels cost more than the decisions they inform are worth.

The harder case is an agentic pipeline where the model decides when to retrieve and with what query. There is no single retrieval boundary to score, since a request may contain zero, one or six searches. Component evaluation is still possible there, but the boundaries are different ones — did the model call the tool at all, were the arguments well-formed, did the trajectory terminate — and forcing the chunker-retriever-generator vocabulary onto it produces metrics that measure the harness rather than the system.

It is also worth being precise about what "cheapest high-signal first" does not claim. It does not claim retrieval is the most important stage, since importance is a property of where your failures are. It does not claim recall is a goal in itself; a retriever tuned until recall@5 reaches 0.95 by returning fifteen chunks and filtering to five on a threshold has moved a number and may have made the answers worse. And it does not claim the component eval replaces the end-to-end one — the projection above is stated in end-to-end terms precisely because that is the score the product is judged on.

Sequencing, then, is a comparison of build cost against what each number can decide, and the winner on a retrieval pipeline is usually recall@k, built once, on a few hundred stratified labels that every other retrieval-side metric will reuse. What makes it first is not that retrieval matters most but that the eval is seconds and cents to run, that its value bounds every stage below it, and that its labels are the reusable asset. What makes it worth building well is that a number with a wide interval loses the argument it was built to settle.

Which raises the question a team with good component evals eventually asks out loud: now that retrieval and generation each have their own metric, why keep paying for the full end-to-end run. The answer is in a pair of releases where the composite moved one point in the right direction and two stages moved fifteen in opposite ones, which is where chapter 8, on cancellation inside an aggregate score, begins.