End-to-end and component metrics

A retrieval-augmented generation (RAG) pipeline answers a question by searching a corpus of documents for it, inserting the handful of passages that come back into a language model's prompt, and having the model write an answer conditioned on them. The arrangement exists because a general-purpose model cannot know your log retention policy or which endpoint you deprecated last month, and re-training it whenever the documentation changes is impractical, so the facts are fetched at question time and handed to it instead. A question therefore passes through at least two places where it can go wrong — the search may never find the passage, or the model may disregard a passage it was handed.

Where a measurement sits is the subject of this topic. An end-to-end evaluation scores the finished output of the whole system against what the user needed. A component evaluation scores the output of a single stage against a standard defined for that stage alone: whether the passage that answers the question appeared in the retrieved list, whether the answer's claims are supported by the context it was given, how long the reranker took. Any feature assembled from more than one model call raises the question — a RAG assistant, a multi-step agent, a classify-then-route chain — and you meet it in practice as a decision about what the eval harness records at each stage and what the dashboard is allowed to show.

This chapter is about what each kind of number can carry: a composite score that stayed flat through a month in which the product was demonstrably failing, the two traces that show why the same score covers unrelated faults, the structural reason a measurement at the end of a chain cannot be inverted onto a link in it, and the reasoning error of letting a flat composite overrule a component signal that moved.

Two failures that scored the same#

A support assistant ran a nightly eval at 03:00 that pushed 300 support questions through the pipeline, sent every answer to an LLM judge — a second large language model prompted to grade the first one's output — with the instruction rate how helpful this answer is on a scale of 1 to 5, and averaged the results. For four weeks the average read 4.08, 4.11, 4.09, 4.12, 4.10.

Two of the answers that support escalated in the same period are worth looking at side by side, because the judge gave them the same score and they failed for reasons that have nothing to do with each other.

Q: "How long are audit logs kept on the Team plan?"
  retrieved chunk ids: [c-8814, c-2201, c-9930, c-4417, c-6652]
  gold chunk id:       c-1188   ← not retrieved
  answer: "Audit logs are retained for 90 days on the Team plan."
  judge score: 2/5

Q: "Which endpoint replaces /v1/exports?"
  retrieved chunk ids: [c-7702, c-3318, c-5561, c-7703, c-1024]
  gold chunk id:       c-7702   ← retrieved, rank 1
  answer: "You can continue to use /v1/exports; it remains supported."
  judge score: 2/5

In the first case, the chunk marked gold for that question — the passage a person has labelled as containing the answer, here the one stating the retention policy for the Team plan — never entered the model's context. No wording of the prompt would have produced a correct answer, because the fact was not there to be used. In the second case, the chunk describing the replacement endpoint was sitting at rank 1 in the context window, and the model wrote an answer that contradicts it — the passage was present and was not used. Both come out of the harness as a 2, both are averaged into the same 4.1, and the 4.1 gives you no way to tell which of these two situations you are in, or in what proportion.

Those two traces do more than illustrate a point; they settle one. The pipeline was in two internal states that share nothing. In the first, the passage carrying the answer was outside the model's context altogether, so no arrangement of words in the prompt could have produced the retention period — the fact was not in the room. In the second, the passage was the first thing in the context, so the evidence was present and the model wrote against it. A judge scoring the finished answer returned 2 for both. The score therefore sends at least two unrelated internal states to the same value, and a quantity that does that carries no information about which of them it came from. One pair of traces establishes it, because a claim of this shape needs a single counterexample and nothing more.

That is the practical meaning of a composite score: it is a measurement of an outcome with no attribution to a cause. The number is accurate about what the user received and silent about which part of the system produced it. A month of a flat 4.1 is consistent with retrieval quietly degrading while generation quietly improves, with both stages holding steady and the eval set being unrepresentative, and with everything being fine except for a long tail of questions the average cannot see. All of those have different fixes and the score distinguishes none of them.

Why the average sat at 4.1 for a month#

Before getting to the structural reason, it helps to see why the number was so inert, because the flatness itself misleads a team into thinking nothing is changing. The 300 scores were not clustered around 4.1; they were 4s and 5s with a thin tail. A typical run distributed as 41 fives, 198 fours, 43 threes, 14 twos and 4 ones, which averages to 4.06. Moving that mean by a tenth of a point requires about 30 questions to change by a full point, and no realistic single release does that.

That figure is worth deriving rather than accepting, because it is what makes the flatness a property of the instrument rather than of the product. An average over three hundred scores is the total of those scores divided by three hundred, so moving the average by a tenth of a point means moving the total by three hundred tenths, which is thirty points of score. A question whose grade falls from a 4 to a 3 contributes exactly one point to that total. Thirty such questions are therefore the smallest change a release can make and still shift the headline by a tenth — and a release that broke a tenth of the eval set outright would announce itself in every other way long before the average noticed.

A judge rubric where the modal score is a 4 for any answer that is fluent, on-topic and not obviously contradictory compresses most of the variation the team cared about into the gap between 4 and 5, where the judge is least consistent — re-running the same 300 answers through the same judge on two consecutive days produced a mean difference of 0.04 and disagreed on the individual score for 31 of them.

The reading a flat line invites is that nothing moved, and that reading has the claim the wrong way round. What a flat line reports is that this instrument did not resolve whatever moved. Those are statements about two different things — one about the system, one about the measurement — and telling them apart takes a single question, which is how large a change this number would have had to see. The derivation above answers it at thirty questions out of three hundred. Anything smaller was never going to appear on this dashboard whether it happened or not, so its absence from the dashboard is not evidence either way.

Against that, three escalations in a week is roughly 0.6% of the traffic through the feature, which is a rate the eval set was never going to resolve. Even if all three failure types were present in the eval set at their true production frequency, they would occupy two questions out of 300, contributing about 0.02 to the mean. The number was not lying and it was not sensitive enough to be the instrument anyone should have been watching for this class of problem.

The habit of watching it comes from somewhere reasonable. Evaluation practice for language models grew up around single-model benchmarking, where a model is a single component and the only place to take a measurement is at its output; a benchmark score for a model genuinely is a component score, because the model is the component. Pipelines inherited the format without inheriting the conditions that made it sufficient. Once a system has four stages, a score at the output has become a composite, and continuing to report it as though it were a property of the thing you can change is the source of most of the confusion in this topic.

Why one score cannot locate a fault#

The reason is structural rather than a defect in the judge. A RAG pipeline is a chain of transformations — a question goes in, some retrieval step selects evidence, a generation step writes an answer conditioned on that evidence — and the end-to-end score is taken at the last link only. A measurement taken at the output of a chain is a function of every link, so its value cannot be inverted back onto any single one of them.

The steps under that sentence are worth walking, because "cannot be inverted" is the kind of phrase a reader nods at and then cannot reuse on a different pipeline. Retrieval takes a question and produces a list of passages. Generation takes that list and produces an answer. The judge takes the answer and produces a number. Each stage's output is the next stage's only input, so the number at the end depends on what happened at every stage and on nothing else. Now ask what reading it backwards would require. It would require each possible score to correspond to one and only one combination of stage behaviours, so that seeing the score tells you which combination you are in. The pair of traces above is a combination that breaks the requirement: two different retrieval outcomes, two different generation outcomes, one score. Once a single score has two sources, nothing recovers the stage from that score alone — not a better judge, not a larger eval set, not a more carefully worded rubric — because the information was not degraded by a noisy measurement. It was never carried in the first place.

The same argument tells you the shape of the fix, and the fix is not a cleverer number at the end. What retrieval did is still visible in one place, which is retrieval's own output, before generation consumes it and replaces it with prose. A probe placed there sees a list of passages, and it can therefore be asked a question that only means anything about a list of passages: was the passage that answers this question in it. That measurement has one stage between it and its subject instead of three, which is why it moves when that stage moves and sits still otherwise. The harness that produces the 4.1 makes the contrast obvious once you look at it:

def run_eval(questions, judge):
    scores = []
    for q in questions:
        answer = assistant(q["text"])          # retrieval + generation, opaque
        scores.append(judge(q["text"], answer))
    return sum(scores) / len(scores)

The call to assistant is where the whole pipeline lives, and nothing inside it is recorded. The function's inputs are a question and its output is a number, so the number can only ever be about the pair. If you want a measurement that can be about retrieval, you have to take a measurement at retrieval's output, with a notion of correctness that applies to a list of passages rather than to an answer. That is the entire idea of component evaluation, and it is less a technique than a decision about where the probes go.

The two kinds of evaluation named above are what follows from that choice of probe point, and each yields a corresponding kind of metric: an end-to-end one computed on the final answer against a reference or a rubric, and a component one computed on a single stage's output against a standard defined for that stage. Their practical consequences differ along enough dimensions to be worth setting out side by side.

Read the last row twice, because it drives most of the sequencing decisions later in this topic. Scoring whether the right passage appeared in a retrieved list is a set comparison against labels; it needs no inference at all.

Why that row reads the way it does is worth spelling out, because "cheap" is usually a claim about a discount rather than about a mechanism. An end-to-end score cannot be computed until an answer exists, an answer exists only after the model has been called, and the judge is then a second model call on top of the first. A retrieval score is computed on a list of chunk identifiers against another list of chunk identifiers, and comparing two lists of identifiers is something a program does on its own. The gap between the two is not a matter of degree. One number requires generation to have already happened and the other does not, which is the whole reason the retrieval number can run on every pull request while the answer number runs nightly at best.

What each number can and cannot tell you#

Suppose a team reports two figures on the same feature: answer quality 4.1/5 and retrieval recall@5 0.79. The first is end-to-end, computed on the answer the user would have seen, and it is the only one of the two that speaks to whether the product works. It cannot tell you why it reads 4.1 rather than 4.6. The second is a component metric, computed on the retriever's output against labelled relevant passages, and it says that for 79% of the eval questions at least one passage judged relevant appeared in the top 5. It cannot tell you whether users were helped, because a retrieved passage that the generator ignores helps nobody, and because a question can be answered correctly from the model's parametric knowledge with no relevant passage retrieved at all.

The asymmetry to hold on to is that the component number bounds the end-to-end number in one direction and not the other. With recall@5 at 0.79, roughly a fifth of the eval set reaches the generator without the evidence it needs, so a large class of failures is already determined before generation runs. Fixing the prompt cannot recover them. Meanwhile a perfect recall@5 of 1.0 guarantees nothing about the answers, because supplying the right passage and using the right passage are separate events, as the /v1/exports trace above shows.

The word "bounds" is carrying real weight there, and the reasoning behind it is short enough to check line by line. Recall@5 of 0.79 says that for seventy-nine questions in every hundred, at least one passage a labeller marked as containing the answer was among the five that reached the model. For the other twenty-one it was not, so on those the model is being asked to produce a fact out of material that does not state it. Some of them still come out right, because a model can know things it was not told and a near-miss passage sometimes carries enough, so failure on that fifth is not certain. What is certain is that nothing downstream can make the missing evidence appear, since every stage after retrieval only ever sees what retrieval handed it. That is the asymmetry stated as a mechanism: the component number caps what generation can be blamed for and caps nothing about what it can be credited with.

This is also where the most common reasoning error appears. Over one release, recall@5 falls from 0.79 to 0.71 while answer quality stays at 4.1, and the tempting conclusion is that the retrieval drop does not matter — users experience the answers, the answers did not get worse, so ship it. That treats the end-to-end score as ground truth that overrides the component signal, which inverts the reason for measuring stages separately. A composite can stay flat while one stage degrades, either because the eval set is too small to resolve the change, or because another stage moved in the opposite direction, or because the questions whose evidence was lost happen to be ones the model can still bluff its way through convincingly enough to satisfy a judge. Eight points of recall is roughly 24 of 300 questions that used to have their evidence and now do not; if the answer score did not notice, the honest reading is that you have discovered something about your answer score, not that you have discovered the retrieval loss is harmless.

Eight points of three hundred questions is twenty-four of them, and doing that multiplication is what turns a decimal into something a person can act on. A fall from 0.79 to 0.71 is not a statement about retrieval in general; it is two dozen specific questions that used to have their evidence in front of the model and now do not, and those questions have identifiers. Listing them and reading five is usually quicker than arguing about whether eight points is a lot, and it is the step that decides whether the unchanged answer score is reassuring or is itself the thing needing an explanation.

Where the boundary of this idea is#

Component evaluation is not error analysis. Reading failures, coding them into categories and counting the categories — the practice covered in Error analysis — tells you what kinds of failure exist and in what proportion. Component metrics tell you which stage owns them, and they do it continuously, on every eval run, without a human reading anything. The two are complements: error analysis discovers that "answer contradicts a retrieved passage" is a real category, and a groundedness component metric is what turns that category into a number you can track per release.

It is also not a replacement for the end-to-end score, and treating it as one is the failure mode on the other side. Stages can each satisfy their own metric while the assembled answer still fails the person who asked — because the right passages arrived in an order that buried the relevant one, because two retrieved passages contradict each other and the answer picks the wrong one, or because the answer is faithful to its context and unhelpful anyway. Chapter 8, on offsetting movements inside a composite, works through that case in detail. For now the useful posture is that the end-to-end number is the one that says whether you have a problem, and component numbers are the ones that say where it lives.

Finally, component metrics are only as meaningful as the boundaries they are attached to, and not every named box in an architecture diagram is a boundary you can score. What makes a seam measurable at all belongs to chapter 4, on which stage boundaries can carry a probe.

The working conclusion is a division of labour between two kinds of number usually conflated into one. The end-to-end score is a statement about the user and the only thing that establishes whether the product is working; it has no address, because it is a function of every stage at once. A component score is a statement about one stage against a standard written for that stage, and it is what turns "quality is 4.1" into "the evidence arrives for four questions in five, and when it arrives the model uses it". Neither substitutes for the other, and the decision that produces both is what the harness writes down as a question passes through.

What that decision requires first is a written account of what the stages are. "Score retrieval separately" is an instruction with no content until you can say what retrieval consumes, what it emits, and what a correct emission would have been — which is where chapter 2, on the stages of a RAG pipeline, starts.