RAG-specific metrics

Read these first

• Naive RAG end to end, built once by hand without a framework
• Component vs end-to-end evaluation

What you will know

• Measure faithfulness and distinguish it from relevance

  A response may quote the right passage yet answer the wrong question; relevance and support must be judged against different references.

• Detect an answer that is well-written and unsupported

  Fluency and topical overlap are weak evidence, so unsupported claims often survive casual review.

• Measure citation accuracy separately from answer quality

  A correct-sounding claim with a mismatched or overly broad citation still misleads users and undermines auditability.

• Run context-precision and context-recall evaluations on labeled retrieval results

  Precision penalizes distractors, while recall penalizes missing supporting passages; both require an explicit relevance judgment.

• Score a pipeline where retrieval is good and generation is not

  A strong retriever can mask hallucination, so aggregate scores can hide the generator as the failing component.

Chapters

1. 1

   Faithfulness and answer relevance judge against different references

   A response can quote the retrieved billing page perfectly and still answer a question the user never asked. Faithfulness compares each claim in the answer against the retrieved passages; answer relevance compares the answer against the question. This chapter fixes the two references in your head and shows the quadrant where one passes while the other fails, which is where most disputes about a fine-looking answer live.

   Read chapter
2. 2

   Fluent prose passes review; unsupported claims survive it

   Reviewers spot-checking 200 answers a week mark them good while an audit finds 12% carry a claim no passage supports. The reason is mechanical: casual review checks fluency and topical fit, both of which an invented sentence passes easily. You will learn the claim-decomposition procedure that makes support visible, splitting an answer into atomic claims and reading each one back against the passages that were retrieved.

   Read chapter
3. 3

   Showing a faithfulness judge the gold answer stops it measuring grounding

   A tempting eval harness passes every metric the same four inputs: question, generated answer, retrieved context, gold answer. That convenience silently breaks one of them. A judge that can see the gold answer marks a claim supported when it is merely correct, so a claim appearing in no retrieved passage passes and grounding goes unmeasured. Wire each metric to only the references it is defined over.

   Read chapter
4. 4

   A citation asserts which passage, and concatenated context cannot check it

   Scoring 0.91 faithfulness while users report that source links do not say what the sentence says is not a contradiction. Concatenating the retrieved passages before judging asks only whether support exists somewhere in the set, erasing the boundaries a citation is a claim about. Here you build citation accuracy as its own metric, comparing each claim against the single passage it names, and you leave with the two candidate causes of a low first score named: either the pipeline attributes claims wrongly, or the checker that aligns claims to spans does.

   Read chapter
5. 5

   The judge and the labelled set are instruments with their own error

   Faithfulness, answer relevance and citation accuracy are in practice LLM or NLI verdicts, and a verdict counts as evidence only once it has been scored against human labels on this task and this corpus — which is also how you tell a real attribution defect from a checker defect when citation accuracy comes back at 0.55. The labelled retrieval set has its own ways of flattering you: gold passages harvested from whatever the current retriever already returned, questions generated from the gold chunks so their wording matches, recall computed at a k the production path never uses, queries that do not resemble real traffic. Run through those before you let a number like 0.94 redirect a quarter of engineering.

   Read chapter
6. 6

   Context precision and recall need a per-passage relevance label

   Precision penalises distractors in the retrieved set, recall penalises supporting passages that never arrived, and neither can be computed until someone judges each passage relevant or not. Work through a reranker that cuts k from 20 to 5: precision jumps, recall slips, end-to-end accuracy rises. The chapter explains what that combination means and which class of query quietly absorbs the lost recall.

   Read chapter
7. 7

   High context recall with low answer accuracy puts the loss after retrieval

   A pipeline reporting 0.94 recall and 0.62 end-to-end accuracy has evidence reaching the prompt on almost every query, so a sprint scoped as improving retrieval is aimed at the wrong component. Before redirecting it, verify what the 0.94 was measured on, then split the remaining loss into unsupported claims and grounded answers to the wrong question. Those are two failures with different fixes, and neither is prompting.

   Read chapter