ROUGE and overlap metrics

ROUGE — Recall-Oriented Understudy for Gisting Evaluation — is a family of measures that count how much wording a candidate text shares with a reference. It is the first thing reached for when an output has no type to coerce and no parser downstream to appeal to: a summary, an explanation, a customer-facing answer, where the only material on hand is a human-written reference and the model's text. It is free, it is deterministic, and it has a twenty-year track record, which together make it very easy to put a threshold on.

This chapter is about what the number counts, which is shared word positions and nothing else. That has a consequence people know abstractly and act against routinely: correctness and overlap are related only through the accident that people who know the answer often use similar words. The chapter works through a pair of summaries where the accident fails, the conditions the metric was originally validated under and how production suites invert them, and the job overlap is genuinely excellent at, which is detecting that today's outputs are not last week's.

What the number counts#

Two candidate summaries of the same incident report, scored against the same human-written reference. The reference was written by an on-call engineer as part of a postmortem template:

The outage began at 02:14 UTC when a configuration push removed the read
replica from the connection pool. Checkout requests queued behind a single
primary and timed out for 41 minutes. The fix was a rollback of the config
change; a guard has since been added to block pushes that reduce pool size
below two.

Summary A says the same things in different words:

A bad config deploy at 02:14 UTC dropped the replica out of the pool, so all
checkout traffic piled onto the primary and failed for roughly three quarters
of an hour. Engineers reverted the deploy, and pushes that shrink the pool
below two members are now rejected.

Summary B lifts three sentences verbatim from the source document and joins them with a conclusion of its own:

The outage began at 02:14 UTC when a configuration push removed the read
replica from the connection pool. Checkout requests queued behind a single
primary and timed out for 41 minutes. A guard has since been added to block
pushes that reduce pool size below two. The incident was caused by
insufficient test coverage in the deployment pipeline.

Summary A is accurate. Summary B ends on a claim the source document does not make and the postmortem explicitly rejects — the config push was covered by tests, and it passed them, because the tests asserted on pool health after the change rather than on pool size during it. Scored with ROUGE-L, the variant below, A gets 0.31 and B gets 0.62. The metric ranks the summary containing a fabricated root cause twice as highly as the one that is right.

ROUGE-L is the longest common subsequence between candidate and reference, measured in words, turned into a precision — the share of the candidate's words that lie on that subsequence — and a recall, the share of the reference's words that do, and combined into an F-score, which is their harmonic mean. The subsequence does not have to be contiguous, so word order matters but intervening words do not.

Each of those terms needs unpacking before the numbers mean anything, and the smallest honest place to unpack them is a fragment of the reference rather than the whole of it. The reference ends pushes that reduce pool size below two, and Summary A renders the same requirement as pushes that shrink the pool below two members. A subsequence is what is left when you delete some words from a text without reordering the ones that remain. A common subsequence is a list of words that can be obtained from both texts that way. Reading the two fragments left to right, pushes, that, pool, below and two appear in that order in both — five words, with reduce and size deleted from one side and shrink, the and members deleted from the other.

Five is the longest such list here, and the two properties that make it the right thing to count are visible in the deletions. Order is respected: had Summary A said below two pool, all three of those words would still appear in the reference and the common list would be only below and two, because pool sits ahead of them there and deleting words can never move it behind. Adjacency is not required: pool and below are neighbours in the candidate and separated by size in the reference, and the match survives, because the intervening word was deleted rather than compared. That is the entire content of "the subsequence does not have to be contiguous", and it is the property that separates this variant from the ones further down.

Two numbers come out of that single count because one count answers two different complaints. A candidate can share wording with the reference and still have missed most of it, which is a complaint about the reference's words being absent. A candidate can also cover the reference completely and bury it under three paragraphs of its own, which is a complaint about the candidate's words being unearned. Dividing the matched count by the reference's length answers the first and is called recall. Dividing the same count by the candidate's length answers the second and is called precision. The numerator is identical in both and only the denominator changes, so the two numbers differ exactly to the extent that the two texts differ in length.

The F-score folds them into one, and the particular way it folds them is doing work that an average would not. A harmonic mean is the reciprocal of the average of the reciprocals, which sounds like an arbitrary detour until you notice what a reciprocal does to a small number: it makes it large. A small half therefore contributes a large term to the average being taken, and dragging that average up drags the final reciprocal down. The result is a combined score pulled towards the smaller of its two inputs rather than sitting midway between them. In the limit the property is absolute — if either half is zero the score is zero — which is why the code below returns 0.0 outright when the matched count is zero rather than evaluating the formula, whose denominator would itself be zero at that point.

def lcs(a: list[str], b: list[str]) -> int:
    dp = [[0] * (len(b) + 1) for _ in range(len(a) + 1)]
    for i, x in enumerate(a, 1):
        for j, y in enumerate(b, 1):
            dp[i][j] = dp[i-1][j-1] + 1 if x == y else max(dp[i-1][j], dp[i][j-1])
    return dp[-1][-1]

def rouge_l(candidate: str, reference: str) -> float:
    c, r = candidate.lower().split(), reference.lower().split()
    m = lcs(c, r)
    if m == 0:
        return 0.0
    precision, recall = m / len(c), m / len(r)
    return 2 * precision * recall / (precision + recall)

The lcs function computes that count without enumerating subsequences, and the shape of its table is the reason the whole metric is cheap. dp[i][j] holds the length of the longest common subsequence of the first i words of a and the first j words of b. When the words at those two positions are equal, the answer extends the best result from one word shorter on both sides, which is the dp[i-1][j-1] + 1 branch. When they are not equal, at least one of the two words cannot lie on the subsequence, and the loop does not need to work out which — taking the better of dropping a word from a and dropping a word from b covers both possibilities. Each cell is filled once, from cells already filled, so the work is one pass over a table with a row per candidate word and a column per reference word.

The reference is 56 words. Summary A is 53 words with an LCS of 17 — recall 0.30, precision 0.32, F 0.31. Summary B is 53 words with an LCS of 34 — recall 0.61, precision 0.64, F 0.62.

Those six figures are four divisions and two foldings, and every one of them can be checked by hand from the three lengths and the two matched counts. Summary A's recall is its matched count of 17 over the reference's 56 words, a shade over three tenths, which rounds to the 0.30 quoted. Its precision is the same 17 over its own 53 words, a slightly larger fraction because the denominator is smaller, which is the 0.32. Summary B's recall is 34 over the same 56, a little over three fifths, giving 0.61, and its precision is 34 over 53, giving 0.64. Precision exceeds recall in both rows for one reason only: both candidates are shorter than the reference, so an identical numerator is being divided by a smaller number on the precision side.

Folding each pair is the harmonic mean described above. For Summary A, twice the product of 0.32 and 0.30, divided by their sum of a little over six tenths, comes to 0.31. For Summary B, twice the product of 0.64 and 0.61, divided by their sum of one and a quarter, comes to 0.62. The doubling in the numerator is what keeps the answer on the same scale as its inputs, since a pair of equal halves would otherwise fold to half its own value instead of to itself.

It is worth being clear that the choice of folding rule is not what produces the gap between the two summaries. Each candidate's precision and recall sit within a few hundredths of each other, and the harmonic mean of two nearly equal numbers lands essentially where their average would. What ranks the fabrication above the paraphrase is the matched count, and nothing else in the formula.

Every quantity in that calculation is a count of shared word positions. Nothing anywhere in it reads the meaning of a sentence, checks a number against a source, or notices that B's final sentence has no counterpart in the reference at all; that sentence costs B a little precision and nothing else.

That last clause is the mechanism rather than an incidental detail, and it is worth tracing through the arithmetic. B's fabricated final sentence enters the calculation in exactly one place: it lengthens the candidate, which enlarges the precision denominator. It adds nothing to the matched count, so the numerator does not move. There is no third term anywhere in the formula that a false statement could reduce, and no term that is allowed to go negative. Overlap has a dilution channel and no penalty channel, so the worst an invented conclusion can do is earn slightly less credit per word than a true one would have earned in its place.

The same arithmetic explains why a verbatim splice is both the strongest possible candidate under this measure and the worst possible thing to be blind to. Copying sentences out of the source maximises the matched count by construction, because copied words match themselves and match in order. The characteristic failure of a model that copies is not inside the copied spans at all — it is at the joins, in the sentence that stitches borrowed material into a conclusion nobody wrote. Overlap scores the spans and cannot see the joins.

So the pair establishes both halves of a single fact. Shared wording is not necessary for a right answer, because A is right and shares little. It is not sufficient either, because B shares a great deal and is wrong.

The belief that pair is aimed at is not a careless one, and it survives most contact with real data, which is why it is worth naming in both of its directions. The first direction is that a high score means the answer is right. That is tempting because it is usually true: someone who knows the answer tends to reach for the source document's vocabulary, so across a large sample overlap and correctness genuinely do move together. The second direction is that a low score means the answer is wrong, and it is tempting for the mirror reason. Both hold on the average example and fail on the examples that decide anything — the confidently wrong answer written in the source's own words, and the correct answer written in somebody else's. An instrument whose errors are concentrated exactly where the decisions are is not a weak version of a correctness check.

Where the metric came from, and what it was validated for#

This is not a defect that went unnoticed for twenty years. ROUGE was introduced by Chin-Yew Lin in 2004 for the Document Understanding Conferences, whose problem was ranking a dozen or so summarization systems against each other across a corpus of documents, cheaply, so that annual evaluations did not require a fresh human study for every submission. The paper's claim, and the basis on which the metric was adopted, is that ROUGE scores correlate well with human judgements when averaged over a system's whole output — correlations in the high 0.8s and 0.9s at the system level. Nobody claimed the per-summary score was meaningful, and Lin's setup used multiple reference summaries per document specifically so that a candidate had several legitimate phrasings to overlap with.

Almost every production use inverts both conditions. Modern suites hold one reference per example, written by one annotator with one set of habits, and they read the score per example or gate a release on the mean of a few hundred. Under a single reference, a candidate is being asked to reproduce not the content but one person's rendering of it, and the metric's headroom collapses: two competent human summarizers of the same document typically score somewhere around 0.3 to 0.5 against each other, which is the same band the model outputs occupy. When your ceiling and your floor are that close, the ranking between two systems inside the band is mostly noise about vocabulary.

That band is worth holding against the pair scored at the top of this chapter. Summary A is an accurate rendering of the incident, written roughly the way a second on-call engineer would have written it, and it scored 0.31 — the bottom of the range two humans reach against each other. A correct summary is therefore already sitting at the floor of the metric's honest range under a single reference, which leaves almost nothing underneath it for the metric to use in reporting that a summary got worse. Summary B scored 0.62, above the top of that band, which is a place no human paraphrase reaches, because B was not paraphrasing.

BLEU, the Bilingual Evaluation Understudy and its counterpart on the machine-translation side, was published two years earlier than ROUGE with the same architecture and the same caveat, and has spent twenty years being cited for both.

The family has variants, and switching between them changes what you are blind to rather than whether you are blind. ROUGE-1 counts shared unigrams, meaning single words, so it ignores order entirely and will score a bag of the right words as highly as a sentence; ROUGE-2 counts shared bigrams, adjacent pairs of words, which restores some local order and drops sharply on any paraphrase; ROUGE-L, used above, takes the longest common subsequence, which tolerates insertions between matched words. Summary B scores well on all three, because verbatim text scores well on any measure of shared surface form.

The three are easiest to tell apart on a fragment small enough to write out in full. An n-gram is a run of n adjacent words, so the reference's pool size below two yields the three bigrams pool size, size below and below two. Summary B reproduces reduce pool size below two verbatim and therefore matches all three. Summary A writes shrink the pool below two members, whose bigrams are shrink the, the pool, pool below, below two and two members, and only below two is shared. One paraphrase of a four-word phrase has cost two of the three bigram matches while leaving the underlying requirement identical, which is what "drops sharply on any paraphrase" means once it is made concrete.

Unigrams fail in the opposite direction, and the same fragment shows it. pool size below two and shrink the pool below two members share pool, below and two as single words, and they would share exactly those three however either phrase were shuffled. Write the reference's fragment as two below size pool and every bigram is destroyed while the unigram count does not move at all. That is the precise sense in which ROUGE-1 ignores order: order is not something it scores badly, it is something the counting has no slot for. ROUGE-L sits between the two because a subsequence carries order without requiring adjacency — on the longer fragment worked earlier it credited five of the reference's seven words, where a bigram count over the shorter phrase credited one of three.

The general rule behind those fragments is that each variant fixes an n and goes blind to everything happening at a longer range than n. A bigram count cannot tell that two correct clauses have been swapped, because the swap leaves every adjacent pair intact except at the single joint between them. ROUGE-L escapes the range limit and not the deeper one: a reordering does shrink its subsequence, but the amount it shrinks by reports how many words moved and never whether the move changed what the sentence asserts.

Where overlap earns its keep#

ROUGE-L costs nothing. The dynamic program above is O(mn) on word counts, which is about 0.4 ms for a pair of 500-word texts on one core; a 5,000-example suite scores in a couple of seconds with no network calls, no API bill, no rate limits, and — the property that matters most here — perfect determinism. Run it twice on the same outputs and you get the same number to the last digit.

Both of those figures follow from the table. O(mn) means the work grows as the product of the two word counts, because the table has a row per candidate word and a column per reference word and every cell is visited once. Two 500-word texts therefore need a quarter of a million cell updates, each of which is one comparison and one maximum, and that is the 0.4 ms. Multiplying by 5,000 examples gives the couple of seconds. The growth being a product rather than a sum is the one thing to watch: doubling the length of both texts quadruples the work, so the cheapness holds for summaries and stops holding for whole documents.

Determinism deserves separating from cheapness, because it is what makes an alarm possible at all rather than merely affordable. The function contains no judgement. Its output is fixed entirely by the two strings handed to it, so two runs over the same outputs cannot differ, and no number can move without somebody committing a change to the repository. Any movement in the number is therefore movement in the outputs. That inference is what an alarm is built on, and it is not available from a scorer that calls a model: a model-based number can move because the outputs moved or because the provider changed something on their side, and a delta alarm built on one cannot tell those two causes apart.

The same property bounds how wrong the instrument can be, which is a more useful guarantee than it first sounds. What a deterministic count gets wrong is settled once and for all in the definition of what it counts, rather than drifting from run to run. That is exactly why the argument of this chapter is an argument about the definition and never about reliability. Overlap is entirely reliable about a quantity that is not correctness, and the reliability is what makes the mistake durable — nothing ever misbehaves in a way that would prompt someone to check.

That combination makes it an excellent detector of change and a bad judge of quality, and the difference between those two jobs is the difference between alerting on a delta and gating on a level. A threshold says this output is good enough, which overlap cannot support. A delta alarm says the output distribution moved, which it supports very well: when mean ROUGE-L against a frozen reference set drops from 0.47 to 0.31 between two builds, something changed — a prompt edit that truncated the output, a template that started emitting a duplicated header, a decoding change, a silent model version bump on the provider's side. The metric will not tell you whether the change was good. It will tell you, within seconds of the build and before any human looks at anything, that today's outputs are not last week's outputs, which is a question worth asking on every commit and expensive to answer any other way.

Used that way it needs a band rather than a floor: alert when the mean moves by more than the run-to-run variation of the suite, in either direction, and treat the alert as a prompt to go and read ten outputs.

Two findings that would have justified the block#

Siding with the 0.41 over the raters requires evidence that the two are measuring different populations, and there are exactly two shapes that evidence can take.

The first is that the references are a contract rather than a style preference. If a downstream router keys on vocabulary the references contain, or a regulator specifies the phrasing, then overlap with them is a functional requirement and the metric is measuring the right thing after all. This is rarer than it is claimed to be, and it is checkable in the same way the previous chapters' comparators were: read what consumes the text.

The second is that the rater instructions asked a different question from the one the gate exists to answer. Raters asked which summary is more useful will reward compression; a product that requires summaries to be complete is asking something else, and a 68–32 preference under the first question licenses nothing about the second. The regulatory subset above is exactly this case caught by data rather than by argument.

There is also one structural change that genuinely improves the metric, and it is the one the original setup had: multiple references per document, which lets a candidate match any of several legitimate phrasings and roughly doubles the useful range of the score. It also multiplies the annotation bill, and for most teams the same budget spent on a correctness check buys far more, which is the trade the rest of this topic is about.

What overlap measures is how much of one annotator's phrasing a candidate reproduced. That quantity is cheap, exactly reproducible, and worth computing on every build, which is why the answer to a bad gate is a demotion rather than a deletion — the mirror-image response of ripping ROUGE out of the pipeline throws away a perfectly serviceable instrument because it failed at a job it was never suited for. What the quantity cannot do is arbitrate between two answers, and no threshold on it will make it able to.

Which leaves the arbitration problem exactly where it was. Overlap fails at it because it compares surface forms, and the obvious repair is to compare meanings instead — to embed both texts and measure the angle between them, so that the paraphrase in Summary A finally gets the credit it deserves. That repair works, and it introduces a blindness of a different and more dangerous kind.