BM25 saturation and length normalisation

Ranked by TF-IDF earlier in this topic, a 6,200-word quarterly release-notes page beat the 180-word page that defines an authentication timeout, because raw term counts have no ceiling and no awareness of length. Saturation is the fix for the first — a curve that rises quickly over the first few occurrences of a term and then flattens, so the fortieth mention adds almost nothing. Length normalisation is the fix for the second — dividing the term-frequency contribution by how long the document is relative to the collection average, so a long page gets no free credit for having more room to repeat itself.

Both live in the same expression, both are controlled by a constant you can set in a config file, and neither touches inverse document frequency. Running the two formulas over that pair of documents, on the same corpus statistics and with no change to the index, reverses the result:

The saturation curve, and the parameter that shapes it#

BM25 replaces the raw count tf with a ratio that rises quickly at first and then flattens:

def tf_component(tf: float, k1: float, K: float) -> float:
    """K carries the length term; K == k1 for an average-length document."""
    return (tf * (k1 + 1)) / (tf + K)

Read it one substitution at a time on an average-length document, where K equals k1. At tf = 1 the numerator is 1 times 2.2 and the denominator is 1 plus 1.2, so the factor is exactly 1.000; at tf = 5 it is 11 over 6.2, or 1.774. As tf grows without bound the ratio approaches k1 + 1 and stops, so with the conventional k1 = 1.2 the ceiling is 2.2, steep at the low end and nearly flat past ten occurrences:

Take two documents of the same length, one containing the query term 20 times and one containing it 5 times. Raw term frequency says the first is four times better. BM25 says it scores 2.076 against 1.774, which is higher by 17 percent. The direction is preserved, since more occurrences really is more evidence, and the magnitude is cut by a factor of twenty. That is the correct shape for the underlying phenomenon — the second mention of a term genuinely tells you more than the fortieth — and it removes the exploit where any sufficiently long document can accumulate its way to the top.

The claim about a factor of twenty needs one clarification, because the two ratios in that paragraph do not look twentyfold apart. Four times against 1.17 times is a ratio of roughly three. What has been cut by twenty is the advantage: three hundred percent more under raw counting becomes seventeen percent more under saturation, and three hundred over seventeen is close to twenty. The distinction is worth keeping straight, since the quantity a ranking cares about is the margin between two documents rather than the ratio of their scores — two documents scoring 1.774 and 2.076 are nearly tied, and two scoring 5 and 20 are not.

k1 sets how quickly the curve flattens. At k1 = 0 the factor collapses to 1 for every non-zero count, making the score purely binary presence-or-absence. At k1 = 3 the curve stays close to linear over the first ten occurrences before bending. Values between 1.2 and 2.0 are conventional, and the honest thing to say about tuning k1 is that it rarely earns the effort: across TREC collections and most production corpora, moving k1 inside that band shifts nDCG@10 by one or two points. It is the parameter people tune because it is the one they understand, and it is the one that matters least.

The parameter that decides your ranking is b#

The length term is where the reversal in the table above came from. BM25 scales the saturation constant by the document's length relative to the collection average:

import math

def bm25(tf: dict[str, int], dl: int, avgdl: float, df: dict[str, int],
         N: int, k1: float = 1.2, b: float = 0.75) -> float:
    total = 0.0
    for term, f in tf.items():
        idf = math.log((N - df[term] + 0.5) / (df[term] + 0.5) + 1)
        K = k1 * (1 - b + b * (dl / avgdl))
        total += idf * (f * (k1 + 1)) / (f + K)
    return total

dl is the length of this document, avgdl is the mean length across the collection, and b controls how much of the length ratio is applied. At b = 0 the expression 1 - b + b*(dl/avgdl) collapses to 1 and K is just k1, so document length is ignored entirely and the formula behaves like saturated TF-IDF. At b = 1 it becomes dl/avgdl and the length ratio is applied in full, so a document twice the average length has its K doubled and needs substantially more occurrences to reach the same score. Everything in between is a partial application. That is the whole of what b does, and it is the answer to give when someone asks: it controls how strongly document length is normalised, from ignore to full.

The sentence that expression hides is that b moves short and long documents in opposite directions, and seeing that is what makes the reversal in the opening table stop being surprising. The factor 1 - b + b*(dl/avgdl) is exactly 1 when the document is of average length, whatever b is. Above average the ratio exceeds one, so the factor exceeds one and K grows. Below average the ratio is under one, so the factor drops below one and K shrinks. A smaller K makes the saturation ratio larger for the same term count. So raising b does not simply penalise long documents; it penalises long ones and rewards short ones at the same time, which is why a modest value of it can close a large gap.

Running the reference page through the formula makes that concrete. At 180 words against a 420-word average the length ratio is a little over four tenths, so at b = 0.75 the whole factor is a little under six tenths and K is that same fraction of k1 rather than k1 itself. Three occurrences of authentication then reach a little over four fifths of the way to the ceiling — which is where the saturation table above puts a document of average length holding five occurrences. Being short has bought this page two extra mentions it does not have to contain. Multiplying that ratio by the inverse document frequency of authentication — the rarer of the two query terms, and the factor BM25 carries over from TF-IDF untouched — accounts for most of the document's 12.56.

The release notes go the other way and go much further. At 6,200 words the length ratio is nearly fifteen, so the factor is a bit over eleven and K lands near thirteen and a half — more than eleven times what it was. Its eleven occurrences of timeout then produce a saturation ratio a shade under one. That figure is worth stopping on, because a saturation ratio of one is exactly what a single occurrence in an average-length document scores. Eleven mentions spread across fifteen average documents' worth of text are worth one mention in one average document, which is the length normalisation stating its purpose in a single number.

On a corpus with an average chunk length of 420 words, the two documents from the opening table move like this as b varies:

The ranking flips between b = 0 and b = 0.3, and everything above that only widens a gap already decided. How narrow that first row is deserves attention: with length ignored, the release notes win by under half a point out of eleven, so the two documents are all but tied. The short page's advantage is three occurrences of the rarer term; the long page's advantage is eleven saturated occurrences of the commoner one, and those two very nearly cancel. Introducing even three tenths of the length ratio breaks the tie decisively, because it multiplies the long document's K by about five while nudging the short document's slightly downward. One lever moving both documents in opposite directions is why a small change in b produces a large change in order, and it is why the sweep below has a peak rather than a plateau. On a collection where lengths span 40 to 4,000 words — which is what API reference pages look like, since a one-line enum constant and a full authentication guide live in the same set — b determines the result order for most queries. A length ratio of 100 between the shortest and longest document means K varies by nearly two orders of magnitude across the collection at b = 1, and it varies not at all at b = 0. That is exact rather than approximate: at b = 1 the length factor is the length ratio, so K inherits the collection's length spread unchanged, and the shortest and longest documents differ in K by the same hundredfold that separates 40 words from 4,000. Nothing else in the formula has that range. k1 moves over a band from about 1.2 to 2.0 and is worth one or two nDCG points on this kind of set; b reaches across two orders of magnitude of K and is worth ten. They are not comparable levers, and the reason k1 is the one people reach for is that its effect is easy to picture and its authority is small.

Sweeping b against your own judgments#

The default in Lucene, Elasticsearch, OpenSearch and every library that copied them is b = 0.75, chosen because it worked well on the TREC ad-hoc collections of the 1990s, which were news articles of broadly comparable length. It is not a constant of nature and there is no reason your documents resemble the ones it was fitted on.

Sweeping it takes a labelled query set and an afternoon. Fifty to a few hundred queries with the correct chunk marked is enough to see the shape:

for b in [0.0, 0.25, 0.35, 0.5, 0.75, 1.0]:
    scores = []
    for q in judgments:                      # {"query": ..., "gold": chunk_id}
        ranked = search(q["query"], k1=1.2, b=b, size=10)
        scores.append(ndcg_at_10(ranked, q["gold"]))
    print(f"b={b:<5} nDCG@10={sum(scores)/len(scores):.3f}")

b=0.0   nDCG@10=0.521
b=0.25  nDCG@10=0.663
b=0.35  nDCG@10=0.711
b=0.5   nDCG@10=0.694
b=0.75  nDCG@10=0.608
b=1.0   nDCG@10=0.537

That is a real shape rather than a monotone curve, and the peak is well below the default. Ten nDCG points separate b = 0.35 from b = 0.75 on a 340-query judgment set, which is a larger improvement than most teams get from changing embedding models, at a cost of one config value. Assembling the judgments that make such a sweep meaningful is covered by topic 3.4, on building a golden dataset.

The failure mode at high b is specific enough to recognise without instrumentation. Full length normalisation makes short documents cheap to score highly, so a 40-word stub that mentions the query term once — a glossary entry, a deprecated stub page, a section heading that got chunked on its own — outranks the 900-word page that actually explains the thing. If your top results are consistently too short to be useful, lower b. If they are consistently sprawling pages that mention the term in passing, raise it.

In Elasticsearch the parameter lives in a named similarity — Lucene's word for a pluggable scoring function with its constants filled in, which has nothing to do with cosine similarity — applied per field, and it has to be defined when the index is created or set while the index is closed:

PUT /docs
{
  "settings": {
    "index": {
      "similarity": {
        "tuned_bm25": { "type": "BM25", "k1": 1.2, "b": 0.35 }
      }
    }
  },
  "mappings": {
    "properties": {
      "body":  { "type": "text", "similarity": "tuned_bm25" },
      "title": { "type": "text", "similarity": "tuned_bm25" }
    }
  }
}

Chunking is the fix that makes b stop mattering#

Tuning b is a way of compensating for a length distribution you accepted upstream. The other move is to stop having that distribution. If chunks are built to a target of 200 to 400 tokens with an overlap, rather than one chunk per source page, the length range compresses from 40–4,000 words to roughly 120–450, and the whole question loses its force. Re-running the same sweep after rechunking the same documents:

b=0.25  nDCG@10=0.741
b=0.5   nDCG@10=0.758
b=0.75  nDCG@10=0.752

Put the two sweeps side by side and "nearly flat" becomes a number. On the original chunking the sweep ran from 0.521 to 0.711, a spread of nineteen points across the parameter's range. After rechunking, the three values span under two points, and the best of them is above the best the parameter ever reached before. The cause is upstream of the formula. b acts on the ratio between a document's length and the average, so its authority over the ranking is bounded by how much that ratio varies — a hundredfold spread of lengths gives it a hundredfold range of K to play with, and a spread of under four gives it almost nothing. Compressing the lengths did not improve the parameter; it removed the variation the parameter existed to correct.

One detail in that comparison is a unit trap worth naming. The chunking target is stated in tokens and the resulting range in words, and those are different units — a chunker counting tokens does not produce a predictable word count. It happens not to matter here, because everything b responds to is a ratio of lengths, and a ratio is the same number whichever of the two units both sides are measured in. A figure quoted in mixed units would matter a great deal if the formula used an absolute length, and BM25 never does.

The peak is higher than anything the parameter sweep achieved on the original chunking, and the curve is nearly flat, which means the parameter has stopped being load-bearing. That is the more durable outcome, because a flat curve does not need re-tuning when the collection changes, whereas a tuned b = 0.35 was fitted to one length distribution and will drift when someone imports a new document set. The two moves compose in an obvious order: rechunk first, then sweep, and expect the sweep to find less to give you.

Cosine normalisation, BM25F and BM25+#

The most common wrong answer to "what does BM25 add over TF-IDF" names cosine normalisation and stopword handling, and neither is part of it. Cosine normalisation belongs to the vector space model, which BM25 is not built on; length normalisation in BM25 is the b term dividing inside the saturation denominator, and it behaves differently from dividing by a Euclidean norm — most visibly in that it is tunable and cosine is not. Stopword handling is an analysis-time decision, and in any case IDF already reduces common terms to near-zero weight without anyone maintaining a list.

Two extensions exist for cases where plain BM25 is genuinely insufficient, and knowing their names keeps you from reinventing them. BM25F handles multi-field documents properly: rather than scoring the title and the body separately and adding, which double-counts and interacts badly with per-field lengths, it combines the term frequencies across fields with per-field weights before applying saturation, so a term appearing once in a title and once in a body saturates as a single count of weighted occurrences. Boosting a title clause in a bool query is the common approximation and it is not the same thing; the difference shows up on short titles, where the boosted clause saturates almost immediately and the boost stops doing anything. The mechanism is the length term acting on a field whose average length is a handful of words. A title is far below its field's average only rarely and is short in absolute terms always, so K for that field is small, and a small K means even one occurrence lands close to the ceiling. A title containing the term once and a title containing it three times therefore score almost identically, and multiplying both by a boost scales two numbers that were already the same. The clause becomes a flat bonus for containing the term at all, which is a useful thing and is not what anybody thought they were configuring. BM25+ adds a small constant δ to the term component, on the observation that at b near 1 a very long document containing the term can score arbitrarily close to a document not containing it at all, which is a lower-bound problem rather than an upper-bound one. That observation is one limit taken in the formula rather than an empirical finding, and it takes ten seconds to check. At b = 1, K is k1 times the length ratio, so as a document grows without bound K grows without bound with it. The term component is the count times k1 + 1, over the count plus K. Hold the count fixed and send K to infinity and the whole ratio goes to zero. A document containing the term contributes nothing, which is precisely what a document not containing the term contributes. The additive constant is there to put a floor under that, and the reason it stops mattering after chunking is the same reason b does: with lengths inside a narrow band, no document is long enough to get near the limit. It matters on collections with extreme length spread and is a rounding error on chunked ones.

Scores are relative to a corpus that keeps moving#

The average document length in the formula is a live statistic rather than a constant. Lucene computes it per segment from the stored field lengths, so it drifts as documents are added and as segments merge, and a bulk import of unusually long documents shifts avgdl for every query, changing scores on documents nobody touched. Scores in a lexical index are corpus-relative by construction, which is why comparing a raw BM25 score from today against one from last month tells you very little, and why absolute score thresholds — "only show results scoring above 8.0" — are a fragile way to cut a result list.

The direction of that drift is worth predicting rather than discovering. K depends on this document's length divided by the average, so raising the average lowers every existing document's ratio. A lower ratio gives a smaller K, a smaller K gives a larger saturation ratio, and a larger saturation ratio gives a higher score. So importing a batch of unusually long documents raises the scores of every document already in the index, none of which changed. A threshold set last month against the old distribution now admits more than it did, and the change arrives with no deploy, no config edit and nothing in a diff to point at.

BM25 also has no model of term dependence. It scores each query term independently and sums, so it cannot tell "record player" from "player record"; word order and adjacency are invisible to it unless you add a phrase or proximity clause on top, which is a separate query construct rather than part of the scoring function. It has no notion of document quality, freshness, or authority either — a deprecated 2019 page and its 2026 replacement score identically if their text is identical, and separating them requires a boost from outside the formula.

Saturation and length normalisation are the whole of what BM25 adds, and both are modifications to one factor of a product whose other factor is unchanged from Spärck Jones's original statement of term specificity. The rare-term behaviour that makes lexical search useful is therefore not something BM25 invented, and the two parameters are the only tuning surface there is: k1 shapes how fast repetition stops paying, b decides how much length is held against a document, and b is the one that will change your rankings.

Everything BM25 knows still comes from term overlap. If a user asks about "login expiry" and the document says "session timeout", every term score is zero and the tuning of k1 and b is irrelevant, because there is nothing to tune. That gap is what dense retrieval exists to close, and it is the fair criticism of lexical ranking. The converse gap is less discussed and considerably more actionable: there is a class of query where BM25 does not merely compete with a dense retriever but beats it outright, and the reason is entirely contained in the IDF term.