Analysers and keyword fields

Between the JSON you POST to a search engine and the postings lists it later walks, there is an analyser: a chain that turns a string into the sequence of terms that will actually be stored. Elasticsearch calls it an analyser, Lucene an Analyzer, PostgreSQL a text search configuration, and in every case it runs in three stages. Character filters rewrite the raw string, a tokeniser cuts it into tokens, and a chain of token filters rewrites, splits, merges or drops those tokens. Whatever comes out the far end is what gets written to the index. The original string is kept in _source for display, but nothing in the query path ever looks at it again.

That makes the analyser the one component in a lexical stack whose mistakes are unrecoverable downstream. Ranking operates on candidates, candidates come from postings lists, and postings lists contain only terms the analyser emitted — so a document whose identifier was shattered into fragments is not a badly ranked document, it is a document the engine never considered. The settings involved are usually inherited rather than chosen, which is why an index can pass every review it gets and still be unable to find a part number.

Analysis runs once, at write time, and it is destructive#

The word destructive is the operative one. The chain is a lossy transform applied before storage, more like a downsampled image than like a view over the original. You cannot query a term the analyser did not produce, in the same way you cannot recover 4K detail from a 480p encode by improving the display.

The comparison is an analogy and it breaks in a way worth knowing about before it misleads anybody. Downsampling loses detail evenly, so every part of the picture gets slightly worse and the result still resembles what it came from. Analysis is selective. It leaves ordinary prose almost intact and destroys certain strings completely, and the strings it destroys are the ones a user is most likely to type when they know exactly what they want. Worse, the residue is not a blurred version of the original — it is a set of fragments that match other documents confidently. A downsampled image looks like a worse copy of the right thing; a shattered part number looks like a confident answer about the wrong thing, which is a failure mode no amount of improving the display would ever produce.

The fastest way to see what your own configuration does is to ask it directly, which every engine supports without touching the index:

curl -s localhost:9200/parts/_analyze -H 'Content-Type: application/json' -d '{
  "analyzer": "english",
  "text": "A-100-B"
}' | jq '.tokens[] | {token, position}'

{"token": "100", "position": 1}
{"token": "b",   "position": 2}

Two tokens, and neither of them is the part number. The standard tokeniser applies the Unicode text segmentation rules from UAX #29, which treat a hyphen between alphanumerics as a word boundary, so A-100-B becomes A, 100, B before any filter runs. The lowercase filter maps those to a, 100, b. A stopword is a word such as the or and that occurs too widely to distinguish one document from another, and an analyser drops them so the index does not carry postings lists that separate nothing. The English stopword list contains a, so the first token is dropped entirely. What reaches the index is the token 100, which appears in 214,000 chunks of one engineering corpus, and the token b, which appears in 61,300 of them, mostly as a variable name, a list marker, or a column header.

Those two document frequencies are the whole of the damage, and they are worth reading against the thing that was destroyed. A part number is, by design, a term with a document frequency of one: it names exactly one part, so exactly one document should hold it. The analyser has replaced a term of frequency one with two terms whose frequencies are in the hundreds of thousands. In the language of the previous chapter, on inverted indexes and the walk over postings lists, the query has gone from the cheapest shape an engine serves — one very short list bounding everything else — to two long lists with no short one to drive from. The cost model and the answer both change in the same instant, and neither change happens in the part of the system anybody is looking at.

That is the root cause of the entire class of bug, and it is worth stating without hedging: the tokeniser split on the hyphens, leaving fragments that match nearly everything. Stemming and lowercasing are the usual suspects, because they are the settings somebody changed, and they are downstream of the damage.

Stemming does its own damage elsewhere and it is worth knowing what it looks like, because it presents differently. Porter stemming strips suffixes so that retry, retries and retried collapse onto one indexed term, and the algorithm is a sequence of suffix-stripping rules applied without a dictionary. It maps authenticating, authenticated and authentication to authent, which is the behaviour you wanted, and it also maps universe and university to univers, which merges two unrelated terms into one postings list and inflates that term's document frequency. The reason that merge degrades ranking rather than merely widening it takes one more step, and it runs through the scoring. Rarity is what a lexical scorer pays for: a term in few documents is worth more than a term in many, and the next two chapters make the exact form of that precise. Merging universe into university produces one postings list holding the union of both sets, so its document frequency is the sum of the two. A larger document frequency means a smaller weight. So the merge does two things at once and they compound. Documents about the wrong subject enter the candidate set, and every document about the right subject is scored with a weight lowered by the arrival of the wrong ones. Over-stemming of this kind degrades ranking quietly rather than returning nothing, so it shows up as "results are a bit off" rather than as a ticket. The minimal_english or light_english stemmers strip fewer suffixes and collide less; kstem uses a dictionary and refuses to stem words it does not recognise, which is usually the right choice for technical corpora where a large share of the vocabulary is not English at all.

There is also a second analyser in the system that people forget exists. Elasticsearch applies one at index time and, by default, the same one at query time, but search_analyzer can override it — and when the two disagree, queries silently miss. An index analysed with an edge n-gram filter, which stores every prefix of a token so that a user typing auth still matches authentication, and searched with the same filter will match auth against the ngram of auth in the query as well as the document, producing results nobody asked for; the standard fix is to n-gram at index time and use standard at search time. The symptom of a mismatch is a large result set with nothing relevant in it, and the diagnosis is running _analyze twice, once with each analyser, and comparing the token lists.

That symptom is worth pausing on, because it is exactly the symptom of the part-number failure above and the two have unrelated causes. A shattered identifier and a pair of disagreeing analysers both produce a long list of confident, irrelevant results, and the response the symptom invites in both cases is to adjust the ranking. Running _analyze twice separates them in a minute. If the index-time and query-time token lists agree with each other and neither contains the string you typed, the analyser is destroying it and the fix is a mapping. If the two lists disagree, the string survived on one side and not the other, and the fix is a search_analyzer setting with no reindex attached. Two causes, one symptom, one two-command test — which is the shape most worth having in a debugging routine.

What the same string looks like under four configurations#

The bottom row is how most production mappings end up, a mapping being the per-field schema that fixes which analyser runs over a field and how the output is stored, because it is the only one that serves both query classes. The word_delimiter_graph filter with preserve_original: true and catenate_all: true emits the untouched token alongside the pieces, so a user typing the full part number matches the preserved token and a user typing 100 still finds it. The cost is index size: emitting five terms where the standard analyser emitted three inflates the postings for that field by roughly 60 percent, and on a field that is mostly identifiers it can be worse. Count the terms in the bottom row of the table against the third row and the sixty percent falls out — five over three is two thirds more postings for the same source string, and postings are what the field costs. The reason it gets worse on identifier-heavy fields is that the count is per value rather than per field: a value with more internal punctuation produces more pieces, so a field of part numbers with two hyphens each pays the multiplier on every row, while a field of prose pays it only on the occasional hyphenated word.

The alternative, and the one to reach for when the field is only ever an identifier, is a multi-field mapping. Elasticsearch will analyse the same source value more than once and store the results under different names:

PUT /parts
{
  "settings": {
    "analysis": {
      "normalizer": {
        "lower_exact": { "type": "custom", "filter": ["lowercase"] }
      }
    }
  },
  "mappings": {
    "properties": {
      "part_no": {
        "type": "text",
        "analyzer": "english",
        "fields": {
          "exact": { "type": "keyword", "normalizer": "lower_exact" }
        }
      }
    }
  }
}

part_no keeps the behaviour that helps prose queries, and part_no.exact holds the whole string as a single term, case-folded so that a-100-b and A-100-B land on the same entry. A keyword field is not analysed at all — it has no tokeniser — so the normalizer is the only mechanism available for case folding, and it is restricted to character filters and a small set of token filters for exactly that reason.

Character filters are the third lever and the least used. They run before the tokeniser and rewrite the raw string, which makes them the only place you can protect a pattern from segmentation rather than repair it afterwards. A pattern_replace filter that turns hyphens inside an alphanumeric run into underscores, or into nothing at all, will get a part number past the UAX #29 rules intact:

"char_filter": {
  "glue_part_numbers": {
    "type": "pattern_replace",
    "pattern": "(?<=[A-Za-z0-9])-(?=[A-Za-z0-9])",
    "replacement": "_"
  }
}

This is a sharp tool and it applies to every value in the field, so a body-text field carrying it will also glue well-known into well_known and break ordinary prose queries. It belongs on fields that are known to hold identifiers, which is the same restriction that applies to everything else in this chapter.

The query then addresses both, with the exact field weighted so that a literal hit cannot be buried under partial ones:

{
  "query": {
    "bool": {
      "should": [
        { "term":  { "part_no.exact": { "value": "a-100-b", "boost": 10 } } },
        { "match": { "part_no":       { "query": "A-100-B" } } }
      ]
    }
  }
}

The boost on the exact clause is doing a specific job, and the job explains why a large constant is the right shape for it rather than a tuned one. A document matching only fragments accumulates score from terms whose document frequencies are in the hundreds of thousands, and a lexical scorer pays little for a common term, so there is a low ceiling on what fragment-only evidence can ever be worth. A document matching the preserved whole string matches a term of frequency one, which is the most any term in this field can be worth. The boost has only to guarantee that the second cannot be overtaken by any accumulation of the first, and since the fragment ceiling is low, a round multiplier clears it comfortably. Tuning it finely would be tuning a margin rather than a ranking.

Proving the term is there, rather than believing it is#

Reading a mapping tells you what you configured. It does not tell you what is in the index, because mappings change, reindexes are partial, and an alias — a name that resolves to an index and can be repointed at a different one without the client noticing — can point somewhere other than where you think. The check that settles it is a term dump: ask the index which terms actually exist for a specific document and field.

curl -s 'localhost:9200/parts/_termvectors/9f31?fields=part_no.exact,part_no' \
  | jq '.term_vectors | map_values(.terms | keys)'

{
  "part_no":       ["100", "b"],
  "part_no.exact": ["a-100-b"]
}

That output is the proof. The exact string is present under one field name and absent under the other, and any query that fails now is a query problem rather than an indexing problem — which is a genuinely useful thing to be able to assert, because it cuts the search space of a debugging session in half.

For a whole-index view rather than a single document, Lucene's CheckIndex and the Luke tool will enumerate the term dictionary directly, and Elasticsearch's _terms_enum API will do a prefix scan over a keyword field without loading documents:

curl -s localhost:9200/parts/_terms_enum -H 'Content-Type: application/json' -d '{
  "field": "part_no.exact", "string": "a-1", "size": 5
}' | jq -r '.terms[]'

a-100-b
a-100-c
a-105-b
a-110-a
a-118-x

Because analysis runs at write time, changing an analyser does nothing to documents already stored. A new mapping applies to new writes only, and Elasticsearch will refuse to change the analyser on an existing field for exactly this reason. Correcting one means creating a second index with the fixed mapping, running _reindex from the old one, and moving an alias across when it finishes — which is a scheduled operation rather than a setting to flip during an incident. The alias switchover is the only instantaneous step in that sequence, and it is instantaneous because it renames a pointer rather than moving data. Everything before it is a full re-read and re-analysis of every stored document, since the whole point is that the terms currently in the index cannot be transformed into the terms you want without going back to the original text — which lives in _source and has never been touched by any query.

Where preservation becomes the wrong default#

The failure described here pushes in one direction, and it is possible to overcorrect. An analyser that preserves everything destroys the recall gains that made that team turn stemming on in the first place: with no stemming, a document saying "the connection timed out" will not match a query for "connection timeout", and with no lowercasing, Refund and refund become unrelated terms with independently computed statistics. The 74 percent hit rate on natural-language queries came from those filters. Turning them off across the board trades one broken query class for another.

The shape of the answer is therefore per-field rather than per-index. Fields that hold prose get an aggressive analyser; fields that hold identifiers, error codes, SKUs, service names and enum values get a keyword field with at most a lowercase normaliser; fields that hold both get the multi-field treatment. Deciding which is which requires knowing what your users actually type, and the cheapest source of that is a week of query logs sorted by frequency, where identifier-shaped queries are usually obvious by the presence of digits, underscores or a leading uppercase run.

There is also a cost nobody mentions until the cluster complains. Keyword fields over high-cardinality identifiers produce a term dictionary entry per distinct value, so a field holding 4.4 million unique chunk hashes adds 4.4 million dictionary entries, and the dictionary is the structure that wants to stay in memory. Preserving originals on a field of UUIDs is a way to spend gigabytes of heap on terms no user will ever type.

The size of that bill is the term count multiplied by what a term costs to hold, and identifiers are the worst case on both factors at once. On the count, a field of unique values has as many distinct terms as the collection has documents, where an English prose field's vocabulary stops growing long before its document count does. On the cost, the compression that makes a term dictionary small works by sharing prefixes between neighbouring terms, and a random identifier shares nothing with the identifier next to it. So the structure that a corpus of English keeps small by both mechanisms, a corpus of identifiers keeps small by neither. That is the reason to ask, before adding a keyword sub-field, whether anybody will ever type a value of it — a term nobody queries is pure cost with no retrieval on the other side of it.

An analyser is the only part of a lexical stack whose output is the whole of what the index knows, so the questions worth asking about it are all of the form "what terms did this produce" rather than "what did I configure". _analyze answers that for a string you are about to index, _termvectors answers it for a document already stored, and _terms_enum answers it for the dictionary as a whole. Between them they turn an argument about whether search is broken into a two-minute check with an unambiguous result.

With the chain settled per field, an index for a parts lookup contains the terms its users actually type, and a query for a part number produces a candidate set containing the part. That leaves the question the walk over postings lists deliberately did not answer: given several hundred documents that all contain the query terms, which one goes first.