Embedded text is searched; payload fields are filtered

Two places exist to put a piece of context about a chunk, and they behave nothing alike. One is the embedded text: the string handed to an embedding model, which is usually the chunk's own words with a header prepended in front of them. The other is the payload: a set of fields filed alongside the vector in the store, under names like doc_title or effective_date. In most pipelines the same strings end up in both, which makes one of them look redundant.

Neither is redundant, and the reason is worth being precise about, because the mistake it produces is silent in both directions. A team that drops the payload because the header already carries the title loses the ability to filter and notices weeks later. A team that adds fields to the payload expecting rankings to improve gets no movement at all and concludes that metadata does not help. Both are reasoning from an assumption that the two destinations pool into one searchable description of the chunk, and they do not pool: each feeds a different half of a query, and neither half can do the other's job.

What each half of a write feeds#

Here is what a write looks like in the terms a vector store sees. The upsert — the write that inserts a record or replaces whatever is already stored under the same id, which is the operation an ingestion pipeline wants so that re-running it does not duplicate anything — takes a header-prefixed string and a set of fields. The two comments in it mark the two edits a real release made in one go, and they are numbered because the rest of this chapter turns on telling them apart:

from qdrant_client import QdrantClient, models

client.upsert(
    collection_name="kb",
    points=[
        models.PointStruct(
            id=chunk.uuid,
            # change 1: this string was embedded
            vector=embed(f"{chunk.header}\n\n{chunk.text}"),
            payload={
                "text": chunk.text,
                # change 2: these fields were added
                "doc_title": "Returns Policy v4",
                "section": "Damaged goods",
                "effective_date": "2025-11-03",
            },
        )
    ],
)

The mechanism is visible in the call itself, and it is worth walking one step at a time rather than taking on faith. embed() is called with a single argument: the header and the chunk text joined together. Its return value is what lands in the vector field. The payload is a separate argument to PointStruct, constructed after embed() has already returned, so no value inside it was available to embed() and none of them could have affected what it produced. A vector is therefore a function of one string. Similarity search compares a query vector against stored vectors and nothing else, so a field that cannot change the vector cannot change a similarity score — however faithfully it is stored, and however obviously relevant it looks to a human reading the record.

That is not a Qdrant quirk — it is the shape of every vector store, whatever the field is called. Pinecone calls it metadata, Weaviate calls them properties, pgvector puts them in the other columns of the row. In all of them, similarity is computed over the vector, and the vector came from the text you handed the embedding model. The payload is a JSON object stored beside the point, a point being this store's word for one stored record — an id, a vector, and the fields filed alongside them — and no part of that payload participates in the distance computation.

So a header is the only one of the two that can move a ranking, and it moves it by the mechanism chapter 1 established: where a chunk's text says "goods that arrive damaged must be reported within 48 hours of delivery" and never says which policy it belongs to, the header supplies the missing subject and the vector moves toward queries that name the policy. Adding doc_title to the payload of that same point changes the vector by nothing at all, because the vector had already been computed from a string the payload was not part of.

The payload buys something the header cannot. A filter is a predicate over the stored fields, evaluated to decide which points are eligible to be scored:

hits = client.query_points(
    collection_name="kb",
    query=embed("what changed in the returns policy"),
    query_filter=models.Filter(
        must=[
            models.FieldCondition(
                key="doc_title", match=models.MatchValue(value="Returns Policy v4")
            ),
            models.FieldCondition(
                key="effective_date", range=models.DatetimeRange(gte="2026-01-01")
            ),
        ]
    ),
    limit=10,
).points

That request says: among points whose doc_title is exactly Returns Policy v4 and whose effective_date is on or after 1 January 2026, return the ten nearest to this vector. The gte is the part that matters. It is an ordering comparison over a typed value, and there is no way to express it in embedded text, because the embedding model has no notion of comparison.

The reason that is a hard limit rather than a tooling gap is visible in the chapter's own example. "Effective 2026-01-14" and "effective 2025-12-30" embed to two points that are near each other precisely because they are both dates in a similar phrasing, which is the opposite of what the query needs. Similarity is computed over how text reads; an ordering comparison is computed over what a value is. Those two are free to disagree, and on dates they routinely do: the two strings above sit a fortnight apart in value and almost on top of each other in phrasing, while "January" and "the fourteenth of the first month" sit at the same value and far apart in phrasing. A mechanism that scores by phrasing cannot be made to answer a question about value by writing the value more clearly.

The reverse is equally strict, and it follows from the same single fact about where the vector comes from. A date sitting in the payload cannot make a chunk more similar to anything, so adding effective_date to every point improves no query's ranking by any amount. If you want the model to read the effective date in the answer, that is a third thing again — it has to be in the text the retriever returns, which may or may not be the text you embedded.

Three destinations, not two#

It helps to stop thinking of the header and the payload as two copies of the same information and start counting the destinations a piece of information can have. There are three, and a given field usually wants two of them.

Read the first two columns against each other rather than down. Every row that says Yes in the first says No in the second, and every row that says No in the first says Yes in the second. There is no row with Yes in both and no row with No in both. That exact complement is the table's whole argument in one shape: the two properties partition the three carriers with nothing overlapping and nothing left over, which means no single place to put a string gives you both. It also settles the redundancy question. Putting doc_title in the header and in the payload is not a copy that could be deduplicated, because the two copies are the only way to obtain two properties that no carrier supplies together.

The header's cell in the middle column is the one that surprises people. "Doc: Returns Policy v4 > Section: Damaged goods" is a string that a human can read a document title out of, and it is not a field. Nothing can filter on it without substring matching against the embedded text, which is neither indexed nor reliable — a header reading Returns Policy v4 > Damaged goods > Exceptions for perishables matches a substring test for "Damaged goods" and a header reading Returns Policy v4 > Section 3 does not, even though both chunks live in the same section. Parsing structured values back out of a display string is the kind of thing that works on the sample you tested and fails on the document with a > in its title.

One qualification on the first column, since most stores blur it slightly. Qdrant, Weaviate, Elasticsearch and pgvector will all build a full-text index over a text payload field if you ask, so a payload field can participate in lexical matching and in a hybrid score. That is worth using — it is how a hybrid index gets a BM25 arm at all — and it does not change the claim. A full-text index over section supports term matching against section; it puts nothing into the vector, so it cannot make a chunk semantically nearer to a paraphrase that shares no terms. Two mechanisms, both real, and only one of them is the similarity computation.

The generator column matters too and is easy to skip. What the model reads is what your prompt-assembly code puts in the prompt, which is whatever you pulled from the payload — usually payload["text"]. If you embedded header + text and stored only text, the header influenced retrieval and then disappeared before the model saw anything, so the model receives twenty-one words about a thirty-day notice period with the subject once again missing. That is a real failure and a common one: retrieval improves, answer quality does not, and the header gets blamed for not working. Store the header too, and decide deliberately whether to prepend it to the returned text.

What a "since January" query actually needs#

Filtering on effective_date >= 2026-01-01 narrows the search to chunks that came into force this year, which is necessary and not sufficient. "What changed" is a comparison between two states of a document, and a filter over one field cannot express a comparison the index does not hold. You need to know which revision each chunk came from, which revision preceded it, and which chunks were removed rather than added — and a chunk that was deleted in the January revision is not in the index at all, so no query over the current index will find it.

The two code blocks above make that concrete when they are read together, which is worth doing because it looks at first like a bug in one of them. The point written by the upsert carries an effective_date of 2025-11-03. The filter demands 2026-01-01 or later. So the filter is perfectly well-formed, and it excludes the very point the write created — correctly, since that section of the policy did not come into force this year. Nothing is wrong with either piece of code. What the pair demonstrates is the limit of filtering as such: a filter selects among the records that exist now, and "what changed" is a question about the difference between two sets of records. No predicate over one field of the current index can reach the earlier state, because the earlier state is not in the index to be selected from.

That is a stored-field problem with a stored-field answer, and it is the subject of the next chapter. What matters here is the diagnosis, and it holds for a release that ships both changes at once: a query of this shape fails identically before and after, because neither change addresses it. A header could not, on principle. A payload could have, if it had carried the right fields, and doc_title / section / effective_date are not them.

Deciding where a field goes#

The decision reduces to two questions asked of each piece of context, and they are independent, so a field can answer yes to both.

Does a user's phrasing of a question contain this information in natural language? "Parental leave", "the returns policy", "damaged goods", and the name of a product all show up in queries as words, so putting them in the embedded text lets the similarity computation use them. Version identifiers, tenant IDs, and ACL group names — an ACL being an access control list, the record the source system keeps of who is permitted to read a given document — do not show up in queries as words — nobody types tenant_id: acme-corp — so embedding them adds tokens that no query will ever match and, per chapter 2, adds shared tokens across every chunk of that tenant.

Will something need to restrict, sort, group or count on this? Dates, versions, sources, tenants, languages, and document types all end up in a WHERE clause eventually, which means they need to be typed fields with the right type. effective_date stored as the string "3 November 2025" is not a date and will not compare; store an ISO-8601 value and let the store index it as a timestamp.

Those two renderings of the date are worth holding side by side, because they carry identical information. "3 November 2025" and "2025-11-03" name the same day, and neither has been abbreviated or rounded. Nothing has been lost in going from one to the other, so the reason the first fails a gte comparison has nothing to do with what it says. It fails on type: a store can compare timestamps because timestamps have an order defined on them, and it cannot compare arbitrary strings that way. That distinction separates a requirement from a decision here. Storing an orderable type is a requirement, forced by what a range predicate is; choosing ISO-8601 in particular over some other orderable encoding is a decision, made because it sorts correctly as text and every store parses it.

Getting the second question right and the first one wrong is the more expensive mistake, because it is silent. A header built by concatenating everything the pipeline knows — tenant:acme-corp | doc:9f31c2a4-77bd-4e1e-9a0e-3d5f | rev:v9 | Returns Policy v4 > Damaged goods | eff:2025-11-03 — is 40 tokens of which about 8 will ever appear in a user's phrasing. Those two figures are both counts of tokens as an embedding model splits them, not counts of words, and the distinction matters here: a UUID looks like one word and costs a model many tokens, so counting the string by eye understates it badly. Taken as a ratio, about a fifth of that header is query-facing and the rest is carried for nothing. The UUID and the revision string are noise to the embedding model, the tenant slug is identical across every chunk that tenant owns, and by chapter 2's argument that shared component pulls the tenant's vectors toward each other for no retrieval benefit at all. The same fields in the payload cost nothing per query, filter precisely, and are typed. A header that reads like a log line is a reliable sign that the two destinations have been collapsed into one.

Applied to a release of this kind, the header should carry the document title and section path — words that appear in queries — and the payload should carry doc_id, revision, section_path as a list, effective_date as a timestamp, source_type, and the tenant and ACL references. Overlap between the two is expected and is not duplication in any sense that costs you: doc_title in both places serves two mechanisms that cannot share.

Storing a field is not enforcing it#

One more thing follows from the same mechanism, and it is worth stating now because it is where the biggest mistakes in this area live. Adding tenant_id to the payload of every chunk makes tenant-scoped search possible. It does not make search tenant-scoped. The field sits in the payload doing nothing until a query supplies a filter that references it, and a query without that filter searches the whole collection and returns whatever is nearest, cheerfully crossing every tenant boundary in the index.

This is the stumble the chapter exists to prevent, and it is tempting for a reason worth naming. Every point in the index carries tenant_id, correctly, on every write, forever — so the field looks like a property of the data and reads like a guarantee. What it cannot do is discriminate, because it holds still while the thing that decides the outcome varies. The field is present identically on a query that filters and on a query that does not; the only thing that differs between a scoped search and an unscoped one is a clause in the request. A defence that is equally present when it works and when it fails is not enforcing anything.

The gap between those two states is one line of code that is easy to omit and produces no error when omitted. A search with no filter is a valid, successful search. Nothing in the vector store knows that tenant_id was meant to be a boundary rather than a facet; to the index it is a string field like section. Whatever enforces the filter has to live in your code — a repository layer that takes the tenant from the request context and refuses to build a query without it, rather than a convention that every caller remembers. Chapter 6 takes apart the version of this that goes further wrong, where the boundary is written into the header as text and something is expected to honour it.

The three-destination table has a limit of its own, and a hybrid index is where it shows. Its first two columns are a clean complement for a search that scores by vector distance alone, and a store that also runs a full-text index over a payload field has given that field a way to affect the score — so for that field the two columns stop being exclusive. The partition still holds for the similarity computation, which is what the chapter's rule is about. Where it stops being a safe shorthand is any sentence of the form "the payload cannot affect ranking", which is true of similarity and false of a hybrid score, and the two are easy to say interchangeably when only one index is in the room.

Everything downstream of a chunk consumes it through one of two channels, and the channels are sealed from each other. What you embed decides what the chunk is near; what you store decides what can be selected, sorted or refused. Words a person would type into a search box belong in the first. Values something will compare, group or enforce on belong in the second. A field that is both — a document title, a section name — belongs in both places, and writing it twice is not waste.

Which raises the question the next chapter has to answer, because "add the fields you will need later" is not actionable until somebody says which fields and why. The forcing case is not a query at all. It is an incident: a user reports that the assistant quoted a data-retention rule that was rescinded in March, and the trace holds the answer and the chunk text and nothing that says where either came from.