Silently picking one version is a policy nobody wrote down

Detecting that two documents disagree produces a flag, and a flag is not an answer. Somewhere between the retriever finding both passages and the user reading a sentence, one of the two values wins, and the question this chapter is about is where that decision lives. In most systems it lives nowhere anybody chose: both passages go into the prompt, the generator writes something, and the rule that produced it is whatever the model does with two conflicting statements in its context. That is still a rule. It is simply one nobody wrote, nobody reviewed, and nobody can look up afterwards.

Conflict resolution, as the term is used here, is a step that sits between retrieval and generation, takes the retrieved passages plus any conflict flags, and returns a modified set of passages together with a record of what it did and why. It is a small amount of code. What makes it worth a chapter is that the alternative is not the absence of a policy but an invisible one, and the two differ most sharply in a case where the answer controls access to production data — a contractor asking whether he may reach a production database, with a 2022 onboarding policy and a 2024 security standard both in the index and both saying different things.

The ordering of two chunks as the deciding input#

When a detected conflict is left unresolved, the two passages go into the prompt in rank order and the generator resolves the disagreement on its own terms. Running the same contractor query a hundred times against the same index made the shape of that resolution visible: with the 2022 passage placed first, the answer permitted access 68 times out of 100; with the passages swapped, it permitted access 29 times out of 100. The dominant input to a production access decision was the ordering of two chunks in a prompt, which is a fact about vector arithmetic and nothing else. Changing the generator from one model to a newer one in the same family shifted those figures to 54 and 22 without a single line of application code changing.

Read those numbers as gaps rather than as levels, because neither level means much on its own. Between the two runs the query is the same, the index is the same, the two passages are the same, the model is the same and the prompt is the same except for which chunk is printed first. Sixty-eight permits against twenty-nine is therefore not two measurements of a model's opinion about contractor access. It is one measurement of how much of the outcome the ordering owns, and the answer is a gap of roughly forty points on a scale of a hundred.

The model swap is a second, independent term, and it is read the same way. The newer model took the two figures to fifty-four and twenty-two, so both orderings became more restrictive while the gap between them narrowed from roughly forty points to roughly thirty. That leaves two claims licensed and one not. The gap surviving a model change, and remaining larger than the shift in either individual level — fourteen points on the permissive side, seven on the restrictive — is what licenses calling ordering the dominant input rather than a quirk of one checkpoint. The levels moving together is what licenses calling the model a second input that shifts the whole response surface, so that a routine model upgrade silently re-tunes an access-control outcome. What is not licensed is any claim about which of the two orderings is right, because nothing in the experiment consulted the standard.

The causal chain underneath that is short and each link is checkable. Two passages disagree, so the prompt contains two incompatible statements. A generator produces a distribution over continuations rather than a lookup, so both statements remain available to it. Position in the context is one of the features that distribution depends on, so moving a passage from second to first changes the distribution. Sampling from a changed distribution changes the outcome frequency. Nothing in that chain is unusual or a defect; it is what a generator is. The defect is asking it to arbitrate.

What "let the model decide" actually commits you to#

An instruction like if sources disagree, prefer the more recent and explain which one you trusted feels like a policy, and reads like one in a prompt review. It has three properties that a policy does not have. It cannot be stated in advance in a form anyone can check, because the only description of it is the model's behaviour on inputs you have not seen. It varies between runs, so two users asking the same thing get different rules applied to them and neither can tell. And it cannot be reconstructed after an incident, because the artefact that produced it — the model's weights interacting with a particular token ordering — is not something you can point at in a post-incident review and say this is what fired and this is why.

Naming the stumble here matters more than restating the three properties, because the belief they contradict is one a careful engineer holds for a good reason. That sentence is written in the imperative, it names a rule, it names a tie-break, and it even demands an explanation from the system — everything that makes a written policy recognisable as one is present in it. What is absent is the part that makes a policy useful, which is a mechanism that binds behaviour to the words. The sentence is an instruction to a component that is free to weigh it against everything else in its context, and the numbers in the previous section are what that freedom looks like when measured.

Each of the three properties is a consequence of that, and each fails for its own reason. Statability fails because the rule's definition is a function of inputs, so writing it down means enumerating inputs rather than stating a condition. Stability fails because sampling is stochastic, so the same input gives different outputs and neither user can detect that a different rule reached them. Reconstructability fails because the only artefact retained after the fact is the answer text, and an answer that cites the 2022 policy is compatible both with the model having applied the recency instruction and lost, and with it never having applied the instruction at all.

The comparison worth holding is to a feature flag. Nobody would ship access control by writing a comment asking the service to be careful; the rule lives in a function, its inputs are logged, and when it produces the wrong outcome you can read the function and say what it did. A conflict resolution rule is the same category of object, and the reason it usually is not written down is that the conflict was never surfaced as an event in the first place — the pipeline had no place to put the decision because it did not know a decision was being made.

Once the detection from the previous chapter exists, there is a place to put it. The resolution runs between retrieval and generation, on the retrieved set plus the conflict flags, and it produces a modified context and a record.

Walking that diagram is worth the minute it takes, because the shape of it is the argument. A retrieved set arrives and meets a test — do two of these passages make claims on the same key — and a set that fails the test goes straight to generation unchanged, which is the ordinary path and the common one. A set that passes goes to a lookup, and what it is looked up by is the document class rather than the query or the content, which is the decision the next section is about. The lookup selects one of four branches, each of which produces a different modified context: two of them drop passages, one keeps everything and changes how the answer is templated, and one produces no answer at all.

Every one of those four paths reaches the conflict log, including the refusal, and that is the property that makes the whole arrangement reviewable rather than merely different. The branch that fires and the passages it dropped go into the log alongside the answer. When the contractor case comes back six months later as an audit question, the answer to why did the system say yes is a row, not an investigation.

Four policies, and the conditions each one fits#

The policies are not ranked. Each is correct for a particular relationship between the conflicting documents and a particular cost structure on being wrong. Each is also keyed to the document class, meaning the kind of thing a page is — a security standard, a runbook, a set of meeting notes — recorded as a field at ingest and carried on every chunk, which is the unit almost every rule in this topic is configured against.

The table is easier to use if the four rows are read as answers to two prior questions rather than as four options in a list. The first question is whether the two documents stand in a relationship that ranks them: a class hierarchy ranks them, which is what authoritative source consumes, and a revision line ranks them, which is what newest version consumes. If neither relationship holds, no rule can pick a winner without inventing one, and the two remaining rows are the two honest responses to that — hand the choice to the user, or hand it to a human who owns the documents. The second question decides between those two, and it is a question about the reader rather than about the documents.

Precedence has a subtlety that the table compresses. Authoritative source is a claim about document classes, not about individual documents, so it needs a class taxonomy that already exists in the index — security_standard above team_wiki above meeting_notes — and it degrades badly when a corpus has thirty overlapping classes with no agreed order. On a corpus with four or five well-understood classes it is the cheapest policy to operate, because the ordering is a config file that a compliance owner can read and sign.

CONFLICT_POLICY = {
    # (doc_class_a, doc_class_b) or a single class for intra-class conflicts
    "security_standard": "escalate",
    "access_control":    "surface_both",
    "api_reference":     "newest_version",
    "runbook":           "newest_version",
    "hr_handbook":       "authoritative_source",
    "_default":          "surface_both",
}

CLASS_PRECEDENCE = ["security_standard", "hr_handbook", "policy",
                    "runbook", "team_wiki", "meeting_notes"]

def resolve(conflict, passages):
    classes = {p.doc_class for p in passages}
    policy = next((CONFLICT_POLICY[c] for c in CLASS_PRECEDENCE
                   if c in classes and c in CONFLICT_POLICY),
                  CONFLICT_POLICY["_default"])
    if policy == "authoritative_source":
        best = min(passages, key=lambda p: CLASS_PRECEDENCE.index(p.doc_class))
        kept = [p for p in passages if p.doc_class == best.doc_class]
    elif policy == "newest_version":
        kept = [max(passages, key=lambda p: p.effective_from)]
    else:                       # surface_both and escalate keep everything
        kept = passages
    return Resolution(policy=policy, kept=kept, dropped=
                      [p for p in passages if p not in kept],
                      conflict_key=conflict["key"])

Reading resolve closely is worth doing, because it makes one design decision that the prose around it does not announce. Selecting the policy is a walk down CLASS_PRECEDENCE, taking the first class that is both present among the retrieved passages and has an entry in CONFLICT_POLICY. So the policy applied is the one belonging to the highest-precedence class in the retrieved set, not the one belonging to the class the conflict is about. A conflict between two access-control pages that happens to retrieve alongside a security standard is resolved by the security standard's rule, escalate, and the user gets a refusal rather than two values. That is defensible — a security standard in the candidate set is a reason for caution — but it is a decision, and it means the same conflict resolves differently depending on what else the retriever returned.

The precedence list is doing two jobs at once, which is the other thing to notice. It orders classes for the purpose of selecting a policy, in the next expression, and it orders passages for the purpose of choosing a winner, in the min call inside the authoritative_source branch. Those are separable concerns that happen to share a list here, and they come apart the moment a class belongs in one ordering and not the other.

Two entries in CONFLICT_POLICY cannot be reached by that walk at all. access_control and api_reference have policies but do not appear in CLASS_PRECEDENCE, so a conflict confined to either class finds no match in the walk and falls through to _default. For access_control the outcome is right by coincidence, since _default is also surface_both. For api_reference it is wrong: the table and the config both say a conflict between two revisions of an API reference should resolve by newest_version, and this code will surface both instead. The comment above the dictionary points at the same gap from the other side, describing a key shape — a pair of classes — that neither the dictionary nor resolve ever constructs. Adding the two missing classes to CLASS_PRECEDENCE is the fix, and the reason to raise it here rather than quietly is that the failure is silent: _default always matches, so nothing raises, nothing logs an anomaly, and the conflict log records surface_both as though it had been chosen.

Choosing the policy is a design-time act performed with the people who own the documents, and it is worth insisting on that rather than letting an engineer pick defaults during implementation. The question put to a security owner is concrete — when your standard disagrees with a team wiki, do you want the wiki suppressed, or do you want the user to see both and come to you? — and owners answer it quickly and consistently, because it is the same question they already answer for humans who find two conflicting pages. What they cannot answer is a question phrased as prompt wording. The output of that conversation is the twelve-line config above, reviewed in the same pull request as any other access-relevant change, and that reviewability is most of the value; the runtime cost is a dictionary lookup and a list comprehension, roughly 0.3 ms against a retrieval path that already spends 40 to 90 ms.

That last comparison is a gap and not a level, and it settles one argument without touching another. A few tenths of a millisecond against a path already spending tens of milliseconds is well under one percent of the budget even when the path is at its fastest, so the objection that resolution costs latency is not available. What the comparison says nothing about is the cost that actually bites, which is maintaining the class taxonomy the lookup depends on: labelling every document with a class, keeping the labels right as documents move, and agreeing an order over the classes. Those are people-shaped costs, they do not appear in a latency budget, and they are the reason this layer is sometimes not built.

The template that renders a surface_both resolution is doing real work and deserves to be written rather than left to the generator. It states each value, the document it came from, that document's effective date and scope, and the fact that the two disagree, in that order, and it ends by naming who owns the discrepancy. What it must not do is average, hedge, or produce a sentence like "generally around 8 to 12 hours", which is how a generator handed two numbers and an instruction to be helpful will frequently reconcile them. An answer that invents a range from two point claims is worse than either claim, because it is now unattributable to any source.

Truth, corpus hygiene, and the supersession relation#

A conflict policy does not adjudicate truth. newest_version does not know that the newer document is right; it encodes a belief about how a particular class of documents evolves, and when that belief is wrong the policy is confidently wrong in a stated way, which is an improvement over being confidently wrong in an unstated way but is not the same as being right. Nothing in the resolution layer will tell you that both the 2022 policy and the 2024 standard are out of date because the actual rule changed in a Slack thread that was never written down.

It also does not clean the corpus, and treating it as though it does is the most common way this layer becomes permanent. Every conflict that fires is a defect in the documents, and the log is a work queue for the people who own them. A team that ships resolution without wiring the log to owners ends up with a system that gracefully handles four hundred contradictions forever, at the cost of two extra retrieval-time hops and a permanently degraded answer for every one of them. On the corpus in the previous chapter, 178 conflicts were flagged at the 0.95 candidate stage, of which 61 were genuine; routing those to owners closed 44 within six weeks, which removed the conflict from the product rather than managing it.

Those three figures close only part of the way, and the part that does not close is the interesting one. Roughly a third of what the candidate stage flagged turned out to be genuine, which is the precision of the detector rather than a property of this layer. Of the genuine ones, a little under three quarters closed inside six weeks, leaving seventeen that did not — and nothing in the figures says whether those seventeen were disputed, unowned, or simply not looked at. The remediation figure is therefore evidence that routing conflicts to owners closes most of them and is not evidence that it closes all of them, so a plan that assumes the queue drains to zero is assuming something the numbers do not show.

The policies themselves also assume something they have not been given. newest_version needs to know which of two documents is the newer revision of the same thing, and the resolution code above reads p.effective_from as if that field is populated and means what it says. In the SSO case the handbook and the runbook are different documents that happen to overlap, not two revisions of one document, so newest_version there would be comparing publication dates of unrelated artefacts. The max call would still return something, and what it returns would still be logged as a resolution, which is the shape of failure this whole layer is supposed to remove.

A resolution layer does not make a contradicted corpus correct, and it is not built for that. What it does is convert an event that currently happens inside a model into an event that happens inside a function: a branch with a name, an input that can be read, an output that can be diffed, and a row that survives the request. Which branch is right is a question about who is harmed by each error and whether they can detect it, and it is answered once, with the people who own the documents, rather than differently on every run.

Getting the supersession relation right is a different problem from detecting the conflict, and it shows up first in a form where the conflict never fires at all — a document that was replaced years ago, still in the index, still ranking first.