Knowledge cutoff

Read these first

• Hallucination

What you will know

• Inspect the provider's model documentation and record the approximate training-data cutoff for the model you use

  The date may be approximate, differ across model revisions, or omit post-training updates, so it is an operating assumption rather than a freshness guarantee.

• Run paired prompts that answer the same question with and without supplied context, then classify which claims came from the context

  A model can blend recalled facts with prompt content, especially when the two sources conflict, making attribution from the final answer unreliable.

• Reproduce a confident answer about a post-cutoff event using a fixed prompt and settings, and capture the unsupported claims

  Plausible continuation and pattern completion can produce fluent falsehoods without any internal signal that the event is newer than the training data.

• Decide whether a given question needs retrieval, a tool, or neither

  The wrong choice either adds avoidable latency and cost or leaves the system exposed to stale, private, computed, or action-dependent facts.

• Compare a base model with one fine-tuned on a small fact set, testing held-out paraphrases and conflicting newer facts

  Fine-tuning may alter memorization in brittle ways, while reliable updates still require a source of truth and a refresh path.

Chapters

1. 1

   Published and effective knowledge cutoffs

   Providers publish an approximate training-data cutoff, and it is worth recording against the exact model revision you call. It can shift between revisions of the same family, it says nothing about post-training, and it is fuzzy at the edges rather than crisp — coverage thins for months before the published date, and thins fastest on narrow subjects. This chapter separates the published date from the effective cutoff you can measure, and gives you a registry and a per-domain probe for both.

   Read chapter
2. 2

   Parametric recall past the cutoff

   Ask about something that happened after the training-data cutoff and the usual result is not a refusal but a fluent, confident answer assembled by pattern completion from what similar events look like. Nothing inside the weights flags the gap, because there is no lookup that can miss. This chapter shows a specimen, explains why no refusal is available, gives the harness for reproducing the failure at fixed settings, and measures the two interventions everyone tries first.

   Read chapter
3. 3

   Knowledge conflict and parametric override

   Retrieval can put today's price directly in the context and the assistant still quotes last year's, with the correct chunk ranked first every single time. Recalled facts compete with supplied ones and sometimes win, most often on the facts a model has read most. This chapter measures the context-wins rate per fact, gives the table of tells that separates this failure from a retrieval miss, and explains why no retrieval metric can see it.

   Read chapter
4. 4

   Paired prompts and attribution

   The obvious diagnostic for a knowledge conflict is a paired prompt: the same question asked with and without the supplied context. It is worth running and it has a ceiling. The model merges recalled and supplied material into one continuation, so the wording never reveals which source a claim came from. This chapter sets out what the pairing genuinely licenses, why citations do not close the gap, and how to report the finding without overclaiming.

   Read chapter
5. 5

   Sentinel values in supplied documents

   Replace the contested fact in a staging copy of the document with a distinctive value the weights cannot possibly hold, re-ask the identical question, and the answer settles what a paired prompt could only hint at. This chapter covers the two kinds of sentinel and the different questions they answer, the ladder that separates parametric override from a model rejecting an absurd value, and how the harness becomes a regression test for a property nothing else in a retrieval system checks.

   Read chapter
6. 6

   Weights, retrieval or tool calls

   The capital of Portugal, your company's current refund window, and this customer's balance right now need three different mechanisms, and a fourth case — issuing the refund — cannot be indexed even in principle. This chapter gives the three properties that decide which mechanism a fact needs, the staleness budget that should accompany any index, the cost of routing everything through retrieval anyway, and the schema and handler that encode the decision.

   Read chapter
7. 7

   Fine-tuning and factual recall

   Training on the company handbook so the model 'just knows it' yields brittle recall that misses held-out paraphrases, returns the trained value when a newer one is supplied, cites nothing, and refreshes only when you retrain. This chapter gives the four-slice evaluation that makes those failures visible before the architecture decision, the structural argument about refresh paths, and the three cases where fine-tuning is the right tool.

   Read chapter