Parametric recall past the cutoff

Chapter 1 established roughly where a model's training data stops and how to record it against a pinned revision. This chapter is about what the model does at that edge, which is the part nobody predicts correctly on first contact. The expectation almost everyone brings is that a question about a post-cutoff event produces some form of "I don't have information about that", the way a database query for a missing row produces an empty result.

What happens instead is an answer — fluent, specific, structured, and wrong in a particular way that makes it hard to catch. It is right about the genre of the answer and wrong about the facts, because genre is what the weights hold in abundance and the facts were never collected. The failure has a knowable trigger, which distinguishes it from confident falsehoods in general and makes it both easier to reason about and easier to test than the broader phenomenon topic 1.11 covered.

Nothing in the weights carries a date#

Here is a specimen, trimmed only where the response repeated itself. The question concerns version 4.0 of a client library, which shipped eleven months after the recorded cutoff of the revision answering it:

> What changed in Atlas SDK 4.0?

Atlas SDK 4.0 is a major release focused on async-first APIs and
stricter typing. The headline changes:

  * The synchronous client is deprecated in favour of AsyncSession.
    Session remains available but emits a DeprecationWarning.
  * Full type annotations across the public surface, with a py.typed
    marker so mypy picks them up without stubs.
  * Retry configuration moved from per-call keyword arguments to a
    RetryPolicy object passed at client construction.
  * Minimum supported Python raised to 3.9.
  * The `stream_events` compatibility shim, deprecated in 3.0, has
    been removed.

Migration is mostly mechanical; the retry configuration change is the
one that requires touching call sites.

Every item on that list is wrong. Version 4.0 was a security release with two CVE fixes and no API changes at all; the async client had shipped in 2.4, the type annotations in 3.1, and RetryPolicy does not exist under any name. What the list is not is random. It is an extremely good guess at what a major version bump of a Python client library in the late 2020s contains, assembled from the several thousand such changelogs the model has read, and it is right about the genre in every particular: the deprecation-with-warning pattern, the py.typed marker, the minimum-Python bump, the closing sentence about which change requires touching call sites.

Getting from that specimen to a rule means saying what a training run actually leaves behind, because the specimen is a consequence of it. Pretraining shows the model an enormous quantity of text with a piece of each passage hidden, asks it to predict the hidden piece, and nudges the weights slightly in whatever direction would have made the true continuation more likely. Two features of that procedure carry everything below. The nudges are individually tiny and there are an enormous number of them, so no single document moves the weights very far. And they land on parameters that every other document also moves — nothing in the model is set aside for Python changelogs, and no cell of it holds the contents of one release note.

What that procedure leaves is therefore not a store of records but a set of tendencies. A fact that appeared ten thousand times has nudged the weights ten thousand times in a consistent direction, and that accumulated tilt is what recall is made of. A fact that appeared once has nudged them once, faintly, in a direction a hundred other documents were pulling away from. A fact that never appeared has nudged them not at all — and the absence of a nudge leaves no mark, because it is the ordinary condition of every parameter with respect to almost every fact in the world.

The reason there is no refusal is structural. A database that has not been told about version 4.0 returns zero rows, because it stores facts as discrete records and can distinguish a record's absence from its presence. Model weights store nothing discrete. What they hold is a compressed statistical account of how text about client libraries, version numbers, changelogs and deprecations tends to continue, and the question above is a perfectly ordinary continuation problem regardless of whether the specific release was ever in the training data. There is no lookup that can miss, so there is nothing to report as missing. Andrej Karpathy's Deep Dive into LLMs like ChatGPT frames this well: the weights are a lossy, compressed recollection of the training data, while the context window is working memory the model can read exactly. Only the second one has a notion of "not present".

Running both cases through the machinery shows how exact that symmetry is. Asked about version 4.0, the model does what it does for every input: it computes a probability for each token that could come next, one of them is chosen, and the whole computation runs again with that token appended. There is no branch in that procedure for insufficient evidence, and there could not be, because a branch needs a quantity to test and no quantity of the required kind is computed anywhere. The model returns a distribution over next tokens for an input it has seen a thousand near-copies of, and it returns a distribution over next tokens for an input unlike anything in its training data. In both cases the distribution is the only thing there is to return.

What does differ between the two cases is how concentrated that distribution is, and the trouble is that concentration is not carried on any channel a reader can see. The text that comes out has been sampled from the distribution and then written in whatever register the training data uses for the genre, and the training data writes about software releases in a confident register. Even where a provider exposes the model's own per-token probabilities, those report how predictable each word was given the words in front of it, which is high for fluent invented prose exactly because it is fluent. A well-formed sentence about a release that never existed is easy to continue.

The contrast with a search index sharpens it further. A search engine that has never crawled a page returns an empty result set, and the emptiness is itself information — you learn that the index does not cover the query, which is a different state from the index covering it and disagreeing with you. A model collapses those two states into one output shape. "I have read a great deal about this and here is the consensus" and "I have read nothing about this and here is what such a thing usually looks like" are rendered in the same register, the same length, the same level of specificity. There is no channel on which the difference could be transmitted, because there is no place in the computation where the difference exists.

Code generation is where this costs the most, and it is worth naming because backend engineers meet it daily. A model that has read four years of documentation for version 2 of a library and nothing about version 3 will produce version 2 calls that import cleanly, type-check under a permissive configuration, and fail at runtime with an attribute error. The failure surfaces late, in a place that looks like your bug, and the generated code carries every superficial marker of correctness — idiomatic naming, sensible error handling, a docstring. Nothing in the artefact says "this API was current eighteen months ago".

What the model does have is a rough sense of how confident tokens usually are in a given neighbourhood, and this is why the failure is not uniform. Asking about a version number one release beyond what it saw produces a smooth, wrong answer. Asking about a version number twelve releases beyond — Atlas SDK 15.0, when the model has only ever seen 3.x — often produces hedging, because the shape of the question is now unusual: nothing in the training data pairs a 3.x-era library with a 15.0 changelog. The hedging is a response to distributional strangeness, not to recency. That distinction matters because it means the cases most likely to reach production without hedging are exactly the plausible ones — the next release, next quarter's pricing, this year's rate rather than last year's.

Stated plainly, the model is reliable about the shape of an answer and unreliable about its contents, and the split follows directly from how the two were learned. Shape is what thousands of documents agree on, so thousands of nudges pushed the weights the same way and the tilt is deep. Contents are what one document carries, so a single faint nudge is all that was ever available, and for anything past the cutoff not even that. A question about a release the model never saw asks for the contents and gets the shape, delivered at the confidence the shape deserves.

Building the artefact#

The argument you will need to win is with someone who believes the model would have said it did not know, and a single screenshot invites the reply that the prompt was leading. What settles it is a reproduction at fixed settings, run enough times to show the answer is the model's default rather than one unlucky draw.

"Fixed settings" needs a note. On older models it meant pinning the sampling parameters, usually temperature 0. That option is disappearing — several current frontier models reject temperature, top_p and top_k outright, so there is no knob to pin, and none of them ever guaranteed identical outputs anyway. What you can pin is the model revision from the registry in chapter 1, the effort or reasoning setting — the dial some providers expose for how much internal deliberation the model does before it starts answering — the system prompt, and the exact user text. Then run the same input n times and record the spread, because the variation across runs is itself part of the finding.

import json
from collections import Counter

FIXED = {
    "model": "atlas-4-1-20250312",          # pinned revision from models.yaml
    "system": "You are a helpful engineering assistant.",
    "effort": "medium",
    "max_tokens": 800,
}

QUESTION = "What changed in Atlas SDK 4.0? List the changes."

RUNS = 20


def reproduce() -> list[str]:
    answers = []
    for i in range(RUNS):
        answers.append(call_model(**FIXED, user=QUESTION))
    return answers


def summarise(answers: list[str]) -> dict:
    declined = sum(
        1 for a in answers
        if "i don't have" in a.lower() or "after my training" in a.lower()
    )
    return {
        "runs": len(answers),
        "declined_or_hedged": declined,
        "answered_confidently": len(answers) - declined,
        # crude but useful: how often the same invented item recurs
        "recurring_claims": Counter(
            line.strip("* ").strip()
            for a in answers for line in a.splitlines()
            if line.strip().startswith("*")
        ).most_common(8),
    }


if __name__ == "__main__":
    print(json.dumps(summarise(reproduce()), indent=2))

Then extract the claims and mark each one, because "the answer was wrong" is much weaker than a per-claim ledger:

The last two rows are the instructive ones. The stream_events removal is a real fact from before the cutoff, attached to the wrong release — recall degrading into plausible placement rather than into absence. And the closing assessment is not a factual claim at all, which means no fact-checking pipeline built on entity matching will flag it, even though it is the sentence a reader is most likely to act on.

The interventions that do not work#

Telling the model its cutoff is the first thing everybody tries, and it is worth measuring rather than assuming, because it partially works in a way that makes things worse. Adding "Your training data ends approximately November 2024. If a question concerns events after that date, say you do not know" to the system prompt moved a forty-question post-cutoff set from three declines to seventeen. On a matched set of forty pre-cutoff questions the model could previously answer correctly, the same instruction produced nine new refusals — the model has no reliable way to date the events in a question either, so it declines on a guess. Trading a silent wrong answer for an unreliable refusal is sometimes worth it and is never a solution.

Those nine refusals are worth following through, because the instinct on reading them is that the model was careless, and it was not — it obeyed. The instruction asks for a conditional: decline when the question concerns an event after a given date. Executing a conditional requires evaluating its condition, and the condition here is the date of the event the question is about. For any event the model never saw, dating it is the same problem as answering about it, attempted with the same machinery, and the machinery is the one whose unreliability motivated the instruction. So the rule gets applied to the only thing the model can assess, which is how recent the question sounds rather than when the event happened. Questions phrased in the register of current news get declined whether or not their answers predate the cutoff, and a post-cutoff question phrased in the register of settled documentation slips through. The instruction cannot be repaired by better wording, because the fault is not in the wording.

Asking the model how sure it is fails for a related reason, and that intervention is cheap enough to be worth measuring before someone proposes it in a design doc. Appending "Then rate your confidence in this answer from 1 to 10" to the same twenty runs produced a mean self-reported confidence of 7.8 on the fabricated changelog and 8.1 on a matched set of questions the model answered correctly. The number is a learned formatting behaviour, produced by the same continuation machinery as the changelog, and it tracks how confident text about this kind of topic usually sounds rather than anything about whether the underlying facts were available. Verbalised confidence is not a probe into the model's state; it is more output.

The sequence explains why the two figures land where they do. By the time the model writes the rating, the changelog it invented is sitting in its own context, and what sits there is specific, well-organised and plausible. A rating produced by continuing that text is a rating of how that text reads, and it reads like a changelog somebody would score highly. The quantity a useful rating would have to consult — whether the release notes for version 4.0 were ever in the training data — is the same quantity that has no representation anywhere in the model, so there is nothing for the number to be a report of. It is a plausible number in the same sense, and by the same mechanism, as RetryPolicy is a plausible type.

There is a further reason the model cannot locate its own edge, and it is worth stating because it removes the last version of the hope that better prompting will do it. The model has no clock. Nothing in the computation that produces a response consults a calendar, and no parameter was set aside to record when training ended. The only way a date reaches the model is as text — in the system prompt, in a retrieved document, or in the user's own message — so a deployment that never injects one is asking a system with no notion of now to work out whether something is in the future. Two services calling the same revision, one stamping today's date into the prompt and one not, will place the same release on opposite sides of the present. Supplying the date is worth doing and it does not close the gap either, because knowing today's date tells the model nothing about when a release it never saw was published.

It is also not fixed by a larger or newer model, since every model has this property with the date moved. And the honest limitation underneath all of it is that you cannot enumerate the affected questions in advance. You know the date, roughly, from the registry; you do not know which incoming user questions concern post-cutoff facts, because dating a question requires knowing the answer. Any system that routes on "is this about something recent?" is making a guess with the same machinery that produced the problem.

One framing is worth avoiding in a design review, because it sends the conversation somewhere unproductive: this is not lying. There is no internal representation of the true answer that the model is concealing. There is a continuation, produced by the same mechanism that produces correct continuations for questions inside the training distribution, and correctness is a property of the training data rather than of the process. A model with no signal for absence cannot report absence, and a model whose fluency is uniform across everything it says cannot make the gap visible in its output.

Which points at the obvious move. If the weights cannot be trusted to hold the current answer and cannot be trusted to admit it, stop asking them: put the current changelog, the current pricing page, the current policy directly into the context window where the model can read it exactly. That is the right instinct, and the next chapter is about what happens when you do it and the model quotes last year's number anyway.