Scoring the next piece of text

Something you have already done many times is call a function that somebody else deployed as a service: an application assembles an input, sends it over the network, and a result comes back with fields in it. A hosted language model is reached in exactly that way. There is an endpoint, there is a payload, there is a response, and for the first few minutes of working with one, that framing carries you further than you would expect.

It stops carrying you at the point where you ask where the behaviour came from. A function whose behaviour a programmer wrote is a set of explicit branches: somebody decided that an expired session returns null rather than a user id, and that decision is legible in the source, attributable to a person, and changeable by editing a line. A language model has no line to edit. Its behaviour was produced by a separate, automated process run over an enormous body of existing text before anybody deployed it, and nobody involved wrote a rule for any particular answer — not for the answer you are about to get, and not for the answer anybody else got yesterday.

That difference is easy to nod along to and hard to actually hold, so here is where it first bites. An engineer is building a small internal tool that annotates pull requests as they open. The tool sends a hosted model the diff for a pull request together with an instruction to title it.

Here is the diff for a pull request. Give me a one-line title for it.

--- a/api/session.py
+++ b/api/session.py
@@ -41,7 +41,9 @@ def current_user(session):
-    return session.user.id
+    if session is None or session.expired:
+        return None
+    return session.user.id

Back from that request comes a title that is exactly what was wanted.

Fix null pointer when session is expired

The tool then retries the identical request — same model, same diff, same instruction, nothing touched in between and no configuration changed anywhere — and gets something else.

Return None instead of crashing on an expired session

Both titles describe the change accurately and either would be fine sitting on the pull request. To an engineer who has spent a career with functions that return the same output for the same input, a service that does not is a service with a fault in it, and the reasonable next move is to file a support ticket or go hunting for whatever is corrupting the payload in transit. Neither search will turn anything up, because nothing is broken, and saying precisely why nothing is broken is what the vocabulary below is for.

What's fixed once, and what happens on every call#

Here is the vocabulary. Six pieces stand between that instruction going out and a title coming back, and they divide cleanly by when they happen.

Two of them are settled long before anyone sends anything. Weights are the fixed numerical content of the model, and training is the process that produced them. Both were finished before the model was ever made available, and neither moves afterwards: the ten-thousandth request of an afternoon meets exactly the weights that the first one met, and a request that arrives next year will meet the same ones again.

The next two happen fresh on every single call and leave nothing behind. Inference is what a provider's infrastructure does when a request lands, and next-token prediction is what inference does at each step of assembling an answer. Once the response has gone back over the network, neither has left any trace anywhere.

The last two are the only part of the arrangement a caller touches directly. The API request is the message that goes out, and the API response is the object that comes back. Everything an engineer controls and everything an engineer observes lives inside one of those two objects.

Sorting the six by timing is one view of them; the diagram shows another. The dependency between them runs in one direction only. Weights are just numbers until you know what set their particular values, which is training; training has nothing to work on unless there are weights to adjust; inference is the act of running weights forward and has no meaning before them; next-token prediction is a single step of inference; and the request and the response are the envelope wrapped around all of it, so the arrangement has to exist before there is anything to wrap.

Weights#

Among those six pieces, the first two — weights and training — settle before any request exists. That fixed half is where any answer becomes possible at all, so it is where the vocabulary starts, and it starts with the weights themselves.

Weights are numbers. Not a metaphor for numbers, not a data structure with numbers in it — a very large collection of individual numerical values, sitting in memory on the provider's hardware, unchanged between one request and the next. What those numbers happen to be is the entire difference between a program that produces fluent pull-request titles and a program that produces well-formed nonsense. The arithmetic that consumes them is identical in both cases, the code is identical, the hardware is identical; only the values differ.

Take one of them on its own. A single weight can be thought of as a knob controlling how strongly one piece of the model's computation pushes toward one candidate token rather than another whenever a particular pattern shows up in the input. Turn that number up and the push gets stronger, so the token it favours climbs relative to its rivals whenever that pattern appears; turn it down and the token drops. One knob moved slightly will not change any answer you would notice. Move enough of them and what the model tends to produce changes, because every candidate token at every position is being scored by the combined effect of all of them at once.

That is enough to see what varying them does. Hold the request completely fixed — the same pull-request diff, the same instruction to title it, no setting touched — and suppose only the weights were different, pushed toward terse imperative phrasing where the earlier ones pushed toward descriptive sentences. The titles come back systematically differently.

Guard expired sessions
Null-check session before user lookup

Nothing about the request explains that shift, because nothing about the request moved. This case is one half of a pair: it holds the request fixed and varies the weights. The other half — where the weights hold still and the request varies — is worked through under next-token prediction below, and the two halves together are what separate a property of the model from a property of the call.

So: weights are the numerical parameters of a language model, set during training and left unchanged by anything a request does.

The size of the collection is the first thing that goes wrong in people's heads. There are billions of these values in a model of the kind a provider hosts, and nobody assigned any of them by hand. Nobody can point at one and say what it means, either — a weight has no interpretable job title, and asking which one holds the model's knowledge of Python is a question with no answer, because the behaviour is the aggregate of all of them, fixed at the moment training stopped, and never any individual value.

Training#

If the weights are already fixed by the time any request exists, their particular values arrived from somewhere earlier, and that somewhere is training. Weights and training are the fixed half taken together — the weights are the artefact, training is the event that produced it.

The process is repetitive to the point of monotony, and it is a loop.

Nothing in that loop is clever. What it accumulates is.

The contrast between the two ends of that process is the whole content of it. Before any adjustment has happened, the weights are arbitrary values, and the model's guesses about what comes next are correspondingly arbitrary. Ask both ends of the process for the first word of a title for a pull-request diff, and the same position produces two very different orderings.

The untrained row has no ordering because nothing in arbitrary values has any reason to prefer one candidate over another. The trained row has one, and nobody wrote a rule saying titles start with imperative verbs. That ordering is the residue of a very large number of small corrections, each one nudging the weights toward whatever the real text did next.

So: training is the process that repeatedly adjusts a model's weights against a large text corpus until its next-token predictions match observed continuations.

Training is the expensive part, and it is offline. It runs on enormous amounts of hardware for a long time, it happens rarely, and it happens against a body of text that was collected up to some point and then stopped. That last detail has a consequence engineers meet constantly: a model carries a fixed sense of how far the world had got when its text was gathered, and everything after that is simply absent from the values. The name for where the text stops is the knowledge cutoff, and the name for what training baked into the weights is parametric knowledge; topic 1.13 owns both and works out what each means and what to do about it.

Inference#

Training finished long before any particular person showed up with any particular request. Answering one live call therefore has to be a different act from training, and the question a reader is now holding is what actually happens when the pull-request-titling request lands, given that the weights it will use are already fixed and were fixed by a process that has already ended.

What happens is unglamorous: the fixed weights are run forward on the new input to produce output. The provider loads the request's text, pushes it through the arithmetic that the weights parameterise, and text comes out the other side. Training and inference both consist of running text through the same weights-shaped computation, which is why they are so easily confused; the difference between them is direction. Training reads what actually came next in real text and writes the weights accordingly. Inference only ever reads.

For the running case that means: the fixed weights process this specific diff and this specific instruction, and produce a title. Since the weights are not written to, "the model learned something between the first call and the second" is already ruled out as an explanation for two calls diverging, whatever the eventual explanation turns out to be.

So: inference is the process of running a model's fixed weights forward on a new input to produce output, without changing the weights.

"Running the weights forward" is still too coarse to be useful, though. A title is a specific sequence of specific words in a specific order, and nothing said so far explains how any individual word in it got chosen rather than a different one. That happens a level down.

Next-token prediction#

To understand what happens a level down, look at next-token prediction: it is not a separate mechanism sitting alongside inference. It is what inference consists of, performed once per position in the output and then performed again.

At every position, the model takes everything in front of it — the request's text plus however much of the answer has already been produced — and scores every token in its vocabulary as a candidate for what comes next. Every one of them, including the absurd ones, receives a score. Then one token is chosen, appended to what came before, and the whole scoring runs again on the now slightly longer text. A forty-character title is that loop turning over and over until the model scores an end-of-message token highly enough to stop.

Watch that loop turn on the running request, partway through its answer. The instruction and the diff are in front of the model, and the words Fix null pointer when session is have already been produced. The scoring at the next position looks like this.

so far:  Fix null pointer when session is
highest: expired
then:    invalid
then:    missing

All three are grammatical, all three are plausible English, and expired outranks the others because of what this particular diff contains: the added branch tests session.expired by name. Now change exactly one thing. Keep the instruction word for word, keep the weights untouched — no provider could change them mid-afternoon anyway — and put a different diff underneath.

--- a/api/config.py
+++ b/api/config.py
@@ -12,6 +12,8 @@ def load_config(path):
-    return json.load(open(path))
+    if not os.path.exists(path):
+        return DEFAULTS
+    return json.load(open(path))

With that new diff in front of it, the model reaches a structurally identical position in a structurally identical sentence, and the ordering has moved.

so far:  Fix null pointer when config file is
highest: missing
then:    absent
then:    expired

missing has come to the top and expired has dropped to third, with nothing in the weights having moved by so much as one value. That is the other half of the pair opened under weights: there, the request was held and the weights varied; here, the weights are held and the request varies. Between them they locate every influence on an answer in one place or the other.

So: next-token prediction is the computation a language model performs at each step, scoring every possible next token given the text so far.

That definition is also the resolution of the two-different-titles puzzle, and the resolution is mechanical rather than mysterious. Scoring produces a distribution over candidates — a whole ranked field, expired above invalid above missing and so on down through the vocabulary. A distribution is not an answer. Something still has to reach into that field and take exactly one token out of it, and that something is a separate mechanism with its own name and its own settings: sampling, which topic 1.6 owns. Sampling is free to take a token other than the top-scoring one, and if it does so even once, at any position in the title, everything scored after that point is being scored against different preceding text. One different word early on is enough to send the rest of the sentence somewhere else entirely, which is how two runs of one request arrive at two titles that share almost no words. The broader pattern that behaviour belongs to is non-determinism, and topic 1.17 covers where else it shows up.

Two other explanations are worth shutting off, because both are the sort of thing a careful engineer reaches for when one request returns two answers with no configuration touched in between. The first is that the provider quietly handed the second call to a different model. It did not: the request names a specific model by name, and the provider runs the model it was asked for, so a difference in the answer is not evidence of a difference in the model. The second is that the second call somehow read the request differently. It did not: turning the request's text into tokens is a fixed procedure with no random element anywhere in it, and the same input text becomes the same tokens on every call, forever. Both halves of the input to the second call — the weights and the tokenised request — were bit-for-bit what the first call got. The divergence enters at the step where one token is picked out of the scores, and at no point before it.

There is a harder limit on all of this, and it sits here because it is a property of scoring itself. Scoring a plausible continuation and having a basis for that continuation being true are two different operations, and next-token prediction only ever performs the first. Nothing in the loop checks anything. Send the same hosted model a factual question it has no real basis for answering, and it will score a plausible continuation just as readily as it scored expired.

Which regulation governs how long a payment processor must retain
transaction records?

Retention is governed by the Payment Records Retention Regulation 2019,
regulation 14(3), which requires processors to keep transaction records
for seven years from the date of settlement.

That answer is the limit at work: the instrument does not exist. The number, the year, the sub-paragraph and the retention period were all composed the same way expired was composed: as tokens that score well as a continuation of a question phrased like that one. The behaviour has a name — hallucination — and topic 1.11 covers its shapes and what reduces it.

Two readings of that answer would leave it looking like bad luck rather than ordinary behaviour, and both are wrong. It is not a stale record: the model did not retrieve a regulation that once existed and has since been repealed, and the way to establish that is to look the citation up in the register that would hold it, where an instrument of that title and number is absent altogether rather than present and superseded. And it is not a badly phrased question: the question above is perfectly clear, and rewording it produces another fluent, specific, differently-numbered answer, because the mechanism that produced the first one is the mechanism that will produce the second. What the confident register of the answer is worth as evidence, in either direction, is nothing. Fluency is what the scoring produces for true and false continuations alike, so a hedged answer is not more likely to be wrong and a crisp one is not more likely to be right.

API request#

Everything above happens inside a provider's infrastructure, on their hardware, with their copy of the weights. A caller does not run inference. A caller sends a message across a network to a machine that runs inference on their behalf and sends something back, and that message is the first piece of this whole arrangement that a working engineer actually types.

An API request is the message an application sends to a hosted model, containing the input text and any generation settings. For the pull-request titling tool, shaped for Anthropic's Messages API, it looks like this.

{
  "model": "claude-sonnet-5",
  "max_tokens": 64,
  "messages": [
    {
      "role": "user",
      "content": "Here is the diff for a pull request. Give me a one-line title for it.\n\n--- a/api/session.py\n+++ b/api/session.py\n@@ -41,7 +41,9 @@ def current_user(session):\n-    return session.user.id\n+    if session is None or session.expired:\n+        return None\n+    return session.user.id"
    }
  ]
}

Three things are in there: a model named explicitly, a messages array carrying the diff and the instruction as one user turn, and a max_tokens setting putting a ceiling on how much may be generated. Nothing else.

That object, together with the fixed weights, is the entire input to the computation. There is no third source, since inference is running weights forward on an input and this object is the input. The corollary is worth stating now because a great deal later depends on it: if this is genuinely everything the model gets, then anything not sitting inside this object or inside the weights themselves is not available to the model at all, no matter how the instruction is worded and no matter how confidently the answer that comes back implies otherwise.

API response#

What comes back for that request is the other half of the same HTTP call, and it is a JSON object in the same way the request was.

An API response is the object a hosted model returns for a request, bundling the generated content with metadata about how and why generation stopped.

{
  "id": "msg_01Xh9vK2mQpR7sTnBw4Lz3Yd",
  "type": "message",
  "role": "assistant",
  "model": "claude-sonnet-5",
  "content": [
    { "type": "text", "text": "Fix null pointer when session is expired" }
  ],
  "stop_reason": "end_turn",
  "stop_sequence": null,
  "usage": { "input_tokens": 118, "output_tokens": 11 }
}

Of everything the response bundles, the part the caller actually asked for is the title, one string inside a content block, and it is the field almost every first integration reads and the only field most of them read. Two others are sitting right beside it. stop_reason reports how generation ended, and topic 1.8 works through its values and what each one obliges a caller to do. usage reports how much text went in and came out, which is the quantity a provider bills on, and topic 1.3 covers reading and forecasting it.

Reading only the content is a cheap habit to acquire and an expensive one to keep, which the same request demonstrates with one setting altered. The model, the instruction and the diff are all exactly as above; max_tokens is now 4 rather than 64, a cap low enough that the model will be cut off mid-title.

{ "max_tokens": 4 }

{
  "id": "msg_01Qw8bN5rTxV2kLmJd6Ha9Pc",
  "type": "message",
  "role": "assistant",
  "model": "claude-sonnet-5",
  "content": [
    { "type": "text", "text": "Fix null pointer" }
  ],
  "stop_reason": "max_tokens",
  "stop_sequence": null,
  "usage": { "input_tokens": 118, "output_tokens": 4 }
}

Read the content field of that capped response on its own and there is nothing alarming about it. Fix null pointer is a terse, grammatical, entirely plausible pull-request title, and it looks exactly like the sort of thing a model might produce if it were being unhelpfully brief. Held against the completed title, the generated text in each reads as ordinary language and gives no sign by itself of which one ran out of room. Where the two differ is not in how the text reads but in the metadata around it. That is the whole point of the pair: an API response carries information beyond its content, so a caller who reads the content alone is not reading everything the response is reporting about the call, and will confidently diagnose a truncation as a quality problem.

What the request has to carry, every single time#

That assumption — that each call arrives independent, carrying its own input and returning its own metadata — runs deeper into how models work than it looks, and two established facts collide inside it. Nothing about a call changes the weights, and the request plus the weights is the entire input to the computation. Put them together and something follows that surprises almost everybody the first time: whatever a hosted model appears to remember from one call to the next cannot be sitting inside the model, because there is nowhere inside the model for it to sit.

The clearest place to see that is a different shape of product from the titling tool, built out of the same request and response pieces. A chat-style coding assistant runs a long conversation with one engineer: dozens of turns, each one apparently picking up where the last left off, the assistant referring back to code discussed twenty minutes earlier. Here is what the first turn sends.

{
  "model": "claude-sonnet-5",
  "max_tokens": 1024,
  "messages": [
    { "role": "user", "content": "How do I read a file in Rust?" }
  ]
}

And here is what the thirtieth turn sends.

{
  "model": "claude-sonnet-5",
  "max_tokens": 1024,
  "messages": [
    { "role": "user", "content": "How do I read a file in Rust?" },
    { "role": "assistant", "content": "Use std::fs::read_to_string, which..." },
    { "role": "user", "content": "What if the file is too big for memory?" },
    { "role": "assistant", "content": "Read it in chunks with BufReader..." },
    "…twenty-five further turns, alternating in the same way…",
    { "role": "user", "content": "Can you make that generic over Read?" }
  ]
}

Same endpoint, same model, same shape of object. The only difference is how much is in the array. The application kept every turn of the conversation and sends all of them again on each new turn, because the alternative is an assistant that has no idea what was just said.

The billing arithmetic follows immediately. usage.input_tokens counts everything the request carried, and every turn resends the entire transcript so far, so the input grows with the conversation even when no individual message got any longer. Turn thirty is carrying twenty-nine turns of history plus one new question, and it costs accordingly, with nothing about the engineer's own messages having changed at all.

Two other accounts of that cost curve are the ones people reach for, and both are worth displacing explicitly. The provider is not storing the conversation and billing for an open session: each turn is a separate, self-contained HTTP request, priced on what that particular request carries, and in the gap between two turns there is nothing on the provider's side to charge for. And there is no per-conversation setting that quietly makes the model work harder or think longer as a conversation lengthens: what grew was the quantity of text the request carried, not the effort applied to it. The name for the resent transcript is conversation history, and the name for the property that forces it is statelessness; topic 2.3 owns both and works through what they cost and how to manage them.

That mechanism settles a case most engineers meet directly, usually from a colleague. "I corrected the model twice in yesterday's chat, so it should get this right today." It will not — a correction changes no weights, whatever form it took — and the part worth explaining is why it nevertheless appeared to work yesterday.

Because the colleague did observe something real: within yesterday's chat, the correction did stick, and the model did stop making the mistake. That was never the model remembering. It was the application appending the correction to the transcript and resending it as part of every subsequent request, so the model was reading the correction fresh each time as ordinary input text. Open a new chat today and no such resending happens, so the correction is simply not present anywhere in front of the model.

What would actually make a correction stick has three answers, and they differ in cost and in blast radius rather than in correctness. Include it in the request every time it matters — immediate, requires no infrastructure, and pays for itself on every single call in both tokens and discipline, since forgetting once means the behaviour is gone for that call. Have the application store it and resend it, as part of history or as a standing instruction that goes out with every request — this scales to many sessions and many users, and it is application engineering, a thing you build and version and test, rather than a property of the model. Or change the weights themselves, which is a training project with a budget and a schedule attached, not a setting anybody flips per request.

Three things a model cannot know unless the request supplies them#

Weights that only a training run can change are half of a larger fact: a request plus those fixed weights is the whole input, and that fact explains more than a conversation's cost curve. It also draws a hard boundary around what a model can possibly answer well. The boundary has three edges, and they are worth naming separately because each one is missed in a different way and each is repaired by different work.

The three edges

The first edge is the knowledge cutoff, already met as the model's fixed sense of how far the world had got. What the boundary adds is that no wording of a request talks the weights into holding a library released after the text was gathered; the fact itself has to be copied in. Topic 1.13 covers how to work out where a model's cutoff sits and what changes around it.

The second edge is private text, and the way it is usually missed deserves a worked case. A team wants a hosted model to answer questions about their internal engineering handbook — a document that lives in their own repository and was certainly never part of anybody's training text. The instinctive first attempt is to say so in the instructions.

{
  "model": "claude-sonnet-5",
  "max_tokens": 512,
  "system": "You have read the company engineering handbook and can answer questions about it.",
  "messages": [
    {
      "role": "user",
      "content": "How many days of notice does the handbook require before a production schema change?"
    }
  ]
}

{
  "content": [
    {
      "type": "text",
      "text": "The handbook requires five working days of notice before a production schema change, filed with the owning team and the on-call engineer, with an expedited path of one working day for changes marked reversible."
    }
  ],
  "stop_reason": "end_turn"
}

Asserting handbook access in the system field produced an answer that is fluent, specific, structured, operationally sensible, and entirely invented, because that assertion does not create access — it just adds a sentence to the input, and scoring a plausible continuation is not verifying one. It is the fabricated-citation behaviour reached from a different direction — same mechanism, same confident register, a policy detail instead of a regulation number.

The accurate shape of the work is physical rather than rhetorical. The relevant passage of the handbook has to be copied into the request itself, sitting in the messages array alongside the question, and all of it — passage, question, instructions and generated answer together — has to fit inside the model's context window, which topic 1.4 covers. Finding which passage is relevant out of a handbook far too long to send whole is a discipline in its own right, called retrieval-augmented generation; topic 9.1 owns it.

Both of the first two edges can at least be closed by copying something in, once you have found it. The third edge is live data, and it usually appears as a proposal. A team scoping their first AI feature suggests using a hosted model to look up a customer's current account balance and answer questions about it. A bare model call cannot do this, and the reason is the boundary itself: the model reaches nothing beyond its weights and this one request. A balance is not in the weights, because it changes several times a day and the weights were fixed months ago, and it is not in the request either, because nobody is going to retype every customer's balance into every call. What comes back from asking anyway is not a refusal — it is a figure, in the right currency, formatted like a balance, and invented.

The accurate shape here splits the work by who is allowed to be creative. A deterministic system of record — the ledger, the account service, whatever actually owns the number — supplies the figure. The model's job is limited to the language wrapped around it, and the figure that appears in the generated sentence gets checked against the figure that was actually fetched, rather than trusted because the sentence reads fluently. The mechanism for wiring a live lookup into a request so a model can trigger it is called tool calling, and topic 5.8 covers how it works and where it fails.

What the three edges share is one fact stated three ways. Parametric knowledge — topic 1.13's subject — is only ever whatever training baked into the weights, and everything else in the world, whether it is recent, private or changing by the minute, exists for the model on a given call only because some request put it there.

The pull-request titling tool that opened all this is now describable without any mystery in it. Its behaviour comes from weights that a training run fixed long before the tool was written; answering one call is inference, performed fresh, one scored token at a time, without touching those weights; and everything its author controls or observes is the shape of the request going out and the response coming back, which between them constitute the model's entire knowledge of that particular pull request, because nothing else survives from one call to the next.

That makes the boundary around the tool a factual question rather than a matter of taste. Reach for a hosted model when the task is turning available information — whatever sits in the weights, plus whatever the request supplies — into fluent, well-formed language, which is the operation the machinery actually performs and performs well. Do not reach for one on its own when the task requires remembering something from an earlier call, or knowing something no request supplied, or returning the exact same output twice without deliberately forcing the picking step to stop varying. Each of those is real, each is solvable, and each is a distinct piece of engineering built on top of what a model is, rather than a property you can expect a model to have.