Base, instruct, reasoning and multimodal variants

A model is usually chosen as a string. meta-llama/Llama-3.1-8B goes in one config field and meta-llama/Llama-3.1-8B-Instruct goes in another, and the suffix is the entire difference between two directories published by the same team from the same training run. This topic is about what that suffix changes, which is not how much the weights know but how text has to be laid out on the way in and what the checkpoint does when it gets there.

The four published forms are base, instruct, reasoning and multimodal. Each is a checkpoint — the weights frozen at the end of some stage of training, packaged with the tokenizer that converts text into the integer IDs the model reads. A token is the unit that tokenizer produces, a chunk of text a few characters long, and it is also the unit providers bill by in both directions, with a thousand tokens working out at roughly 750 English words. Running one narrow task through all four and putting the outputs next to the token counts takes about ten minutes and settles arguments that otherwise run for a week.

One pretraining run, four checkpoints#

The prediction objective — given some text, produce the next token — is run over an enormous corpus of ordinary documents in a stage called pretraining. What gets published afterwards can be finished in several different ways, and each way is released as its own checkpoint with the same architecture and parameter count as its siblings. Those siblings are the variants. A base variant stops where pretraining stops. An instruct variant continues training on examples of a request followed by a helpful reply. A reasoning variant is trained further still, to write out intermediate work before committing to an answer. A multimodal variant carries an extra encoder so that images, and in some families audio, can enter alongside text.

Every one of those later stages is the same procedure as pretraining rather than a different kind of operation, and the rest of this topic rests on that fact. Training means passing text through the model, comparing the token it scored highest at each position against the token that actually came next, and nudging every weight a small amount in the direction that would have made the real token score higher. Nothing in that procedure knows what a request is, or what helpfulness is, or what a turn is. It knows only which documents it was shown. So a later stage changes what a checkpoint does by changing the documents, and the behaviour that comes out the other side is whatever the new documents made probable.

Variants exist because pretraining on its own produces a model that continues documents, which is some distance from a model that answers questions, and each later stage buys a behaviour the pretrained weights do not have. Each stage also leaves a mark on the input side, because the extra training was done on text laid out in one particular way, with particular marker strings separating who said what, and the finished checkpoint expects that same layout when you call it. Choosing a variant is therefore choosing both a behaviour and an input format, and it is usually done by whoever copied a model string out of a README.

The description most people reach for at this point is that a base checkpoint is the same model with the safety training taken off, and it is worth blocking before it does damage, because it is wrong in both directions. It is tempting for an honest reason: an instruct checkpoint declines some requests and a base one does not, so it looks as though something was present in the first and removed from the second. But post-training adds to the weights rather than subtracting from them. Declining is a behaviour that later training put in, not a filter draped over the pretrained weights that can be peeled back off. A base checkpoint does not refuse for the same reason and in the same way that it does not answer — neither behaviour was ever demonstrated to it. In the other direction, nothing the instruct checkpoint knows has been withheld from the base one, because both were shaped by the identical pretraining corpus and one of them simply stopped there.

What a base checkpoint does with an instruction#

A base model is the artifact that comes out of pretraining and nothing else. It has been optimised for one objective and has never been shown a single example of the pattern "here is a request, here is a helpful response that satisfies it". Ask it a question and it does not decline to answer, because declining is not a behaviour it has either; it estimates what text most plausibly follows the text you gave it, and emits that. Documents that open with a task instruction are, in a web-scale corpus, overwhelmingly specifications, homework sheets, README files and prompt collections, so a task instruction is continued the way those documents continue.

That sentence compresses four steps, and each of them is checkable on its own. The checkpoint scores every entry in its vocabulary for the position immediately after the text you handed it. Those scores were set by a procedure whose only reward was scoring highly whatever token really did come next in the training documents. Your text — an imperative instruction, then a receipt laid out in columns — is the opening of a document, and in a corpus of ordinary web text the openings that look like this are followed by more of the same document: a second worked example, a note to whoever is meant to be doing the exercise, a section heading. So the highest-scoring token is the first token of one of those continuations, and the loop that samples it, appends it and calls the function again does the rest. Nowhere in that sequence is there a step that asks whether the instruction has been satisfied, because a language model has no such step to skip.

from openai import OpenAI

client = OpenAI(base_url="http://vllm-8b:8000/v1", api_key="unused")

PROMPT = (
    "Extract the merchant, date and total from the receipt below "
    "and return JSON with keys merchant, date, total.\n\n"
    "PRET A MANGER 11 BROAD ST\n"
    "2026-03-14  14:22\n"
    "FLAT WHITE            3.40\n"
    "TOTAL                 3.40 GBP\n"
)

r = client.completions.create(
    model="meta-llama/Llama-3.1-8B",   # the base checkpoint
    prompt=PROMPT,
    max_tokens=512,
)
print(r.choices[0].text)
print(r.choices[0].finish_reason, r.usage.completion_tokens)

What comes back is grammatical, plausible, thematically consistent and completely useless:

Extract the merchant, date and total from the receipt below
and return JSON with keys merchant, date, total.

TESCO EXPRESS 42 HIGH ST
2026-03-15  09:04
MEAL DEAL             3.90
TOTAL                 3.90 GBP

## Notes for candidates

Receipts are supplied as raw OCR output, so whitespace and column
alignment are not reliable. Do not assume the total is the last line
of the document; some receipts include a loyalty balance after it...

length 512

It has invented a second exercise and then started writing the instructions for a candidate assignment, because that is what a document beginning with an extraction task usually contains next. Nothing stopped it, either: finish_reason came back as length because a base checkpoint has no notion of a turn that ends, so it ran into max_tokens and was cut mid-sentence. Fluent, on-topic, not an answer, no natural stopping point — that combination is the standard signature of a chat-style prompt sent to a base model.

"No notion of a turn that ends" is a phrase worth taking apart, because stopping is not a decision a model makes. Stopping is a token. The serving stack samples tokens in a loop and ends that loop when the sampled token is one it has been told terminates the sequence, and for a base checkpoint the only such token is the one that marks the end of a document. Part way through a page of worked examples, the end of the document is not a plausible next thing, so that token never becomes the likeliest choice, so it is never sampled and the loop never ends. What cut the output off was the caller's own max_tokens ceiling. Nothing failed here: the single condition that terminates generation was simply never met.

The instruct sibling, Llama-3.1-8B-Instruct, was produced by taking those same pretrained weights and continuing to train them on instruction-response pairs, then on preference data that rewards responses people judged useful. Two things came out of that process. The weights learned the behaviour of answering, and the checkpoint acquired a serialisation — a fixed way of marking where the user's text ends and the model's turn begins — which the tokenizer ships alongside it. Both matter, and the second is easy to lose.

The first of them deserves stating without the shorthand, because "learned the behaviour of answering" is the kind of phrase that sounds like an explanation and is not one. Instruction tuning does not install an answering mode beside the document-continuing one, and there is no switch between the two. It runs the same nudge-the-weights procedure described above over a far smaller collection of documents, in which a request is always followed by a reply that satisfies it. The effect lands at exactly the positions where the two behaviours disagree. Immediately after a request, the first token of a direct answer used to score below the first token of another exercise; afterwards it scores above. That is the whole of the change, and it accounts for the shape of everything else in this topic. Behaviour here is a property of which continuation is most probable, so anything that alters what the text in front of the cursor looks like alters the behaviour with it — which is why the markers in the next chapter matter as much as the training did.

The four rows side by side#

The table below is one receipt-extraction task run across all four variants at 7 to 11 billion parameters, 900 items, greedy decoding — always taking the highest-probability next token, so a run repeats exactly — and max_tokens=512 throughout.

The reasoning row is the one that surprises people who have only read the marketing. Its visible answer is 41 tokens, comparable to the instruct variant's 38, and its accuracy is one point higher. To produce those 41 tokens it generated 640 more, working through the receipt line by line, second-guessing whether 3.40 GBP should be serialised as a number or a string, and re-reading the date format. Those 640 tokens are sampled one at a time like any others and billed at the output rate, whether or not the API hands them back. On a host charging $0.60 per million output tokens, 900 receipts cost about $0.02 through the instruct variant and $0.37 through the reasoning one. That ratio does not improve with scale — at 40,000 receipts a day it is $0.91 against $16.35 — and the extra tokens also occupy the context window, which is the fixed token budget a single request has to fit inside, and the KV cache the server holds for every token already in the sequence. That last effect is why the reasoning variant's throughput on the same GPU was roughly a fifth of the instruct variant's.

Why hiding those tokens does not make them free is worth spelling out, since "the API does not return them" sounds a great deal like "you are not paying for them". Producing one token costs the provider one forward pass through the model, and that pass has already finished by the time anything decides whether the token will appear in the response. Withholding is a formatting choice taken downstream of the work, not a way of avoiding it. The same argument settles the context question: each of those tokens was appended to the sequence the moment it was produced, because appending is the only way the tokens after it can read it, so they occupy the window whether or not they are ever shown to you.

How the hidden tokens reach you differs by stack, and the difference matters for both parsing and accounting. The distilled checkpoint above emits its work inline, wrapped in <think> and </think>, so it arrives in the same content field as the answer and anything downstream has to strip it. vLLM, the open-source inference server, can split the two for you if you start it with a reasoning parser, in which case the trace lands in reasoning_content and content holds only the answer. Most hosted reasoning APIs do not return the trace at all and report it in the usage record instead:

"usage": {
  "prompt_tokens": 412,
  "completion_tokens": 681,
  "completion_tokens_details": {"reasoning_tokens": 640}
}

Same cost in all three arrangements, three different ways to be surprised. An inline trace breaks any parser that assumes the response starts at the first character, which is how a service that worked against an instruct model returns JSONDecodeError the day someone swaps in a reasoning one. A hidden trace makes max_tokens behave in a way that reads as a bug: the budget is spent on thinking, generation stops before the answer begins, and the response comes back empty with finish_reason: length. A trace in a separate field is the tidiest of the three and still counts against the context window, so long conversations with a reasoning model fill it faster than the visible transcript suggests.

Which of those three arrangements you are dealing with is a provider policy rather than a property of the weights, and it has moved between model generations more than once. The arrangement to design against is the one your provider documents for the model you are calling, confirmed against a real response you have printed, rather than the one described in a tutorial written for an earlier generation of the same family.

The two front doors of a multimodal checkpoint#

A multimodal checkpoint has two ways in. Text enters through the tokenizer as usual; an image enters through a separate processor that resizes it, splits it into patches, runs it through a vision encoder, and projects the result into the same embedding space the text tokens live in. The API's content-part format — a list of typed parts rather than one string — is what routes a piece of input to the second door.

Both doors end in the same place, and that is why neither raises an error and why picking the wrong one is so quiet. What the model reads is a sequence of vectors, one per position, all of the same width. The tokenizer builds that sequence out of text by cutting it into chunks and looking each chunk up in a table. The vision path builds it out of a picture by cutting the picture into patches, running those through a separate stack of weights trained to describe images, and projecting the result to the same width. Hand a base64 string to the tokenizer and it produces vectors too — correctly, because a base64 string is text and the tokenizer's job is text. The sequence that reaches the model is well-formed on either path. What it lacks, on the text path, is any vector that was computed from the picture at all. The format below is what puts a piece of input on the other path.

{
  "model": "Qwen/Qwen2.5-VL-7B-Instruct",
  "messages": [{
    "role": "user",
    "content": [
      {"type": "text",
       "text": "Extract merchant, date and total. Return JSON with keys merchant, date, total."},
      {"type": "image_url",
       "image_url": {"url": "data:image/jpeg;base64,/9j/4AAQSkZJRg..."}}
    ]
  }],
  "max_tokens": 256
}

Sent that way, a receipt photograph reported 1,176 prompt tokens. That number is the image itself: Qwen2.5-VL uses dynamic resolution, so a 1,024×768 photo becomes a grid of visual patches merged into that many embedding slots, and those slots are counted and billed as input tokens exactly like text. Sending a phone photo at full resolution rather than a downscaled one is therefore a billing decision as much as a quality one — the same receipt at 3,024×4,032 came in at 4,290 tokens with no measurable accuracy gain.

The image-token count has a second consequence. Those tokens occupy the context window like any other, so a conversation carrying four receipt photographs at 1,176 tokens each has spent 4,704 tokens before a word of instruction is written, and a document-review feature that pastes in twenty pages of scanned PDF can exhaust a 32k window on images alone while the visible transcript is three sentences long. Providers differ in how they count, too: some quote a flat per-image price, some scale with resolution in fixed tiles, and some — like the dynamic-resolution scheme above — vary continuously with the aspect ratio you send. Downscaling on the client before upload is a real lever, and it costs nothing in quality when the source is a 12-megapixel photo of an A5 receipt.

Four rows are not a quality ladder#

Reading that table as a ranking leads to bad decisions in both directions. The base and instruct checkpoints share their pretrained weights, so whatever the base model knows about receipts the instruct model also knows, and the zero in that row is a statement about serialisation and prompt style rather than about capability. There is a way to write this task for a base model that works, and it is worth knowing because base hosting is materially cheaper.

Nor is "reasoning" a strictly larger version of "instruct". Spending generation budget on intermediate work helps on problems where the answer depends on several dependent steps, and does very little on problems that are essentially transcription. It also introduces a latency profile that is qualitatively different, since the thinking happens before the first visible character appears.

Repository names are a weak guide to which variant you are holding. The suffixes -Instruct, -it, -chat and -sft all mean instruction-tuned by convention, and the convention is honoured unevenly: community re-uploads drop suffixes, quantised copies rename freely, and a merge of two checkpoints inherits whatever the uploader typed into the form. The properties that actually distinguish the variants are all inspectable in the repository — whether the tokenizer carries a chat template, what the generation config lists as its end-of-sequence IDs, and whether the config declares a vision tower — and reading those takes less time than reading the model card.

The variant axis is also independent of the axes it gets confused with. Parameter count, quantisation — storing the weights at lower numeric precision so more model fits into less memory — context window length and fine-tuning on your own data are four separate decisions, and a 7B instruct model can beat a 70B base model on your task without either fact being surprising. Nothing here says a bigger model is better; it says that two checkpoints with identical pretrained weights and different post-training expect different inputs and produce different outputs.

Four variants, four post-training histories, four different things done to the same prompt. The base checkpoint continues your document because that is the only behaviour it has, the instruct checkpoint answers because it was trained to and because its markers told it where to start, the reasoning checkpoint answers after spending a generation budget that is billed and usually invisible, and the multimodal checkpoint answers about an image only when the image travelled through the content-part path rather than through a text field. Every one of those differences is measurable in an afternoon, and the token counts are as informative as the outputs.

The explanation offered so far for the gap between the base and instruct rows — that instruction tuning taught one of them to answer — is only half of what happened. The other half is that the client turned a list of message objects into a single flat string before anything reached the model, and it did so using markers that came out of that checkpoint's own tokenizer. That string is the real input. Everything the model does is a function of it, and it is worth looking at directly.