Image content blocks

An image input is a picture carried inside a model request as one part of a message, so that the model receives pixels and words together in a single turn. The model on the other end is multimodal: it was trained on images as well as text, can take either as input, and answers in words about both, which is why a model used this way is usually called a vision model. Nothing converts the picture into text before it arrives. The pixels go into the same input the question goes into, and the answer comes back from reading the two together. This matters because a great deal of what a system needs to know has never been written down anywhere — it exists as coloured pixels in a screenshot, a scan, a chart whose underlying numbers nobody kept, or a photograph — and getting that information into a program used to mean a person retyping it, or a pipeline built in advance against one specific layout. In practice this turns up wherever pixels arrive from outside your own code: support tooling receives customer screenshots, document pipelines work through scanned invoices and delivery notes, analytics work runs over charts whose underlying numbers were never kept, moderation systems classify user-uploaded photographs, and agents that drive a browser or a desktop have to see the screen before they can decide what to click.

What separates a convincing demo from a system you can run is everything that happens after the picture is accepted: what it costs on every request, what the model can still resolve inside it once it has been shrunk, and what it does when it cannot resolve the thing you asked about.

What a vision model reads#

Backend engineers tend to underestimate this, because the mental model many people carry is OCR — a component that pulls character strings out of pixels and hands back a blob of text. What these models do is closer to answering a question about a scene, using the pixels the way they use any other part of the input. They read UI chrome. They read the shape of a chart, including which series sits above which and roughly where the crossover falls. They read tables with visible rules and often without, they read handwriting when it is tidy, and they will tell you that a button is greyed out or that a modal is overlaying the page.

A support engineer at a payments company had exactly one artefact to work with: a customer's photo of a checkout page that had failed, taken with a phone and attached to an email. The browser console was gone, the failure had never reached the company's own logs because it happened inside a third-party iframe, and the toast notification carrying the error string was half-covered by the phone's own status bar. She attached the file to a model call with the question "what error is this page showing, and which input on the form is highlighted?" and got back the visible fragment of the error string, the observation that the card-number field carried a red border, and the detail that the expiry field was empty. The ticket was routed correctly in about ninety seconds, against an afternoon of asking the customer for a better screenshot.

What these models do badly is dense small text. Everything else in the list two paragraphs above — the chrome, the chart shape, the tables, the tidy handwriting — is reliable enough to build on, provided the feature you are asking about survives the shrinking every image gets on the way in — the subject of chapter 3, on what a model can resolve once an image has been resized.

An image is another block in the content list#

A message's content is a list of typed blocks, and a bare string is sugar for a list holding one text block. An image is simply another entry in that list, sitting beside the text block rather than in some separate attachment field:

{
  "model": "claude-opus-5",
  "max_tokens": 1024,
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "image",
          "source": {
            "type": "base64",
            "media_type": "image/png",
            "data": "iVBORw0KGgoAAAANSUhEUgAAB4AAAAQ4CAYAAAD..."
          }
        },
        {
          "type": "text",
          "text": "What error is this checkout page showing, and which input is highlighted?"
        }
      ]
    }
  ]
}

There is no upload step and no handle. The bytes travel in the request body, base64-encoded, on every call that needs them. That single fact has consequences running through the rest of this topic.

That absence of an upload step follows from how the model is called. Each request is answered by assembling the whole of what the model reads out of the body just sent, and nothing you sent earlier persists on the other end unless the provider was separately asked to store it — which is what the uploaded-file form in the table below is for. An image the provider is not storing therefore has to arrive in the body of every request that needs it read again. It is charged on each of those requests because it is present in each, it counts against the request size limit because that limit is on the body it sits in, and in a multi-turn conversation the accumulated history is what the body gets built from, so it rides along until something takes it out.

Charging in tokens for something sent as bytes needs one further step. A model consumes a sequence of vectors — a list of numbers of a fixed length, one such list per position — and nothing else reaches it. Text gets into that form by lookup: the string is cut into pieces, and each piece has an entry in a fixed inventory the model was built with, so that entry becomes the vector standing at the piece's position. A picture has no entry in that inventory, since there is no row in it for a photograph of a checkout page, so the pixels have to become vectors by another route.

That route is an encoder the provider runs before the model reads anything. It cuts the image into patches of a fixed size and turns each patch into one vector, of the same width as the vectors the text pieces produced, since all of them are going into a single sequence and a sequence has one width. Those vectors are then spliced into the sequence at the point where the image block sat in your content list, immediately ahead of the vectors for the text block that followed it.

Three things follow from that splice. The image occupies a run of positions in the same sequence as your words, and the length of that run tracks how many patches the picture was cut into rather than how many bytes its file happens to weigh. Those positions are input positions, no different in kind from the ones your question sits at, which is why an image is billed against the input side rather than under a heading of its own. And the sequence carries positions rather than names, so nothing in what the model reads can be referred to by name later.

Providers accept a small number of source forms, and they differ enough that this is worth having in front of you when you choose:

The inflation in the first row is the one figure in that table you can derive rather than look up. JSON is text and a compressed image is not: most byte values in a PNG correspond to no printable character, and several cannot appear inside a JSON string at all. Base64 gets round that by reading the bytes three at a time and writing each group as four characters from an alphabet that is safe in text. Four characters where there were three bytes is a third more, so the encoded form is a third larger than the file. A third on top of 4 MB is a little over one and a third megabytes, and adding that back gives the 5.3 MB request in the table.

Limits vary too, and they move, so read the reference for the provider you are on rather than trusting a number in a tutorial. As a shape: per-image size caps in the range of 5 MB to 20 MB are typical, requests carrying dozens of images are accepted, and the mainstream providers resize anything above the resolution cap each of them publishes before doing anything else, silently in the sense that nothing in the response reports it. PNG, JPEG, GIF and WebP are the common set. The media_type you declare has to match the actual bytes; sending a JPEG labelled image/png is rejected by some providers and, worse, quietly mishandled by others.

In Python the same call is unremarkable, and the only part worth attention is that you are responsible for reading the file and encoding it:

import base64
from anthropic import Anthropic

client = Anthropic()

with open("checkout_failure.png", "rb") as f:
    encoded = base64.standard_b64encode(f.read()).decode("utf-8")

response = client.messages.create(
    model="claude-opus-5",
    max_tokens=1024,
    messages=[{
        "role": "user",
        "content": [
            {"type": "image",
             "source": {"type": "base64",
                        "media_type": "image/png",
                        "data": encoded}},
            {"type": "text",
             "text": "What error is this checkout page showing, "
                     "and which input is highlighted?"},
        ],
    }],
)
print(response.content[0].text)
print(response.usage.input_tokens)

Log that input_tokens value from the first call you ever make. It is the number the next chapter is about, and having a real one from your own image beats any estimate.

Ask something the image can answer#

The quality of the answer depends on the question far more than most people expect, because "describe this image" invites a paragraph of generic scene description that no downstream system can consume. A question that names the thing you want, in the vocabulary of the domain, gets a different class of answer. Compare "what does this chart show" against "this is weekly active users by plan tier; in which week does the Pro series first exceed the Free series, and what is the approximate gap in the final week". The second question tells the model what the axes mean, which removes an inference it would otherwise have to make from the axis labels, and it asks for two specific values you can validate.

The same applies to output shape. If the answer is going into a database, ask for the fields explicitly and give the model somewhere to put uncertainty, because a model that has no way to say "I could not read this" will produce something rather than nothing:

prompt = """This is a photograph of a retail receipt.

Extract:
- merchant: the trading name printed at the top
- total: the grand total as a decimal string, e.g. "42.30"
- currency: ISO 4217 code
- date: ISO 8601 date

Return JSON with exactly those four keys. For any field whose
characters you cannot actually resolve in the image, use null.
Do not infer a value from context or from a typical receipt."""

That last instruction is not decoration. The failure mode that costs teams the most in this area is a model returning a well-formed, plausible value for something it could not see, and an explicit licence to return null measurably reduces it.

Why it reduces it has nothing to do with making the model more careful. The answer is produced one piece at a time, each piece chosen against everything already in the sequence, and nothing in that loop consults a record of which regions of the image were legible — no such record exists anywhere the loop can reach. What the prompt controls is which continuations are available. A request that names four keys and demands a decimal string for the total leaves nothing but values to continue with where the total goes, so a value is what comes out, and naming null as a licit entry adds a continuation that can win instead.

It does not eliminate the failure, because a half-legible total still makes a plausible value a strong continuation, and a strong continuation can outscore the empty one. That is a thread picked up in chapter 5, where a low-resolution setting produces exactly that failure at scale.

Several images, and no way to name them#

Send several images in one message and a new failure appears: the model's answers stay confident and specific while citing the wrong picture, describing the contents of image four as though they were in image two. Nothing about the images is ambiguous to a human looking at them, so the cause is easy to misread as a weakness in the model's vision. It is not. It is a naming problem.

Each image is expanded into a run of visual tokens that sits inline in the same sequence as your text, and it arrives with no filename, no alt text, no index and no separator carrying a name. "The second screenshot" is therefore a position the model has to reconstruct by counting boundaries across several thousand visual tokens, and then hold that count while answering a question about content. It is the same class of task as asking someone to answer a question about the third paragraph of a page they are looking at from across the room — the content is legible, the ordinal is the part that slips.

The belief that counting contradicts is a natural one. The request carried four separate image blocks and the code that built it looped over four files, so it is hard not to picture the model receiving four pictures and taking each in turn. Those boundaries are a fact about the JSON, which is parsed and discarded on the way in. What reaches the model is one line of positions in which the last patch of one screenshot is followed immediately by the first patch of the next with no marker between them, so the patch vectors carry content while the only evidence for identity is where those positions fall relative to the boundaries.

The answer is produced by the same next-piece machinery whether the count came out right or wrong, since nothing in that machinery distinguishes a well-supported attribution from a mistaken one. A miscount therefore arrives as a fluent sentence about the payment step describing the address form, rather than as an error or a hedge.

The fix is to give each image a name in text, immediately before it, so the ordinal never has to be inferred:

steps = [
    ("Screenshot 1 of 4: cart review", "cart.png"),
    ("Screenshot 2 of 4: shipping address form", "address.png"),
    ("Screenshot 3 of 4: payment step", "payment.png"),
    ("Screenshot 4 of 4: confirmation attempt", "confirm.png"),
]

content = []
for label, path in steps:
    content.append({"type": "text", "text": label})
    content.append({"type": "image", "source": load(path)})

content.append({
    "type": "text",
    "text": "Which of these four steps shows the failure? "
            "Answer with the exact label of that screenshot, "
            "then quote the error text visible in it.",
})

The label does three jobs at once. It gives the model a string it can attach to the adjacent visual tokens through ordinary attention, exactly as a caption under a figure does. It gives the model a name to quote back, which turns a wrong answer into an obviously wrong answer instead of a plausible one — if it says "Screenshot 3 of 4: payment step" and describes an address form, you have caught it. And it carries the count in the label itself, so "of 4" is text rather than something to be tallied.

The first of those three is the splice read forwards. A text block placed immediately before an image becomes vectors sitting at the positions immediately before that image's patch positions, and any position in the sequence can be consulted from any position after it. The label's words therefore stand to those patches as a caption's words stand to the figure above them, which makes attaching the two ordinary work on adjacent text rather than a special faculty for pictures.

The second job carries a condition that tends to get dropped when the pattern is copied. A label converts a wrong answer into an obviously wrong one only when it names something the rest of the answer can contradict. Labels reading "Image one" and "Image two" would be quoted back just as obediently and would agree with any description at all, so it is the content noun that does the catching and the ordinal that removes the counting.

Labels also let you say things the image cannot. A timestamp, the customer's plan tier, the fact that screenshot three was taken after a page refresh, the device — all of it belongs in the label, because it is context the model would otherwise have to guess at or do without. Where the images have a natural order, saying so in the first text block ("these four screenshots are in chronological order, earliest first") is worth the tokens.

The boundaries of an image block#

An image block is not an attachment in the email sense, and it is not a document ingestion pipeline. Nothing is stored, nothing is indexed, and there is no handle you can refer to in a later turn. If the conversation continues, the image is either re-sent inside the accumulated history — at full cost, every turn — or it is gone and the model is answering from whatever it said about the image earlier. Both are legitimate designs, and choosing between them deliberately matters more once you know what an image costs. A common and effective pattern is to send the image once, ask the model to produce a compact structured description, and carry that text forward instead of the pixels.

It is also not a replacement for the text you already have. When a PDF has a real text layer, extracting that text and sending it costs a fraction of what rasterising the page and sending the pixels costs, and it is exact rather than read. Rendering a page to PNG to ask a model what it says, when pdftotext would have answered in 40 milliseconds, is a habit worth catching early. Images earn their place when the information is genuinely visual — layout, a chart with no underlying data, a handwritten annotation, a UI state, a photograph — or when no text layer exists.

And it is not OCR, in the specific sense that the model is not extracting characters and then reasoning about them. It is answering your question directly from a representation of the whole image, which is why it can tell you a button is disabled and also why it can misread a digit while sounding certain. That difference in failure mode is the subject of chapter 6, which compares a vision model against a dedicated parser.

An image input, then, is a block in a list: pixels and a question in one turn, no upload, no handle, no preprocessing you did not write. What you get for that is a model that answers questions nobody anticipated when the picture was taken, on material that exists nowhere in text. What you pay is a question you have to phrase well enough to be answerable, a label per image whenever more than one rides in a message, and an input cost on every single request.

That cost is the part nobody estimates correctly. The engineer who sends one 3000×2000 phone photo of a whiteboard and the one who sends a 400×300 thumbnail are billed very differently, and neither is billed for the number of bytes uploaded — which is where the arithmetic starts.