Stateless requests and conversation history

Stateless here means what it means in any HTTP context: the server keeps nothing between calls, so each request carries everything its answer depends on. A chat request carries a model, a ceiling, an instruction and an ordered array of role-tagged messages; a response carries generated content, a stop reason and token counts. No conversation identifier appears in either direction — the msg_ id on the response is a receipt for one generation, not a handle you can send back — and there is no cookie, no session token, and no server-side transcript being appended to as you go.

That property is easy to assent to and hard to believe, because the product built on top of it is extremely convincing: the assistant answers a follow-up question that contains no noun, refers back to what it said four turns ago, and behaves in every observable way like something that remembers. This chapter makes the property concrete rather than asserting it — a two-request experiment that isolates the single variable — and then works through what follows for storage, because statelessness is an operational gift right up until the state you are carrying lives in one process's memory. It closes on the two things statelessness is routinely confused with: provider-hosted session products, which are a different API, and prompt caching, which is an optimisation over the resend rather than an alternative to it.

Each request is a pure function of its body#

The comparison worth holding is a REST endpoint with no session. Two POSTs with the same body produce the same answer regardless of which of your servers issued them, because everything that determines the answer is in the request. A chat completion behaves the same way: the array you send is the entire universe the model reasons over for that call, and identical arrays produce statistically identical behaviour whether they came from the same worker a second apart or from different data centres a week apart.

This is worth proving to yourself rather than accepting, because you will one day have to convince a colleague under time pressure. The experiment is two requests from a process that has just started, so that no in-memory state can be confounding the result:

import anthropic
client = anthropic.Anthropic()

TURN_1_Q = "My name is Priya and I manage the Lisbon warehouse."
TURN_1_A = "Good to meet you, Priya. How can I help with the Lisbon warehouse?"
TURN_2_Q = "Which warehouse did I say I manage?"

def ask(messages):
    r = client.messages.create(model="claude-opus-5", max_tokens=200,
                               messages=messages)
    return r.content[0].text, r.usage.input_tokens

# Run A — the stored array, all three entries
print(ask([
    {"role": "user", "content": TURN_1_Q},
    {"role": "assistant", "content": TURN_1_A},
    {"role": "user", "content": TURN_2_Q},
]))

# Run B — the newest user message alone, same model, same account
print(ask([{"role": "user", "content": TURN_2_Q}]))

Run A answers "Lisbon" and reports around 40 input tokens. Run B answers that it has no record of you mentioning a warehouse and reports around 15. The two calls are seconds apart on the same API key, and the second one has no access to what the first one sent. There is no cache to warm, no flag to enable, and no state to clear — the difference between remembering and not remembering is entirely the difference between the two arrays.

The two token counts are the receipt for that claim, and reading them is faster than trusting the answers. Run B sent one short question, and fifteen tokens is about what that question plus its markers weighs. Run A sent the same question plus two earlier turns, and it weighs a little under three times as much, which is roughly the ratio you would expect from three entries of comparable length. The counts are not describing how much the model knew. They are describing how much text arrived, and the answers track the text exactly.

What makes Run A look like memory is worth stating in mechanism, because the alternative is a reader who accepts the result and keeps the wrong picture. When the model produces the first token of its answer, every token of the input is available to it at once — the name in entry zero is no less reachable than the question in entry two, because the whole sequence is read in one pass rather than consumed and discarded. So "Priya" is not recalled. It is present. The model resolves "which warehouse did I say" by attending to text sitting in front of it, in exactly the way it would resolve a pronoun in a paragraph you pasted in whole, and the fact that some of that text was written a minute ago and some a second ago is not a distinction the sequence records.

The reason this experiment is the right one to run, out of all the experiments you could construct, is that it isolates the single variable. Sending turn two from a fresh process rules out client-side memory; sending it twice under the same credentials in the same minute rules out account-level or model-level state; and using a fact that appears nowhere but in turn one rules out the model inferring the answer. What is left is the array.

Where conversation history has to live once there are three workers#

Statelessness is a gift operationally, because any worker can serve any turn and there is nothing to replicate or fail over — but only as long as the state each request carries is somewhere every worker can read. Per-process memory fails at every boundary a production system has:

There is a symmetry in that table worth naming, because it explains why the correct answer feels like extra work. Statelessness did not remove the problem of holding a conversation. It moved the problem across the boundary. The provider gets to route any request to any free machine precisely because it carries nothing between calls, and your service inherits the identical wish — any replica should be able to serve any turn — while being the party that now has to hold the state. You cannot solve it the provider's way, because somebody has to remember, and downstream of you there is nobody left. So the work is the ordinary work of putting shared state where all your processes can reach it, and the only thing unusual about it is that the requirement arrived disguised as a property of somebody else's API.

The failure only shows up under conditions that development and staging usually lack, which is why it survives review. One process on a laptop, one process in staging, one process in production for the first few months — a dictionary keyed by conversation id works flawlessly in all three, and then the topology changes and the bug is instant and total.

Sticky sessions are the tempting patch, because a balancer can be configured to pin a conversation to a replica in an afternoon. It converts a constant failure into an intermittent one: the pin survives normal traffic and breaks on exactly the events that matter — a deploy, a crash, an autoscale event, a customer switching networks — and it makes a routine restart a customer-visible outage. It also makes the retry story worse rather than better, because now a retry must be routed back to a specific process that may be the one that just died.

Each of those breakages is the same step reached by a different route, and seeing that is what stops the patch from looking like a cheaper fix. A pin is a promise to send a conversation back to a process that holds something. Anything that ends that process ends the thing it held: a deploy replaces it, a crash removes it, an autoscaler that scales in removes it, and an autoscaler that scales out reshuffles which conversations map to which replica. A customer moving from office wifi to a mobile network arrives with a different source address, and a balancer pinning on address has no way to recognise them. So the pin does not make the history durable. It narrows the window in which its absence is visible, and it narrows it to exactly the moments — deploys and incidents — when you are least able to tell one broken thing from another.

The verification is mechanical and belongs in your test suite rather than in a code review. Instrument the client so that every outbound body is logged with its conversation id and the number of entries in messages, drive a four-turn conversation through the service, and assert the sequence:

# From a request log, grouped by conversation id, ordered by time:
# turn 1  -> len(messages) == 1
# turn 2  -> len(messages) == 3   (user, assistant, user)
# turn 3  -> len(messages) == 5
# turn 4  -> len(messages) == 7
assert [len(b["messages"]) for b in bodies] == [1, 3, 5, 7]

Then run the same test with the requests spread across two workers. If the lengths still climb, the history is genuinely shared; if turn three comes back as len == 1, you have found the bug before a customer did. This test is cheap, it is the only reliable way to confirm the property, and it catches a whole class of regressions — a caching layer that silently drops entries, a truncation policy that fires earlier than intended, a serialisation bug that loses tool blocks.

Store the blocks, not the rendering#

A shared store fixes the topology and introduces one new way to lose information, which shows up as a puzzling error rather than as missing memory. A team storing turns as role plus a text column has to flatten the assistant's content down to a string, so an entry that carried both a sentence and a tool call is written as just the sentence. Rebuilding the array from those rows produces an assistant turn with the call missing and the following user turn still carrying the result — an orphan of exactly the kind the previous chapter's validator rejects before inference, on exactly the conversations where the assistant had done the most work.

The schema that avoids this keeps the content list as sent, verbatim:

CREATE TABLE turns (
  id               uuid PRIMARY KEY,
  conversation_id  uuid NOT NULL,
  seq              int  NOT NULL,
  role             text NOT NULL,          -- 'user' | 'assistant'
  content          jsonb NOT NULL,         -- the block list, unmodified
  created_at       timestamptz NOT NULL DEFAULT now(),
  UNIQUE (conversation_id, seq)
);

Two design points in that table earn their place. seq rather than created_at decides order, because two turns written in the same millisecond by different workers must still have a defined position and timestamps do not guarantee one. And content is JSON rather than text, because the block list is the unit the provider validates — every transformation you apply on the way in is a transformation you must apply identically on the way out, and the safest number of transformations is zero.

Reconstruction is then a query and a map, and it is worth making that function the only place in the codebase that builds an array. Every worker calling the same function against the same rows produces the same bytes, which is what makes it safe for any worker to serve any turn, and — because prompt caching is a prefix match on those bytes — it is also what makes a cache hit possible at all. An array assembled two different ways by two different code paths is two different arrays, however similar they look.

def build_array(conn, conversation_id):
    rows = conn.execute(
        "SELECT role, content FROM turns "
        "WHERE conversation_id = %s ORDER BY seq", (conversation_id,))
    return [{"role": r.role, "content": r.content} for r in rows]

The objection raised against this is usually latency, and it does not survive the comparison. A read from Redis is on the order of one to three milliseconds; the same query against a warm Postgres index is perhaps five to fifteen. The generation you are about to wait for takes one to four seconds. The store contributes well under one percent of the turn's wall-clock time, and it buys the ability to restart a worker without breaking a conversation.

Where the stateless rule is misread#

It is not a claim that no provider offers stateful conversations. Several platforms have built session-oriented surfaces on top of the stateless primitive — server-managed agent sessions that hold a transcript, run tools in a hosted container, and stream events back to you — and those genuinely do keep state on the provider's side. They are a different product with a different API, different pricing, and a different operational posture, and choosing one is a real architectural decision. What does not exist is a stateful mode hiding inside the plain messages endpoint, waiting to be switched on.

It is also not the same thing as prompt caching, which people conflate constantly because both involve the provider retaining something between requests. A cache stores the computed prefix of an array you already sent, so that re-sending the same leading tokens costs less and returns faster; it does not store the array, it will not reconstruct one for you, and a cache hit still requires you to send every token — you are billed for them at a reduced rate rather than not billed for them. Caching is an optimisation over the resend, and the resend is mandatory.

The last clause is the one worth understanding rather than memorising, because it explains three separate rules further on in this book. What the provider stored is not your text but the intermediate state the model computed while reading it, and that state is only reusable for a prefix, because every position's state was computed with all earlier positions visible. To know whether its stored state applies, the provider has to compare what you sent against what it holds — which requires you to have sent it. And the comparison has to succeed byte for byte up to the point of divergence, because a single different token invalidates every state computed after it. Hence the shape of all caching advice you will meet: put the stable material first, append rather than edit, and expect an edit near the front to cost you the whole saving even though the diff looks tiny.

And it is not an argument that the whole transcript must be your storage schema. What must be reconstructible on any worker is the array: the ordered turns with their roles, their content blocks, and — after the previous chapter — the tool-call ids that bind pairs together. What your database holds around that can be richer: soft-deleted turns, moderation verdicts, the model version that produced each reply, the id of the worker that served it. The mapping from your rows to the array is a function you own, and writing it explicitly rather than letting an ORM's default serialisation decide is what keeps the next chapters' failure modes out of your service.

Conversation memory, then, is a feature of your application and not of the API you are calling. The provider gives you a function from an array to a message; everything that makes a sequence of those calls feel like a conversation is code you wrote and rows you stored. Treating it that way turns a class of mysterious model behaviour into ordinary distributed-systems work with ordinary tools: a table, an ordering column, one function that builds the array, and a test that asserts the array grows.

Once history is shared and reconstructible, the service works — and the bill starts behaving in a way that surprises people who reasoned about it linearly. Turn forty of a conversation carries turns one through thirty-nine along with it, every time, and has been paid for at every step along the way. The growth has a closed form, and it is not the one most estimates assume.