Conversation state

Read these first

• Chat API shape

What you will know

• Trace a request from stored turns to outbound messages and verify that no conversation state is held by the API process between requests

  A restart, retry, or second application instance must produce the same input from durable data rather than an in-memory session.

• Implement multi-turn conversation without a framework

  A list, a loop, and a database table is genuinely all it takes. Do it once before reaching for a framework.

• Run concurrent turns against one conversation and verify that their persisted ordering and ownership remain correct

  Without transactional updates or an explicit sequencing policy, two tabs can lose messages or expose one turn's response to another conversation.

• Instrument history assembly and reproduce a stale, duplicated, or cross-conversation message in the outbound request

  Logging conversation IDs, message IDs, revisions, and the final payload makes apparent model memory failures diagnosable as application bugs.

• Decide what to do when history outgrows the context window

  Summarise, drop old turns, or retrieve selectively — but decide deliberately rather than letting it truncate.

Chapters

1. 1

   Server-owned conversation history

   A chat endpoint retains nothing between calls, so multi-turn conversation is a messages row per turn, an ordered read scoped to one conversation, and an array assembled fresh from those rows on every call. The property that proves the design is honest is that restarting the process between turn one and turn two leaves turn two's request body byte-identical, and this chapter shows how to check it and what a sixty-second cache of assembled arrays does to it.

   Read chapter
2. 2

   Client-owned conversation history

   Letting the browser hold the conversation and post the full array each request keeps the backend thin, and it moves ownership to a place the user can edit, so every turn arrives as user-controlled input rather than as your own record. Beyond the obvious loss on a cleared cache, you lose the ability to evaluate, audit or bill against what was really sent. This chapter works out what changes, which products it is genuinely right for, and what an HMAC over the transcript does and does not buy.

   Read chapter
3. 3

   Debugging history assembly

   When users report that the assistant forgot an order number, the first artefact to produce is the serialised array you actually sent for that conversation. Instrument assembly with conversation id, message ids, sequence range, revision and a payload fingerprint, then work the classic bugs: a query filtered by user rather than conversation, a LIMIT that clips the front, duplicate rows from a retry, and stale revisions that reach the wire unnoticed. This chapter also shows how to tell those apart from a genuine attention failure.

   Read chapter
4. 4

   Concurrent turns and lost updates

   Two turns sent on one conversation within a few seconds each read the same stored history, call the model, then append, and their appends collide: a turn goes missing, a reply is stored against the wrong user question, or one conversation's answer is delivered into another. This chapter walks the collision, reproduces it deterministically with a barrier and a slow fake model, adds the in_reply_to column and audit queries that make the corruption assertable, and separates it from a stale read-path cache.

   Read chapter
5. 5

   Sequencing policies for concurrent turns

   Three answers to the question of what happens to the second turn on a conversation: optimistic concurrency on a version column rejects it and retries against fresh history, a per-conversation lock serialises turns but must be held across a call whose p99 is twenty seconds, and branching lets both succeed and defers reconciliation. Compare them by wasted calls, added latency, failure mode and audit story, and decide what each fate renders in the tab that lost.

   Read chapter
6. 6

   The context window budget

   Assembly has a budget: count the tokens of the array you are about to send, reserve room for the reply, and compare the total against the model's window, so crossing the line is an event you detect rather than a 400 or a silently truncated prompt you learn about from a user. This chapter does the arithmetic, shows what tool schemas and images take out of the window before your history does, explains how prompt caching interacts with anything that shortens history, and separates detection from a policy.

   Read chapter
7. 7

   Summarisation, pinned facts, and retrieval

   Once the budget check says a conversation is at the limit, silence is not a policy. Compare a rolling summary of old turns, dropping oldest turns while pinning extracted facts, and retrieving only past turns relevant to the current message — by per-turn cost, by added latency, and by what each one destroys. In a refund conversation the details a summary drops are the consequential ones: the exact amount, the condition attached, the order id.

   Read chapter
8. 8

   Reproducing a past model input

   Asked six weeks later for the exact input behind a complaint, storing every message is not enough. Assembly depends on the code version, the window config in effect that day, and a summarisation step that is itself a non-deterministic model call, so re-running it produces a reconstruction rather than the original. This chapter models the six inputs that feed an outbound array, specifies a per-call record that captures them, and prices the archive against a metadata-only tier.

   Read chapter