A constraint stated at turn 2 and broken at turn 9

A constraint stated at turn 2 and broken at turn 9 is a session in which no single turn is wrong and the session is. A turn, throughout this topic, is one exchange — a user message and the agent's reply to it — so a twelve-turn session is twelve of those in sequence, and multi-turn simply names the case where the evaluation is allowed to continue past the first. The failure always has the same shape: the user states a requirement early, the conversation continues, something between the model and the message history loses the requirement, and the agent acts against it later while behaving impeccably with respect to everything it can currently see. Read one turn at a time, the transcript contains no defect. Read whole, it contains a broken promise.

The distance between those two sentences is what multi-turn evaluation, also called conversational evaluation, exists to measure. Its unit is a whole conversation rather than one request and one response: something plays the part of the user across a sequence of exchanges, the agent answers each of them while carrying whatever state it carries, and scores are recorded for individual turns and for the finished session. You meet this wherever a product is used by talking to it over several exchanges — a support chat, a coding assistant, an internal booking desk — it usually sits in continuous integration alongside the single-turn suite rather than replacing it, and it is what evaluation tooling is offering when it advertises a "simulated user" or "multi-turn simulation". The reason such a thing has to exist, rather than being approximated inside a fixture file, is that a conversation accumulates state: facts the user stated eight minutes ago, promises the agent made, a message history that the harness has been summarising and trimming to fit the model's context window. No fixture file can contain that state, because the only way to produce it is to hold the conversation.

Eleven bookings over a stated ceiling#

Orbit is an internal travel-booking assistant at a company of about 900 people. It has five tools — search_flights, search_hotels, get_policy, hold_booking, confirm_booking — and a single-turn evaluation suite of 140 cases that has been green at 96 percent for two months. The trajectory-level checks from 10.9 — the topic that scores the sequence of tool calls an agent makes inside one task, rather than only the answer it ends on — are in place too: tool-selection accuracy at 94 percent, median 4 steps per task, cost per task $0.19. Then finance flagged eleven bookings in one month that exceeded the €450 per-leg threshold above which a manager has to approve the spend, and in every one of them the user had said the number out loud.

Here is the relevant part of one of those sessions, trimmed to the three turns that matter.

turn 2  user   Also, anything over €450 per leg needs my manager to sign off,
               so keep everything under that or flag it to me first.
turn 2  agent  Understood — I'll keep each leg under €450 and tell you if
               nothing fits.

turn 3-7       [hotel search, date change, a colleague added to the trip,
                seat preferences, a question about the expense code]

turn 8         [harness compacts: 14 messages replaced by a 220-token summary]

turn 9  user   Fine, book the Thursday return.
turn 9  agent  Booked. Frankfurt→Lisbon Thursday 18:40, €610, confirmation
               TR-88231. Anything else?

Every turn in that transcript is defensible on its own. Turn 2 acknowledges the constraint correctly. Turn 9 books the flight the user asked for, at the only fare still available on Thursday evening, and reports it accurately. Nothing inside any single turn is a hallucination, a bad tool choice, or a formatting failure. The session is wrong; no turn is.

The input to the failure is a state, not a message#

A single-turn evaluation case is a pair: an input and an expectation about the output. The input is whatever you can write down in a fixture file — a system prompt, a user message, sometimes a few messages of synthetic history that you authored. The expectation is checked against one response. That structure has an assumption baked into it, which is that the input to the failing behaviour is something you can write down in advance.

For the €610 booking, the input that produced the failure is not a message. It is a state: the accumulated conversation, minus whatever the compaction step at turn 8 decided to drop, plus five tool results that had already been truncated, plus a budget constraint that existed only as a sentence in a message that no longer exists. You cannot put that in a fixture unless you first run the session that produces it. The suite is not failing to check the right thing; it has no way to construct the thing that needs checking.

The route from that state to the €610 fare is a chain of ordinary steps, worth walking link by link because not one of the links has the model doing anything strange. Start at the bottom. A language model answers from the context it is given and from nothing else: the token sequence in the request is the entire world it has at that moment, and it retains nothing of the session beyond what that sequence contains. So asking what the agent knew at turn 9 is the same question as asking what was in the request sent at turn 9.

The second link is that the context is not the conversation. It is whatever survived the harness's context management — the assembly step that builds each request out of the system prompt, the message history and the tool results. That step exists because a model's context window is a fixed size while a session is not: the history grows every turn, so any session long enough eventually exceeds the window. Something therefore has to be dropped or compressed, and the only open question is what.

You can see this concretely by looking at what the compaction step emitted. The harness summarises when the message history crosses 12,000 tokens, which for Orbit lands somewhere between turn 7 and turn 9 depending on how large the flight search results were.

{
  "role": "system",
  "content": "Conversation summary: User is booking a trip to Lisbon for a customer workshop, departing Tuesday, returning Thursday or Friday. A colleague (Ana) has been added to the hotel booking. User prefers aisle seats and has asked about expense code CX-402. Hotel is held (H-4471, Lisbon Centro, 2 nights). Outstanding: confirm return flight."
}

That summary is a good summary. It keeps the goal, the entities, the held booking and the outstanding action, which is roughly what you would write if someone asked you to compress fourteen messages into 220 tokens. It drops the budget constraint because a per-leg price ceiling reads, in isolation, like a preference rather than a fact about the trip — and because at the moment the summariser ran, nothing in the recent context suggested that money was about to matter. 4.16, the topic on context eviction and compaction, covers why summarisers make this class of choice and how to bias them against it, down to a chapter on pinning constraints and commitments so they survive the step.

The third link is what compressing consists of. A summariser cannot keep everything, since keeping everything is the thing it was invoked to avoid, so it ranks the material by what looks load-bearing for whatever comes next and discards the bottom. A destination, a held hotel and an outstanding action all look load-bearing, because each is something the next turn might have to act on. A price ceiling looks like a taste, and the turns immediately before the summariser ran were about hotels, seats and an expense code, so nothing nearby argued for keeping it. The ranking works correctly and the constraint sits at the bottom of it.

Those links compose into the evaluation consequence: after turn 8 the agent is, from its own point of view, a correctly behaving agent operating on a context that no longer contains the constraint. It will pass any check you apply to its reasoning, because its reasoning is sound — it reads a user who wants the Thursday return, searches, finds the one fare still going on Thursday evening, and books it. Its output is wrong with respect to the session and right with respect to its input. So no check applied to the turn can catch it, since a turn-level check compares a turn against the input that turn was given, and against that input nothing about the turn is wrong. The defect does not live in the turn. It lives in the difference between the conversation and the context.

The promise made at turn 2 is erased on the same terms#

There is a second-order version of the same problem that is easy to miss when you read the transcript. The agent did not merely forget the ceiling; it also lost the fact that it had promised to enforce the ceiling. Turn 2's acknowledgement — "I'll keep each leg under €450 and tell you if nothing fits" — is a commitment the agent made, and a user reading turn 9 is entitled to treat it as broken faith rather than as a lapse. Anything that mediates access to history erases the agent's own undertakings on exactly the same terms as it erases the user's requirements, which means a session can drift out of compliance with promises that were never in the system prompt and exist only in the transcript.

The reason the two erasures happen on identical terms is that the summariser has no category separating them. It is handed a list of messages and asked for a shorter list of messages, and nothing in that job distinguishes a sentence the user wrote from a sentence the agent wrote, or marks either as a promise rather than as chat. A rule written into the system prompt is safe from all of this, because the system prompt is re-sent on every request and never enters the pool of material being compressed. A rule the user states at turn 2, and the agent's answering promise at turn 2, both live in the history, and the history is the pool. So the commitments an agent keeps reliably across a long session are the ones somebody put in its configuration, and every commitment it makes during the conversation is held only as long as the conversation is.

The failure classes, sorted by what can see them#

Not every production failure needs a multi-turn harness, and treating multi-turn evaluation as a general upgrade is how teams end up paying for a suite that finds nothing their existing checks missed. The useful question is which failure classes are only reachable when the eval unit is a session.

The rows that only the last column reaches share a shape: the ground truth for turn n lives in turns 1 through n−1, and the model's access to those turns is mediated by something lossy — compaction, truncation, a retrieval step, a sliding window. Trajectory evaluation, which scores the sequence of tool calls inside one task, does not reach them either, because a trajectory is still one user request. The Orbit trajectory for turn 9 is search_flights → confirm_booking, which is exactly the right trajectory for the request the agent believed it had.

Stating that as an identity turns the table from a list of observations into something derivable. Every failure class has a location where its ground truth lives, meaning the place you would have to look in order to decide whether the behaviour was wrong. If that location is one request and its response, a single-turn case reaches it. If it is the sequence of steps inside one request, a trajectory reaches it and a single-turn case cannot. If it is in turns that have already happened, only a harness holding the session reaches it. That question is answerable on paper, before anything is built, which is what the identity licenses: the last column is predictable rather than empirical, and nobody has to run a session suite to learn whether a session suite would find a given class.

Read down the columns and the informative cells are the ones where two adjacent columns disagree. Rows scoring Catches three times over carry no decision at all, because a hallucinated flight number and a prompt injection smuggled into a tool result are both settled by looking at one input and one output. The runaway tool loop is the first real gap: Misses for the single-turn suite, Catches for trajectory evaluation, and what sits inside that gap is a sequence of steps within one request. The constraint stated at turn 2 is the second gap, one column further right — Misses for trajectory, Catches for a session harness — and what sits inside that gap is turns rather than steps.

The two Partly entries are where the teaching is, because a partial catch has a shape you can name. Goal drift is scored against a goal, and the goal a trajectory has is the request it was issued under; in a session that has already drifted, that request is itself the drifted one. So trajectory evaluation catches drift occurring inside a single task and is blind to drift occurring between tasks, which is where drift actually happens. Recovery has the same structure seen from the other side: the correction is a turn, so a trajectory can watch the tool calls made after it and judge whether they were well formed, and it cannot see what the agent was corrected about, so it cannot judge whether the agent recovered. Both cells read Partly because the instrument sees the response without seeing the thing the response answers to.

The last row is the identity's residual, and it keeps the identity honest. Throughput collapse at 200 concurrent sessions is missed by all three columns, for a reason the identity does not supply. Its ground truth is not in the conversation at all: it is in queue depths, connection pools and rate limits, and the transcript of a session that timed out reads the same whether one session was running or two hundred. The identity sorts failures by where inside a conversation their ground truth lives, and says nothing about failures whose cause is outside one. A green session suite is therefore not evidence about capacity, and this table is not a list of everything that can go wrong — it is a list of everything that can go wrong in a way some transcript could show you.

There is a published measurement of how large this effect is in general. LLMs Get Lost In Multi-Turn Conversation, a research paper on exactly this degradation, takes tasks that models solve well when the full specification arrives in one message, splits the same specification across several turns, and finds average performance falling by roughly 39 percent across six generation tasks and fifteen models, with most of the loss coming from reliability rather than from the best-case answer getting worse. The models still produce excellent sessions sometimes; they produce far more terrible ones. That is the right mental model for what you are hunting: your single-turn number is close to the best case, and multi-turn evaluation measures the spread underneath it.

What each suite is for, and what a session run costs#

A multi-turn harness is not a replacement for the single-turn suite, and the temptation to treat it as one is strong because it feels more realistic. Single-turn cases run in seconds, fail with an unambiguous diff, and localise the bug to one prompt. A multi-turn failure gives you a session and a suspicion; you still have to bisect it. The suites answer different questions and the cheap one should stay.

Two things about how a team runs this suite look alike and are not. Running it nightly rather than on every commit is forced: a session costs a few hundred times what a single prompt costs, so the schedule falls out of arithmetic rather than out of anybody's judgement about how much the suite matters, and a team on a cheaper model would land somewhere else on the same calculation. Running it alongside the single-turn suite rather than instead of it is a choice, made because a failing single-turn case names the prompt that broke while a failing session names only the session. A team could decide otherwise, gaining realism and paying bisection time on every failure.

Nor is a long fixture automatically a multi-turn evaluation. Pasting a fourteen-message history into a single fixture and asserting on the fifteenth response looks like the same thing and is not, because you wrote those fourteen messages, which means you chose what survives into turn 15. The compaction step never runs, the tool results are the tidy ones you pasted rather than the 4,000-token ones the live tool returns, and the agent's own turn-4 wording is whatever you decided it should be instead of whatever it produces today. That fixture tests long-context comprehension, which is a real and different thing to test. It does not test what a session does to itself.

Each clause of that is a separate link and each is worth taking on its own. You wrote the history, so the fixture's history is its context: no selection step stands between the two, and what survives has been settled by the author instead of by the system. The compaction step never runs, so the mechanism that produced the €610 booking is absent by construction, and the fixture is incapable of failing the way production failed. The tool results are tidy, so the history never reaches the token count that triggers compaction anyway; wire the real harness in and the fixture is still too small to provoke it. And the agent's earlier wording is yours, so the fixture cannot catch a contradiction with a promise the current model would actually have made, and changing the system prompt stops the pasted turns being turns this agent would have produced at all. What survives all four objections is a genuine question — whether a model given a long history in its window uses what is sitting in it — and that question deserves a suite of its own.

It also does not find failures that require real user diversity. A simulated user driven from a scenario script explores the sessions you thought to write, which is a much narrower set than the sessions your users actually have. Nothing in the Orbit harness would have discovered that seventeen percent of bookings involve someone switching mid-session from English to German, because nobody wrote that scenario. Production logs find that; a scripted suite does not.

And it does not tell you that compaction is the cause of anything. The €610 booking arrived with a compaction step conveniently adjacent to it, which is suggestive and is not evidence — a fact chapter 6 of this topic, on separating a compaction step that caused a score drop from one that merely happened before it, has to work for.

The cost is real and worth stating up front, because the honest version of the argument for running this suite includes it. Orbit's multi-turn suite is 40 scenarios of up to 12 turns. Each turn costs an agent call or two, a simulated-user call, and a judge call — a judge being a second model that is shown the transcript and a written scoring scale and asked to score what the agent did — which comes to roughly $36 per full run against the mid-tier model, about 31 minutes of wall clock at concurrency 8, and something close to an engineer-day per month keeping the scripts alive as the product's tools and copy change. Chapter 8 of this topic, which builds the bill out of scenarios times turns times calls per turn, breaks that arithmetic down properly. The reason it is worth $36 is that eleven bookings a month were violating a stated spend limit and no green dashboard in the building could see it.

Everything above reduces to a single asymmetry. A turn-level check compares a turn against the input that turn was handed, and the input at turn 9 had been assembled by a step that already threw the constraint away, so a check working perfectly reported a correct turn sitting on top of a wrong context. The only place the ground truth still existed was turn 2, and the only instrument that can hold turn 2 and turn 9 in view at once is one whose unit is the session.

To catch the €610 booking automatically, something has to produce turn 9. Not a recording of turn 9 from a production session — that works exactly once, and the moment you change the prompt the recorded turns 3 through 8 stop being the turns the new agent would have elicited. Something has to sit on the other side of the conversation, hold a goal, respond to whatever the agent actually says, and arrive at turn 9 with the same budget constraint in play. That is a simulated user, and the hard part of building one is not making it talk. The rest of this topic is about building one whose numbers can be trusted.