Multi-turn and conversational evaluation

Read these first

• Agent evaluation
• Context engineering — eviction and compaction

What you will know

• Build a simulated user to drive a multi-turn conversation

  The harness needs seeded goals, realistic interruptions, and deterministic replay; otherwise regressions are indistinguishable from user randomness.

• Find the turn at which quality collapses

  Plot success or rubric score by turn over fixed scenarios to separate a late-session failure from ordinary evaluation variance.

• Measure what compaction cost the conversation

  Compare identical traces with and without summarization, including token cost, latency, omitted facts, and downstream task success.

• Demonstrate a failure that passes single-turn checks but fails in a multi-turn replay

  A long-session fixture should expose state or context dependencies that isolated prompts never exercise.

• Report a multi-turn success rate

  A single aggregate hides late-session degradation, so report end-to-end and per-turn rates with scenario counts and uncertainty.

Chapters

1. 1

   A constraint stated at turn 2 and broken at turn 9

   This is the failure class single-turn evaluation structurally cannot reach: the user states a constraint early, the session continues, compaction runs, and the agent violates it later without ever contradicting itself inside any one turn. Every isolated prompt in your suite still passes. Building the long-session fixture that exposes these state and context dependencies is what the rest of this topic is paying for.

   Read chapter
2. 2

   A simulated user is only useful when it is reproducible

   Driving a conversation automatically is easy; driving the same conversation twice is the actual requirement. The harness needs a seeded goal, a fixed scenario script, realistic interruptions and deterministic replay, or else a genuine regression and a randomly worse simulated user look identical in the results. Here you build one and pin down exactly which parts have to be frozen between runs.

   Read chapter
3. 3

   A simulator sharing the agent's model shares its blind spots

   An LLM given a persona and a goal, reporting 87% completion over 30 scenarios, is not evidence, and the reasons are specific: it accepts answers a real user would push back on, it misses the omissions the agent also misses, and 30 scenarios carry an uncertainty range wide enough to swallow the claim. What would make it evidence is human-anchored scenarios, adversarial variants, and a different model driving the user.

   Read chapter
4. 4

   Scoring a turn while the task is still half finished

   End-state assertions yield nothing at turn four of twelve, so a per-turn number has to be built out of something else: adherence checks against constraints stated in earlier turns, goal-progress judgements seeded by the scenario script, and per-turn assertion sets written alongside that script. A judge rubric with anchors fixed before the run supplies the rest, and someone has to own who or what emits each score. This is the quantity the next two chapters plot and compare, so an undefined y-axis here makes both of them unreadable.

   Read chapter
5. 5

   Quality collapses at a turn index you can find on a plot

   Plot the per-turn instrument from the previous chapter across fixed scenarios and look for a drop at one index larger than the run-to-run variance at its neighbours; anything smaller is noise being read as a trend. That is how a genuine late-session failure gets separated from ordinary evaluation spread, and it hands you the specific turn number that the next chapter has to explain.

   Read chapter
6. 6

   Compaction fires at turn 8 and the score drops: prove causation

   Coincidence at a single boundary is not a finding. Run identical scenarios with and without summarisation, comparing task success, omitted facts, tokens and latency, then move the compaction threshold and see whether the drop moves with it. Switching compaction off is not the fix either, since it relocates the failure to the context limit; the outcome is a measurement of what summarisation costs and what must survive it.

   Read chapter
7. 7

   One aggregate number hides everything after turn six

   A single multi-turn success rate can stay flat while late-session behaviour falls apart, because most scenarios end before the damage occurs. Report end-to-end completion alongside per-turn scores, with the scenario count and an uncertainty range attached to both. What that report looks like when someone asks whether a change helped, and how few scenarios make the answer unknowable, are settled here.

   Read chapter
8. 8

   Scenarios times turns times calls is the whole bill

   Price the suite before you defend it: scenarios multiplied by turns multiplied by model calls per turn, plus the simulator's own token spend, plus the judge's, plus wall-clock per suite run and the standing maintenance cost of fixed scripts that rot as the product changes. That figure decides how many scenarios you can afford, which is the other side of the previous chapter's question about how few make a result unknowable. Naming the failure class multi-turn evaluation reaches is only half of the argument that it is worth running.

   Read chapter