Quality collapses at a turn index you can find on a plot

A per-turn quality curve plots one number for each position in a conversation: the average score the agent's reply received at turn 1, at turn 2, and on to wherever the longest conversations end. The score comes from a turn-quality rubric, a fixed scheme a grader applies to each reply, here on a 0 to 4 scale where a high number means the turn did what it should have and 0 is where a reply lands after violating a constraint the conversation had already established. A scenario is one scripted conversation an evaluation suite runs end to end, so the suite yields one rubric score per scenario per turn, and the curve is the average down the scenario axis, read across the turn axis.

The curve locates a failure that a single overall average cannot. An agent that is fine for six turns and useless at the eighth can carry the same suite-level mean as one that is mildly worse everywhere, and the two need completely different fixes. The difficulty is that every such curve slopes downward somewhere, and every column of turn-to-turn differences has a largest negative entry in it whether or not anything is wrong. Separating a real cliff from the largest wiggle a set of runs happened to produce is a question about which spread you divide by.

The drop at turn 8 and the dip at turn 4#

Five runs of the Orbit suite, same commit, same 40 scenarios, same cassettes — recorded tool responses replayed instead of live calls, so the environment answers identically every time — different only in the agent's own sampling. Averaging the turn-quality rubric across scenarios within each run and then across runs gives this.

The turn-4 dip of −0.09 and the turn-8 drop of −0.66 are both negative numbers in a column, and only one of them is a finding. The difference is the SD column next to them, which holds the standard deviation of the five run means at each turn — how far a typical run's number falls from the average of the five, expressed in rubric points, so that 0.06 at turn 1 says the five runs agreed closely there. A step of 0.09 sits inside the run-to-run spread at turns 3 and 4, which means a different set of five runs would as easily have produced +0.05 there; a step of 0.66 is more than six times the spread at turns 7 and 8, and no plausible resampling of those runs produces it by accident.

None of the levels in that table carry the finding. That turn 1 sits at 3.72 and turn 8 at 2.94 is a joint property of how strictly the rubric was written and which 40 scenarios somebody authored, and a harsher grader would push the whole column down while changing nothing about the agent. What survives a change of grader is the difference between two adjacent rows, since a constant added to every cell subtracts out of every entry in the Δ column. The table therefore teaches through one gap, turn 7 at 3.60 against turn 8 at 2.94, and the other rows are there to establish what an unremarkable gap looks like.

The comparison is against between-run spread, not between-scenario spread#

There are two very different variances available in this data and picking the wrong one makes the analysis either paranoid or credulous.

Between-scenario spread is how much the rubric score varies across the 40 scenarios within a single run. At turn 8 that standard deviation is 1.08, because some scenarios are easy and some are not, and a few score 0 on a single violated constraint. Comparing the 0.66 drop against 1.08 makes it look like noise, and a team that does this concludes there is nothing to see and stops.

Between-run spread is how much the mean over the same 40 scenarios varies when you re-execute the suite. That is the 0.12 in the table, and it is the right denominator, because the question being asked is whether re-running would produce a different answer, not whether every scenario behaves the same. The scenarios are held fixed by construction, so their difficulty cancels out of the comparison entirely — the same reason a paired test is stronger than an unpaired one on the same data.

Reaching for 1.08 instead of 0.12 is not carelessness, and the pull is worth naming. It is a genuine standard deviation computed from genuine measurements, it is available from one run rather than five, and it is the larger of the two, so using it feels like the conservative choice — the one that refuses to declare a finding on thin evidence. What is wrong with it is that it answers a question nobody asked. It reports how much the scenarios differ from each other, and scenario difficulty is precisely what a fixed scenario set removes from the comparison. Judged against it, the turn-7-to-turn-8 step has to be large next to the gap between an easy scenario and a hard one, a bar the step has no reason to clear and one that says nothing about whether a re-run would reproduce it.

The cancellation is worth writing out, since it is the whole reason the smaller number is the honest one. Treat a scenario's score at a given turn as three things added together: how hard that scenario is, whatever the turn position itself does to quality, and a wobble contributed by that run's sampling. Averaging over the scenarios inside one run leaves the difficulty term intact, which is exactly why the level of any row means nothing on its own. Now subtract the turn-7 mean from the turn-8 mean. Each scenario contributes its identical difficulty amount to both sides, once with a plus and once with a minus, so that term leaves the difference entirely and only the turn effect plus a much smaller wobble remains. That is what makes a comparison paired: subtract the same units and everything constant about a unit disappears — which holds exactly only while the set of active scenarios is the same on both sides, a condition the last section of this chapter checks rather than assumes.

That wrong denominator is dangerous rather than merely wrong, and the danger has a shape. Between-scenario spread is dominated by how much the authored scenarios differ from one another in difficulty, and that difference is a property of the set rather than of the agent, so it sits at about the same value at turn 3 as at turn 8 whatever the agent does in between. A degradation striking one turn index moves the means and barely disturbs the spread of a population that was already widely spread. So a team watching that number sees something stable and reassuring while the property they are hunting varies completely underneath it. Watching it feels like a defence against over-reading noise and defends against nothing; it is a measurement of the suite's own composition, re-reported every turn.

import statistics as st

def per_turn_means(runs):           # runs: list[dict[turn, list[score]]]
    return [{t: st.fmean(v) for t, v in run.items()} for run in runs]

def drop_report(runs, turn):
    means = per_turn_means(runs)
    before = [m[turn - 1] for m in means]
    after = [m[turn] for m in means]
    delta = st.fmean(after) - st.fmean(before)
    # Paired across runs: the same scenarios sit on both sides.
    diffs = [a - b for a, b in zip(after, before)]
    sd = st.stdev(diffs)
    return {
        "turn": turn,
        "delta": round(delta, 3),
        "sd_of_paired_delta": round(sd, 3),
        "ratio": round(abs(delta) / sd, 1),
    }

drop_report(runs, 8)   # {'turn': 8, 'delta': -0.66, 'sd_of_paired_delta': 0.09, 'ratio': 7.3}
drop_report(runs, 4)   # {'turn': 4, 'delta': -0.09, 'sd_of_paired_delta': 0.11, 'ratio': 0.8}

The two calls at the bottom put both candidates through the same division. At turn 8 the paired standard deviation of the five run-level differences is 0.09, and 0.66 divided by 0.09 is 7.3. At turn 4 that standard deviation is 0.11 while the step is 0.09, and 0.09 divided by 0.11 is 0.8. A ratio near 1 says the step is about the size of the amount this number moves when nothing has changed, so a fresh set of runs would as easily have produced a step of the opposite sign — the whole verdict on turn 4. A ratio of 7.3 says the step is seven times that, and no resampling of these five runs reaches it. The ratio is the comparison; the delta alone, however large in rubric points, is not.

Pairing on run and on scenario tightens it further, and it is worth doing when the drop is smaller than this one: compute each scenario's own turn-7-to-turn-8 delta, then test whether the mean of those 36 deltas differs from zero. For the Orbit data that gives a mean delta of −0.65 with a standard error of 0.11 across scenarios — a standard error being the spread of an average rather than of the individual values behind it, and therefore the number that says how far the average would move if the whole exercise were repeated — so the drop survives even without appealing to repeated runs. The rule of thumb Orbit's team wrote down is that a step has to exceed three times the paired standard deviation before anyone opens an investigation, which at this suite size means about 0.3 rubric points. Anything smaller goes on a watch list and gets re-measured with more runs rather than explained.

The 0.11 in that per-scenario test is worth unpacking, because it is what makes one run's data sufficient. The standard error of an average is the spread of the individual values divided by the square root of how many were averaged. Run that backwards: the square root of 36 is six, so a standard error of 0.11 implies the individual per-scenario deltas are spread by about six times that, roughly two-thirds of a rubric point. That is enormous beside a mean delta of −0.65. Scenarios disagree violently about how far they fall at turn 8, some barely moving and some collapsing, and the average is still pinned about six of its own standard errors from zero, because averaging 36 disagreeing numbers shrinks the uncertainty in their mean by a factor of six while leaving the mean where it was.

Five runs is the smallest number that produces a usable spread estimate and it is not a comfortable one, because a standard deviation computed from five values is itself uncertain by roughly ±30 percent. That uncertainty is tolerable when the effect is 7σ — sigma is the conventional symbol for a standard deviation, so 7σ names an effect seven times the size of the ordinary run-to-run wobble — and it is not tolerable when the effect is 2σ, which is the case in which teams most want a verdict. The practical policy is to size the number of runs to the size of the effect you need to detect: five runs to confirm something obvious, and ten to fifteen before believing a 0.2-point step. Each additional run of Orbit's suite costs about $36 and half an hour of wall clock, so the choice between five and fifteen is a $360 question rather than a project.

Two of the three rules just described are decisions and one is not, and the distinction matters as soon as any of them travels to another suite. The square-root behaviour of a standard error is nobody's choice: halving the width of an interval takes four times the runs, and no policy negotiates that away. The bar of three times the paired standard deviation is a decision Orbit's team wrote down, and a team that wrote two, or four, would have been equally defensible; it buys fewer investigations opened on nothing and costs real steps below about 0.3 rubric points sitting on a watch list instead of being chased. The sizing rule is the same kind of decision with its price stated in money rather than in missed findings. Only the square root survives a change of suite, of rubric, or of team.

Setting a threshold at a multiple of the ordinary spread is the same logic as a control chart on a manufacturing line, and the borrowing is deliberate. A control chart establishes what ordinary variation looks like when nothing is wrong, draws limits at a multiple of it, and treats only excursions beyond those limits as signals worth acting on — precisely so that operators stop adjusting the machine in response to noise, which makes output worse. An evaluation suite has the same failure mode: a team that responds to every 0.05 wiggle in a per-turn curve will spend its quarter chasing sampling variation and shipping prompt changes that do nothing.

The analogy has a limit, and knowing where it fails separates borrowing a control chart's vocabulary from adopting its conclusions. A control chart assumes a stable process turning out independent items and sets its limits from that process's own history, so a point outside them is evidence about the machine. An evaluation suite's scenarios are a fixed authored set rather than a stream: they did not arrive, they were written, and they are the same forty every time. Limits computed by re-running that set describe how reproducibly the suite measures, and say nothing about how the product behaves for a user who is not one of the forty. A step that clears them is a real property of the agent on that set; a step that fails to clear them is not evidence that anybody's sessions are fine.

A cliff and a slope are different phenomena#

The shape of the curve carries information beyond the location of the largest step. Turns 1 through 7 in the table decline gently, about 0.02 per turn, which over seven turns is 0.14 and is barely outside the noise. That gentle slope is the ordinary cost of a growing context: more history to attend to, more opportunities for an earlier error to propagate, longer prompts, a task that is genuinely harder in its later stages. It is the effect LLMs Get Lost In Multi-Turn Conversation, a published study measuring how model performance falls as a conversation is extended, reports at scale, and it is not fixable by any single change; it is the background rate of a longer conversation being harder.

A cliff at one index is a different animal, because a discrete drop implies a discrete cause. Something happens between turn 7 and turn 8 that did not happen between turn 6 and turn 7. The candidates are a short list: a summarisation or eviction step firing at a threshold, a context window boundary, a tool whose result size crosses a truncation limit, a state machine in the agent transitioning, a retrieval step whose recall falls once the index has more than k candidates, or a rate limiter degrading the model to a smaller variant. Which of those it is remains open at this point; that the curve has a cliff rather than a slope is what makes the question answerable at all.

The inference from shape to cause is a short chain and each link is checkable. Everything on the slope list grows smoothly with the conversation: history length, prompt size, the accumulated chance that an earlier mistake is still in scope. A quantity growing by about one turn's worth per turn, feeding a model whose behaviour changes gradually with it, produces a decline that is gradual too. A step in the output therefore requires something in the pipeline that is itself a step — a threshold crossed, a branch taken, a limit hit — since nothing smooth on the input side can manufacture one. That is what narrows the candidates to mechanisms with a boundary in them, and why the plot is worth making before anybody starts reading transcripts.

Two ways this plot lies#

The denominator changes down the table, and that is a composition effect waiting to be misread. At turn 8 the mean is over 36 scenarios; at turn 12 it is over 26, because 14 scenarios finished earlier. If the scenarios that finish early are the easy ones, the late turns are averaging over a harder subset, and part of any decline is the population changing rather than the agent degrading. The check is cheap: restrict to the balanced panel of scenarios that reach turn 12 and re-plot.

The identity underneath that check both licenses it and limits it. A mean over an active subset is a statement about that subset and nothing else. The 2.94 at turn 8 is the average quality of the 36 scenarios still running at turn 8; it is not the average quality of the suite at turn 8, and the suite has no quality at turn 8, since four of its scenarios had already ended. The balanced panel follows immediately: hold the population fixed down the table, and any decline that remains cannot be the population changing, because the population is no longer changing.

The cliff survives at 0.67 on the balanced panel, so composition is not the explanation here. Had the two columns diverged, the balanced panel would be the one to believe, at the price of a smaller sample and a suite that only measures long scenarios.

Both halves of that price are derivable. The first is the square root again: 26 scenarios rather than 36 makes the standard error on every cell of the balanced-panel column larger by the ratio of those two square roots, so the column that settles the composition question is also the column least able to resolve a small step. The second is worse. Scenarios reaching turn 12 are long scenarios, and a suite restricted to them measures a population defined by the very property under investigation. Composition has not been removed; it has been pinned at one value, and the value chosen is the one where late-session failure is most likely to show. Both columns answer real questions, and neither answers what the agent's quality is, which is not a quantity that exists independently of a scenario set.

The second distortion is that turn index is not always the right x-axis. Orbit compacts on a token threshold rather than a turn count — compaction being the step that summarises or drops older messages once the conversation's token count crosses a limit, so that what reaches the model still fits — so the compaction step fires at turn 8 in most scenarios, turn 7 in the ones with large flight-search results, and turn 9 or 10 in the terse ones. Averaging on turn index smears a sharp event across three columns and makes the cliff look shallower and wider than it is. Re-plotting against event time — the offset from each scenario's own compaction step, so that −1 is the turn before it and 0 is the turn after — sharpens the drop from 0.66 to 0.81 and narrows it to a single index.

Every link in that is checkable, and the smearing is not a metaphor. Compaction fires on a token count, so the turn at which it fires depends on how quickly a scenario accumulates tokens. Tool results are most of what accumulates, and their sizes differ by scenario, since a flight search returns many rows while a booking confirmation returns a few lines. One event therefore lands at turn 7 in some scenarios and turn 9 or 10 in others, and the turn-8 column becomes a mixture of sessions just compacted, sessions compacted a turn ago, and sessions not compacted at all. Averaging a mixture divides one sharp step among however many columns the event is scattered across, lowering it in each and leaving several adjacent columns mildly affected. Aligning on each scenario's own event undoes precisely that mixing, since every value in the offset-0 column now comes from a session in the same state, so the whole step lands in one column. The drop growing to 0.81 and narrowing is the signature of un-smearing rather than of a new effect.

def align_on_event(sessions, event="compaction"):
    aligned = defaultdict(list)
    for s in sessions:
        pivot = s.event_turn(event)
        if pivot is None:
            continue                       # no event: excluded, and counted
        for turn in s.turns:
            aligned[turn.index - pivot].append(turn.rubric)
    return {k: st.fmean(v) for k, v in sorted(aligned.items())}

That re-plot is suggestive and it is not a causal claim, which is the honest limit of everything in this chapter. Aligning on compaction and finding a sharp drop at offset 0 tells you the drop is tightly coupled to whatever else happens at that moment in a session — and the compaction step is not the only thing that happens there, because a session at the compaction threshold is also a session with a long history, many tool results, and a user who has been going for eight turns. Every one of those is a candidate explanation and all of them are perfectly correlated with each other in observational data, meaning data gathered by watching sessions run as they normally do rather than by deliberately varying one thing and holding everything else fixed.

No amount of further sharpening repairs that, for a structural reason rather than a shortage of runs. Alignment makes the picture crisper by removing variation in when the event happened; it removes nothing about what else was true when it happened. Everything that co-occurs with crossing a token threshold co-occurs with it in every session, because crossing that threshold is what a long session does, so these runs contain no session where compaction fired over a short history. Distinguishing causes needs a case where the suspected one differs and the rest do not, and observational runs of a fixed suite cannot contain such a case by construction.

What separates the finding from the dip four turns earlier was never the size of either step. Both are small numbers in a column of small numbers, and −0.66 becomes a different kind of thing from −0.09 only after each is divided by how much its own turn's number moves when nothing has changed. That divisor is the spread of run means over a scenario set held fixed, and holding the set fixed is what lets scenario difficulty leave the comparison instead of swamping it.

What the plot has produced is a specific, falsifiable target: a 0.66-point drop, six to seven times the run-to-run noise, at the turn where summarisation fires, surviving the balanced-panel check and sharpening under event alignment. Turning that into a cause requires making the suspected mechanism vary while everything else stays fixed, which is an experiment rather than a plot.