Shadow traffic cannot observe anyone reacting to the answer

The candidate configuration has been running against mirrored production traffic for seven days. Every inbound request was copied to it, its response was scored by the same rubric the offline suite uses — the written criteria a grader applies to an output, one item at a time — and on the 12,000 sampled requests the candidate scored higher than production on 68.3%, tied on 9.1% and lost on 22.6%. Nobody was exposed, nothing broke, and the deck for the review has a single chart on it.

The proposal is to skip the A/B and ship. The objection is one sentence long: none of those 12,000 responses was ever shown to a human being, so nothing in this data set contains a person deciding whether the answer was any good.

What the primary metric requires that a shadow run cannot supply#

Task completion is defined as a resolved status set within 24 hours of the last user turn. Resolution happens because a user read an answer, believed it, acted on it, and stopped asking. Every step of that is a reaction, and a reaction requires an audience. In a shadow run the user is looking at production's answer while the candidate's answer goes straight to a log table, so the entire causal path from output to outcome is missing.

The same holds for every behavioural signal the team relies on. Retries are absent, because a user who gets a bad answer retries against production and the candidate never sees the reformulated query. Escalation to a human is absent for the same reason. Abandonment is absent. Thumbs-down is absent. Follow-up turns are absent, which is worse than it sounds, because it means a multi-turn shadow run after the first turn is being fed a conversation history that production produced, so the candidate is answering turn 4 of a conversation it did not shape — the same contamination described in chapter 1, in a setting where it cannot be fixed by choosing a different unit.

What remains is a rubric score, and the rubric is the artifact the candidate was developed against. A prompt engineer iterating for two weeks against a rubric will produce something that scores well on that rubric, and 68.3% is partly a measurement of that iteration. The offline suite's opinion is a useful prior and it is not the thing the guardrails and the primary metric were declared over.

The three ways a shadow run causes an incident#

Mirroring looks like a read-only operation and is not, because the candidate runs the same code paths as production and those paths do things.

The first hazard is side effects. This assistant has tools: it can create a Zendesk ticket, issue a refund up to $50, and send a password-reset email. A shadow deployment that forwards requests into the normal agent loop will execute those tools for real, and a week of shadowing at full volume on 8,000 daily requests filed several thousand duplicate tickets before anyone noticed. The correct configuration routes every tool call through an executor that refuses anything not on an explicit read-only allowlist.

READ_ONLY_TOOLS = {"search_kb", "lookup_order", "get_account_status"}

async def shadow_tool_executor(call: ToolCall) -> ToolResult:
    if call.name not in READ_ONLY_TOOLS:
        # Return a plausible-shaped result so the loop can continue,
        # and record that fidelity was lost at this step.
        shadow_metrics.increment("tool_suppressed", tool=call.name)
        return ToolResult(
            content=SYNTHETIC_RESULTS[call.name],
            suppressed=True,
        )
    return await execute(call, timeout=3.0, budget=call.remaining_budget)

Suppression buys safety and costs fidelity, and the trade is not small. The moment the candidate's trajectory takes a suppressed branch, everything after it is a simulation, so the suppressed flag has to travel into the analysis and those requests must be reported separately. On this feature, 14% of shadow requests hit a write tool, and comparing the candidate to production on those 14% is comparing a real run to a stubbed one.

The second hazard is data movement. The candidate here runs against a different model endpoint, and the default retention on that endpoint keeps prompt and completion payloads for 30 days in a region the production system does not use. Mirroring full production traffic into it moves customer support transcripts — names, order numbers, occasionally card fragments — into a system that no data processing agreement covers. Shadowing is a copy of production data by definition, and it deserves the same review that a new production dependency would get: retention set to zero or to the production value, the region pinned, the redaction pipeline applied to the mirrored copy rather than only to the served path, and the mirror disabled for any traffic from tenants whose contracts prohibit sub-processing.

The third is dependency load. Every shadow request hits the retrieval index again, and at 100% mirroring that doubles the vector database's queries per second. On this system the index p99 went from 40 ms to 95 ms under the doubled load, which degraded production, because the shadow path shares the cluster. The mirror is also spending real inference money: 8,000 requests a day for seven days at $0.028 is $1,568 for the week, on a system serving nobody. Sampling at 10% and pinning the shadow path to a separate index replica costs a week of extra elapsed time and removes both problems.

The fire-and-forget arrow is load-bearing. If the mirror is awaited, a slow or failing candidate becomes a production latency incident, which is the one failure mode shadowing exists to prevent. The mirror gets its own timeout, its own circuit breaker, and a hard rule that its errors are counted and never raised.

What the week of shadowing was genuinely worth#

The 68.3% is the least interesting number the run produced. The useful findings were elsewhere.

On 3,100 of the 12,000 sampled requests the candidate produced a response that was semantically equivalent to production's, meaning 74% of traffic would see no change at all; the entire blast radius of this change is the remaining quarter, which reframes the risk conversation and tells the reviewers which requests to read. Eleven requests produced a schema violation in the structured output that the offline suite never triggered, all of them on queries containing a particular Unicode quote character that appears in traffic from one locale and in none of the golden examples. The p95 latency on real inputs was 3.2 seconds against the offline suite's 2.4, because real queries are longer and retrieve more passages. Spend came out at $0.028 per request against the $0.021 baseline, which is the number chapter 3's guardrail is checked against, available before any user was exposed. And the safety classifier fired on 0.44% of shadow responses against 0.20% in production, which is the same signal the A/B later confirmed, obtained a fortnight earlier and without serving any of it.

That list is the honest description of the technique: it derisks exposure. Shadowing is how you find out that the candidate crashes on 0.09% of real inputs, exceeds the cost guardrail, or trips the safety classifier at twice the rate, before the first user meets it. Every one of those findings would otherwise be discovered by an A/B test that spends two weeks and exposes tens of thousands of people.

Reading the diff rather than the score#

The output of a shadow run that repays attention is a diff, and diffs need a taxonomy before they need a metric. Sorting each pair of responses into byte-identical, semantically equivalent, materially different in content, materially different in structure, and candidate error turns 12,000 rows into five numbers that a review meeting can act on. Byte-identical and semantically equivalent responses need no review at all, which on this run removed 74% of the volume. Structural differences — a changed citation format, a list where there was a paragraph — go to whoever owns the surface. Content differences are the population for human reading, and errors are a blocker regardless of how the rest scored.

Classifying the middle categories requires a judge, usually an LLM judge or an embedding-similarity threshold calibrated against a few hundred human labels, and the calibration is what makes the numbers usable. An unverified similarity threshold will happily declare two answers equivalent when one of them omits the refund deadline, and the entire value of the 74% figure depends on that not happening. Spending a day labelling 300 pairs by hand to fix the threshold is the difference between a diff taxonomy and a comforting chart.

Sampling for human review then goes where the disagreement is rather than uniformly. Drawing 60 conversations from the materially-different bucket, stratified across the traffic slices that matter, gives reviewers a set where every item is informative, whereas a uniform sample of 60 from all traffic would contain 44 identical pairs and waste most of the session. The reviewers are answering a question the rubric cannot: on the requests where this change does something, is what it does an improvement.

How long to shadow follows from what is being looked for. Crashes and schema violations on rare inputs need enough volume to reach the tail, and a week at 10% sampling on this feature was 5,600 requests, which found an error occurring at roughly 1 in 500. Cost and latency stabilise in hours. The diff taxonomy stabilises in a day or two. Extending a shadow run past the point where its numbers have stopped moving buys nothing, and the instinct to leave it running for another week is usually an instinct to delay the exposure decision rather than to gather evidence.

The neighbours it gets mistaken for#

Shadowing is not a canary. A canary from the previous topic serves real responses to a small share of real users, so it produces exactly the reaction data that shadowing cannot, at the cost of exposing people. The two are complementary in a fixed order: shadow to find the crashes and the cost, canary to find the reactions at low blast radius, A/B to measure the effect.

It is not a load test either, despite doubling traffic. A mirror reproduces the current load pattern, not the peak, and it cannot tell you what the candidate does at three times volume. Nor is it a replay harness: replaying a recorded corpus is reproducible and comparable across runs, while live mirroring sees today's traffic and never the same request twice, which is why the two get used for different jobs — replay for regression, shadow for distribution.

The deepest limitation is that a shadow run scores itself. Its verdict comes from a rubric, an LLM judge, or a diff heuristic, all of which are the same offline machinery applied to fresher inputs. When that machinery and a live experiment disagree about the same change, something has to decide which one the ship follows.