Completion rate settles the argument that novelty cannot

A product manager brings you a slide about the assistant inside your analytics product. Of 8,900 sessions last month, 3,560 — forty percent — consisted of a single user message and then the user left. Her reading is that this is the feature working: users get what they need in one turn, no back-and-forth, maximum efficiency. Her proposal is to invest in handling more query types so that more sessions can resolve in one turn.

Two further cuts of the same data change what the forty percent means. Clustering those 3,560 messages puts 71 percent of them into five intents, the largest being some phrasing of "what happened last week" at 22 percent of all single-message sessions. And joining on user ID shows that 61 percent of the people who sent one of those messages sent a near-identical message in at least three of the previous four weeks. That is not a population getting an answer efficiently. It is a population manually re-requesting the same report on a weekly cadence, because the product has no way for them to subscribe to it. Every one of those sessions is a scheduled digest that has not been shipped, delivered by hand, by the user, forever.

So the answer to "is this success or failure" is that it is neither on its own; it is a signal about a missing product decision. The build is a weekly digest covering the top five intents, delivered where those users already are, plus the conversation retained for the twenty-nine percent tail that does not cluster. And the way to know whether that was the right call is to measure both, which requires deciding what to measure before you build either.

Four measurements, defined so both surfaces can be scored on them#

Completion rate is the fraction of attempts that reach the outcome. The definition has to be anchored on the outcome rather than on the response, and it has to be the same outcome for both prototypes or the comparison is meaningless. For this feature the outcome is "the user obtained the weekly numbers", which in the chat arm — an arm being one of the two versions the eligible users are split between, so that each population is measured on a surface rather than on an opinion — means a response containing them was delivered and the session ended without a repeat or rephrase, and in the digest arm means the digest was opened. Writing that definition down before either arm ships is what prevents each side from later scoring itself on its own most flattering event.

Time to result runs from the user's intent to the result being where they need it — not from request to response. That distinction is the whole reason the earlier work of tracing the before-and-after steps matters; a chat surface that answers in 3 seconds after 40 seconds of context assembly and 30 seconds of copying is a 73-second surface.

Correction effort is what the user does after the first result: rephrasings, follow-up narrowings, manual edits to the output, retries. It is the metric that catches a surface which looks fast because it produces something quickly and something slightly wrong.

Recovery after failure is the fraction of attempts where the first attempt failed and the user still reached the outcome. This is the one teams skip, and it is where surfaces differ most sharply. A form that rejects a value tells you which one and keeps the rest; a conversation that misunderstands leaves the user guessing which part of their sentence to change. Two surfaces with identical completion rates can have very different recovery rates, and the one with poor recovery is the one that generates support tickets.

Instrument the outcome, not the response#

None of the four can be computed from the events a chat surface emits by default, because those events end at the response. The schema needs an explicit outcome event that both arms can emit, plus enough correlation to join an attempt end-to-end.

{"event": "attempt_started",  "attempt_id": "a_9f21", "arm": "chat",   "user_id": "u_412", "intent": "weekly_summary", "ts": "2026-04-02T09:14:02Z"}
{"event": "result_delivered", "attempt_id": "a_9f21", "ts": "2026-04-02T09:14:39Z"}
{"event": "correction",       "attempt_id": "a_9f21", "kind": "rephrase", "ts": "2026-04-02T09:15:11Z"}
{"event": "result_delivered", "attempt_id": "a_9f21", "ts": "2026-04-02T09:15:26Z"}
{"event": "outcome_reached",  "attempt_id": "a_9f21", "ts": "2026-04-02T09:15:31Z"}
{"event": "attempt_abandoned","attempt_id": "a_77c3", "arm": "chat", "last_state": "awaiting_user", "ts": "2026-04-02T11:02:44Z"}

select arm,
       count(*)                                             as attempts,
       avg(case when reached then 1.0 else 0 end)           as completion_rate,
       percentile_cont(0.5) within group (order by secs)    as median_time_to_result,
       avg(corrections)                                     as mean_corrections,
       avg(case when had_failure and reached then 1.0
                when had_failure then 0 end)                as recovery_rate
from (
  select a.arm,
         a.attempt_id,
         bool_or(e.event = 'outcome_reached')               as reached,
         bool_or(e.event = 'correction')                    as had_failure,
         count(*) filter (where e.event = 'correction')     as corrections,
         extract(epoch from
           min(e.ts) filter (where e.event = 'outcome_reached') - a.ts) as secs
  from attempts a join events e using (attempt_id)
  group by 1, 2, a.ts
) t
group by arm;

The attempt_abandoned event is doing quiet work here. Without it, a session where the user gave up is simply absent from the outcome table and the completion rate is computed over a denominator that has silently dropped its failures. Emit it on session close, on tab close, and on a timeout while the surface is waiting for the user.

What the numbers looked like#

Building two surfaces sounds like twice the work and is not, provided both are built to be measured rather than to be shipped. The digest arm was a scheduled query, a template, and an email — three days of work, no new interface, and the five intents hard-coded rather than generalised. The chat arm already existed. What the comparison needs is that both emit the same events against the same outcome definition, not that both are production-quality; a thin arm that loses on completion rate has told you something real, and a thin arm that wins has earned the investment to become real. Teams that insist on building both properly before comparing usually build neither, and then settle the question by seniority.

Four weeks, both arms live to a randomised split of the eligible population, 1,100 attempts per arm.

The cost row was not planned and turned out to be the one the finance-minded stakeholder cared about, for a reason that generalises: the digest computes one summary per user per week and the chat arm recomputed a near-identical summary each time it was asked, so the same output was being paid for four times a month.

The comparison is only legitimate because of three deliberate choices. Both arms scored the same outcome, defined in advance. The run was four weeks rather than one, because a new conversational surface enjoys a novelty period in which usage and tolerance are both inflated, and week-one chat numbers overstate the steady state by a wide margin. And the sample was sized before the run: detecting a ten-point difference around a 60-to-70 percent baseline at conventional power — an 80 percent chance of the run showing the difference as real when it genuinely is — needs roughly 350 attempts per arm, so 1,100 leaves room to slice by user segment without the slices becoming noise.

What completion rate cannot see#

The aggregate also hides the segment that decides what to keep. Splitting the chat arm by intent showed 91 percent completion on the five head intents and 34 percent on the tail — and the tail is the part the digest cannot serve, so the honest reading of the experiment is not that chat lost. It is that chat was carrying a load it was bad at and being scored on the average. The team shipped the digest for the head, kept the conversation for the tail, and set the tail's 34 percent as the number to improve next, which is a far more useful backlog item than "make the assistant better".

Two limits are worth stating plainly, because a metric presented as decisive and then found to be blind is worse than no metric.

Completion rate cannot see a wrong answer that the user accepted. If the assistant reports last week's revenue with a broken currency conversion and the user reads it, believes it and leaves, every one of the four measurements scores that attempt as a success. The pairing that covers this is a small sampled correctness review — fifty results per arm, checked against ground truth by someone who knows the domain — reported alongside the four. In the medication-policy setting from the earlier chapter no completion metric would have caught the paraphrase problem at all, because the nurse completes the task either way; that domain needs the audit trail and a clinical review, not a funnel.

The four also cannot arbitrate a genuine strategic bet. If the argument is that a conversational surface will unlock use cases you cannot enumerate yet, that is a claim about a future request distribution and no four-week measurement of today's distribution refutes it. The honest response is to scope it as a bet with a review date and an intent-coverage count attached, rather than to pretend the numbers have settled something they have not.

Bringing it to the meeting#

The stakeholder who asked for a chatbot usually has a real motive underneath the request, and it is worth naming rather than beating. Often it is that the roadmap needs something visibly modern for a customer conversation or a board slide. That motive is legitimate and it is satisfied by whichever surface demos better, so the useful move is to make the alternative demoable. A digest that lands in a customer's inbox every Monday morning with their own numbers in it is a better artefact for a sales call than a text box, because it can be screenshotted, forwarded, and shown to someone who was not in the room.

What actually ends the argument is the before-and-after flow, shown as two short recordings side by side. The numbers in the table above are the evidence; the recordings are what makes the evidence legible to people who do not read tables. Forty seconds of a user answering "which project", pasting a build ID, copying an answer and reformatting it, against four seconds of a result already on screen, is an argument that does not need a defender.

The whole method reduces to a sequence you can start on Monday with whatever request is on your desk. Take the request that arrived naming a chatbot and write down what the completed task actually looks like. Run the paraphrase question first, because if the user needs the governing artefact the rest is decided. Then run ambiguity, repetition, urgency, approval and correctability across the five surfaces and record the selection with the numbers that drove it and the condition that should reopen it. Inventory the inputs and bind each one to the cheapest rung that can supply it, then trace the work on both sides of the request and delete the courier steps. Build the two candidate surfaces thinly enough to measure — a week each is usually enough — define the outcome once for both, and run them for four weeks. Then bring the four numbers, the sampled correctness check, and the two recordings.