An edit points at the span the user disagreed with

The drafting assistant in a support console writes the agent's reply, drops it into the composer, and lets the agent change whatever they want before hitting send. Three signals come out of that flow. A thumbs control sits under the draft and fires on 0.3% of responses. A regenerate button gets pressed on 4%. And on 99.6% of responses — everything that was eventually sent — the system has both the text it generated and the text that left the building, which means it has the diff. The team building the improvement loop spent two weeks trying to make the thumbs data support a weekly report, ran out of rows the moment anyone asked to slice by customer tier, and then noticed that the third signal had been sitting in the database the whole time.

Here is one of those pairs. The generated draft:

Hi Marcus,

Thanks for reaching out. I've checked your account and I can confirm that your
refund of $248.00 was processed on 3 March and should appear on your statement
within 3-5 business days.

If it hasn't arrived by then, just let me know and I'll escalate.

Best regards,
Support Team

And what the agent actually sent:

Hey Marcus,

Thanks for reaching out! I checked your account — the refund of $248.00 was
approved on 3 March but hasn't been released yet, so the 3-5 business days
start from release. I've chased the payments team this morning.

Give me a shout if it hasn't landed by Friday.

Cheers,
Dani

The character-level Levenshtein distance between the two, which counts the single-character insertions, deletions and substitutions needed to turn one text into the other, is 0.41 once it is normalised by the length of the longer text. Read as a defect score that number is nearly meaningless, because most of the 0.41 is Dani being Dani. She writes Hey rather than Hi, she signs with her own name, she says give me a shout, and she has done all three on 1,140 of her last 1,200 drafts. What is left after those habits are accounted for is a single span in the middle, where the model said the refund was processed and the agent corrected it to approved but not released. That span is the whole signal: it names the failure (the model read a payment status field it did not understand), it names the location (one clause), and it arrives with the full trace attached.

Having the diff at all depends on a capture decision that is easy to get wrong at the point where the composer is wired up. The draft has to be stored with an id at the moment it is inserted, and the sent message has to carry that id forward, so that the pair can be joined after the fact. Products that instead snapshot the composer on blur, or diff against the last autosave, end up measuring the user's typing process rather than their final judgement, and the resulting distances are inflated by every backspace. Products that store only the sent text and reconstruct the draft by re-running the model later measure the difference between two generations, which on a temperature above zero is not zero even when the user changed nothing. The correct capture is a foreign key written once at insert time and a second row written at send, and the AWS reference architecture for production feedback loops is worth skimming for how this join is usually laid out alongside the trace.

Coverage and localisation both point the same way#

The two properties that decide which signal an improvement loop is built around are how much of traffic the signal appears on and how precisely it identifies what went wrong. Thumbs lose on both. A rating exists on 0.3% of responses, so a weekly report over 210,000 drafts has about 630 ratings, which is fewer than one per (customer tier × request type × week) cell once anyone starts slicing, and it says only that something was wrong somewhere in a 90-word draft. The edit diff exists everywhere and hands you a character range.

The row that matters most for the weekly report is the fourth. Both signals are biased; the difference is whether the bias is measurable from inside the data. With thumbs, the population that did not rate is simply absent, and no amount of modelling recovers what those users thought — chapter 1's table showed how wide the resulting uncertainty is. With edits, everyone is present, and the confounds are per-user style and per-request difficulty, both of which are observable columns you can condition on. A noisy signal you can correct is more useful than a sparse one you cannot.

Neither signal catches the failure that should worry you most, which is the answer that was wrong and got sent unchanged because the agent did not know it was wrong. Nothing in this table has a row that fires on that, and the labelled random sample from chapter 1 is the only instrument that does.

Separating what the user fixed from how the user writes#

The mechanical version of the separation is to learn, per user, the transformations they apply almost every time, and to normalise them out before measuring anything. Mining those rules is a frequency count over aligned diff hunks, a hunk being one contiguous stretch where the two texts disagree: any hunk whose (before, after) pair recurs on more than 60% of that user's drafts over the last 200 is a style rule rather than a correction.

from collections import Counter
from difflib import SequenceMatcher

STYLE_THRESHOLD = 0.60      # share of a user's drafts a hunk must appear on
MIN_DRAFTS = 50             # below this, the user has no reliable style prior

def hunks(generated: str, sent: str) -> list[tuple[str, str]]:
    """Replacement pairs between the two texts, in order."""
    matcher = SequenceMatcher(None, generated, sent, autojunk=False)
    out: list[tuple[str, str]] = []
    for tag, i1, i2, j1, j2 in matcher.get_opcodes():
        if tag != "equal":
            out.append((generated[i1:i2], sent[j1:j2]))
    return out

def learn_style_rules(drafts: list[tuple[str, str]]) -> set[tuple[str, str]]:
    if len(drafts) < MIN_DRAFTS:
        return set()
    counts: Counter[tuple[str, str]] = Counter()
    for generated, sent in drafts:
        counts.update(set(hunks(generated, sent)))
    floor = STYLE_THRESHOLD * len(drafts)
    return {pair for pair, n in counts.items() if n >= floor}

Applied to Dani's history, that returns four pairs: ("Hi", "Hey"), ("Best regards,\nSupport Team", "Cheers,\nDani"), ("just let me know", "give me a shout"), and (".", "!") after the first sentence. Subtracting them from the draft above drops the normalised distance from 0.41 to 0.13, and the remaining distance is concentrated in one 74-character span.

The same normalisation has to run at the team and tenant level as well, because house style is not only individual. Onboarding a UK enterprise customer whose brand guide mandates British spelling puts three to eight character-level edits into every draft that contains organize, apologize or color, which on this product raised the mean normalised edit distance across all traffic by 2.9 points in the week the customer went live. No model changed. Anyone watching a single edit-distance line on a dashboard would have opened an incident, which is the concrete way this signal moves without the system getting worse. The others in that family are a template change that alters the draft's default length, a composer that now inserts a signature block automatically, and a new request type — refund disputes, say — where agents rewrite more because the stakes are higher.

A floor, and what sits under it#

Even after style normalisation, most drafts carry small changes that nobody would call a defect: a comma, a name capitalisation, a sentence split in two. Counting those as failures produces a defect rate near 100% and a metric that cannot move. The fix is a threshold, and the threshold has to be calibrated against human judgement rather than chosen because it looks round.

The calibration is a small labelling exercise of the kind chapter 1 described: draw 200 draft/sent pairs at random, have two people mark each as the agent corrected a mistake or the agent did not, adjudicate disagreements, and then sweep the threshold. Precision in the sweep below is the share of flagged drafts that a human agreed were corrections, and recall is the share of the human-marked corrections that the threshold managed to flag.

At 0.12 the flagged population is 11% of drafts, roughly 23,000 a week on this traffic, and roughly three in ten of them are false alarms. That is the operating point the team picked, because the downstream consumer is a triage queue that can tolerate 30% noise, and losing 37% of real corrections is acceptable when the remaining 63% is already more cases than anyone can process. A different consumer justifies a different point: an automated case-generation pipeline that nobody reviews should sit at 0.20, and a weekly trend line can sit at 0.08 because it only needs the ratio to be stable, not clean.

Distance is a blunt proxy in one specific way that the threshold cannot fix. A three-character edit that changes may to may not, or corrects $248.00 to $24.80, is the most severe defect the product can produce and scores 0.01. Teams that care about this run a second, cheaper check over the hunks — a classifier or a small model asked whether any factual claim changed — and route the low-distance, claim-changing edits to the queue regardless of the floor. On this product that rule added about 400 cases a week that distance alone had discarded, and their human-labelled severity was materially higher than the average flagged case.

Distance also flattens together edits that mean different things, and the flagged queue becomes far easier to act on once each surviving hunk is typed. A cheap classifier over the hunk and its surrounding sentence sorts them into a handful of categories, and on a week of flagged drafts from this product the distribution was roughly 31% factual correction, 22% added information the draft omitted, 18% hedging added or removed, 14% length or structure, and 15% residual style the per-user rules had not caught. Those five categories route to different owners: factual corrections are a retrieval and grounding problem, omissions are usually a prompt or context-window problem, hedging changes are a policy and tone question for product, and a rising residual-style share means the normalisation is stale and the rules need re-mining. A single edit-distance number cannot make any of those distinctions, which is why the typed breakdown rather than the headline rate is what belongs in the weekly report.

What the sparse signal still earns its keep for#

Deciding that the weekly report runs on edits does not mean the thumbs control gets removed, and the reasons are specific rather than sentimental. The 630 weekly ratings are the labelled set that calibrates the edit threshold in the first place: thumbs-down responses should be over-represented above the threshold, and if that relationship weakens after a release, the normalisation has drifted. The free-text comments attached to ratings are the only place a user says why, which turns a flagged span into a diagnosable failure mode without an engineer having to guess. And a thumbs-down on a draft that was sent unchanged is the one combination that catches the silent acceptance the edit signal is blind to, which makes those few hundred rows per quarter disproportionately worth reading.

One thing an edit does not do is tell you the sent version was right. The agent who rewrote processed to approved but not released might have been wrong about the payment state, and the version that went to Marcus is a second opinion rather than ground truth. Treating the sent text as the reference output — training on it, or scoring the model against it automatically — bakes in every mistake the agents make, and on a support product the agents are working quickly under a queue-length target. The diff identifies a place where a human disagreed with the model, and turning that disagreement into an expected outcome still requires someone to decide who was correct, which is a review step rather than a pipeline step.

There is also a class of product where none of this transfers. A conversational assistant with no editable surface produces no diffs at all, and the entire argument above collapses to nothing; the same is true of voice, of a search result page, and of any flow where the model's output is consumed rather than shipped onward under the user's name. The diff signal exists because this product asks a human to take responsibility for the text, and that responsibility is what makes the edit a judgement rather than a preference. Where the surface is missing, the loop has to be built on the remaining behavioural traces, and the most common of those is the user asking again.

That is the signal chapter 3 takes apart, and it is the one most often read as a quality metric when it is measuring something else entirely.