Sampled prompts and passive signals beat a rating on every answer

Atlas shipped with a thumbs pair under every answer and a one-line prompt: Was this helpful? In week one, 4.1% of answers were rated. By week eight the figure was 0.6% and still falling, and the decline was not spread evenly across the 84 handlers. Seventy-one percent of them never touched the control again after their second week. Of the ratings that did arrive in week eight, 38% came from nine people, and those nine skewed heavily towards two of the twelve claim types.

The widget had become furniture. It was present on every answer, it never changed, it had no visible consequence, and the handlers' task was to close claims rather than to grade software, so the rational response was to stop seeing it. The team's first instinct was to make it more prominent, which raised week-one response and produced the same curve two weeks later with more complaints in the middle.

What the earlier topic on production feedback loops established about these signals — that rating coverage is too thin to slice, that an edit localises disagreement where a rating does not, that a retry measures effort rather than quality — is the analytical half of this problem. The other half is an interaction design problem: when to ask, what to attach the question to, which interactions to build so that the signal exists at all, and what the system is allowed to keep.

Ask a few people, on purpose#

Sampling the explicit prompt means most answers carry no question, and a small, deliberately chosen share carry one. The immediate objection is that this reduces the amount of feedback, and on this product it did not: a 5% sample with a seven-day per-handler cooldown produced an 11.4% response rate among prompts shown, which came to 0.57% of all answers against the 0.6% the always-on widget was producing by week eight. The volume was the same and the composition was different, because the responders were a broad slice of handlers responding once a week rather than nine people responding whenever something annoyed them.

The sampling has to be deterministic and stable, so that a handler who reloads the page does not get a prompt they dismissed, and so that the same answer analysed twice is treated the same way. It also has to be stratified, meaning the population is split into groups first and each group sampled at its own rate rather than the whole thing at one rate, because the segments that matter analytically are rarely the segments that generate the traffic.

import hashlib
from datetime import date, timedelta

# high-volume, well-understood work gets sampled lightly;
# rare and difficult work gets sampled heavily
STRATA = {"motor_simple": 0.02, "household_standard": 0.05, "subsidence": 0.25}
COOLDOWN = timedelta(days=7)

def bucket(answer_id: str, user_id: str) -> float:
    key = f"{answer_id}:{user_id}".encode()
    return int(hashlib.sha256(key).hexdigest()[:8], 16) / 0xFFFFFFFF

def should_prompt(answer_id, user_id, stratum, last_prompted_at) -> tuple[bool, float]:
    rate = STRATA[stratum]
    if last_prompted_at and date.today() - last_prompted_at < COOLDOWN:
        return False, rate
    return bucket(answer_id, user_id) < rate, rate

The second return value is the part teams forget. Feedback collected at 25% on subsidence and 2% on simple motor claims cannot be pooled without weights, and a quality dashboard built on the raw counts will report that subsidence answers are twelve times more problematic than motor answers when what it has measured is the sampling rate. Recording the rate on every event, alongside the stratum, is what makes the numbers reweightable later.

Cooldowns matter more than sampling rate for the annoyance side. A handler prompted twice in an afternoon learns the same lesson as one prompted on every answer, only faster.

Attach the question to the thing being judged#

A thumbs-down on a five-sentence answer says something is wrong somewhere. The interaction that produces a usable signal is one attached to the artifact the handler is already looking at, offered at the moment they are looking at it, and phrased as the specific complaint they have.

The citation viewer from chapter 3 was where this paid best. A handler who has opened a passage to check a claim is, at that instant, in possession of a judgement about whether the passage supports the claim, and the control sitting in the viewer — This passage does not support the highlighted sentence — asks for exactly that judgement and nothing else. It fired on 2.3% of opened citations, roughly 1,100 events a quarter, and every one of them named a chunk, a sentence and a document version. That is a substantially more valuable record than a thumbs-down, and the handler spent one click while already stopped.

The same logic put a control on the span marks from chapter 4 (this is wrong on a marked span, which also serves as a report that the mark itself was wrong) and made the rejected hunks of chapter 5 into first-class events. It is worth noticing that these controls are not a feedback feature; they are affordances inside the work, and the feedback is a by-product.

Design the interaction so the signal exists#

Passive signals are only available if the interface produced an event when the user acted, and that is a design decision made long before anyone thinks about analytics.

Copying is the clearest case. Handlers were copying answers into emails constantly and the product could not see it, because a text selection followed by a keyboard shortcut leaves no trace on the server. Adding an explicit Copy answer button captured 61% of copies; the remaining 39% still selected and used the shortcut, detectable at lower fidelity through selection events but without knowing what was taken. A copy is a strong statement that an answer was good enough to use, and for eight months the product had been generating that statement and discarding it.

The apply control from chapter 5 has the same property with more resolution. Apply all records one event and one bit of information. Per-hunk accept and reject records which specific changes a handler wanted, and it was the source of the finding that rewrites touching numbers were rejected at four times the rate of rewrites touching phrasing. The interaction design chose how much the team could learn.

The cancel path of chapter 6 is the highest-severity passive signal in the product. A handler who cancels a queued letter within the hold window has made an unambiguous statement that something was wrong enough to stop, at no cost to themselves, attached by assisted_by_answer to the exact answer that produced it. There were 214 of those in a quarter, and they were worth more per event than every thumbs-down the product ever collected.

The line the logging crosses#

Three of the rows above carry content rather than identifiers, and content is where a quality dataset turns into a customer-data problem. A handler's query routinely contains a policyholder's name and a claim number. The retrieved chunks are the insurer's documents, some of which are underwriting material the handler may read and an analytics team may not. Free-text feedback is the worst of the three, because a comment box invites people to paste the thing that went wrong, and an automated scan of Atlas's comments found that 6.8% contained a personal name, a policy number or a bank detail.

The structural answer is to split the event from the content at write time. Events — identifiers, ratings, scores, timings, stratum, sampling rate — go to the analytics store with broad access and a long retention. Content — query text, answer text, chunk text, comment text — goes to a separate store, keyed by the same identifiers, with tenant-level consent controlling whether it is written at all, a shorter retention, and access that is granted per investigation rather than per team.

{
  "event": {
    "event_id": "fb_7712",
    "answer_id": "ans_9f3c2b",
    "kind": "sampled_prompt",
    "rating": "not_helpful",
    "reason_code": "wrong_policy_version",
    "stratum": "subsidence",
    "sampling_rate": 0.25,
    "has_comment": true,
    "content_ref": "cnt_7712"
  },
  "content": {
    "content_ref": "cnt_7712",
    "tenant_consent": "logging_enabled_2026-01-14",
    "retention_days": 90,
    "comment_redacted": "This is quoting the old wording. [NAME] is on the 2025 endorsement.",
    "redactions": [{ "type": "person", "offset": 34 }]
  }
}

Redaction runs before the comment reaches storage rather than before it reaches a report, because a redaction applied at read time is a filter over data that has already been retained. Reason codes carry most of the analytical value at a fraction of the exposure, which is a good argument for offering a short list of them alongside the free-text box; on Atlas, 78% of respondents picked a code and 31% also wrote something.

The obligations that travel with a logged record — what may be exported, what may become an evaluation case, what has to be deleted when a customer leaves — were covered in the production feedback loops topic and apply unchanged here. The interface's responsibility is narrower and specific: the person writing in the comment box should be able to see, without opening a policy document, what is being collected and who will read it. A single line under the box stating that the comment and the answer are shared with the product team and deleted after 90 days is cheap, and on this product it reduced comment length by about 12% and more than halved the rate of flagged PII — personally identifiable information, the names, account numbers and contact details that identify a specific individual — in comments.

What sampling does not fix#

Sampling reduces selection on annoyance; it does not eliminate non-response. The 88.6% who saw a prompt and ignored it are still invisible, and no weighting recovers what they thought. What it buys is that the people who do answer were chosen by the system rather than by their own mood, which makes the residual bias smaller and, more usefully, stable enough to compare across weeks.

Nor do any of these signals fire on the failure that matters most, which is the answer that was wrong and was used anyway. Every row in the table above requires a human to have noticed something. The wrong-and-accepted case produces a copy event, a clean apply, no cancellation and a silent success, and the only instrument that finds it is a labelled random sample reviewed by someone with the domain knowledge to judge — which the feedback loops topic covered, and which none of this replaces.

That is the case the whole topic has been circling: a well-formed, well-cited, wrong answer, with a handler about to act on it, and nothing anywhere in the system that fires.