An evaluator, a dataset and a sample, or it is not an objective

The quality objective is one line in a shared document, agreed with the product owner at the end of a planning session: the weekly average LLM-judge score across production traffic is at least 4.2 out of 5. Twelve weeks of readings sit under it — 4.31, 4.28, 4.33, 4.29, 4.35, 4.27, 4.31, 4.30, 4.26, 4.34, 4.32, 4.29 — and in the last review someone described the flatness as reassuring. The team has shipped two prompt revisions, swapped the embedding model, and raised the retrieval top_k from 4 to 8 in that period. None of those changes registered. The score itself comes from an LLM judge, a second model shown the question, the answer and a written rubric — the list of criteria it has to score against — and asked for a verdict, which is how a team puts a number on correctness at production volume when nothing in a response says whether it was right.

The reason to distrust the number is not that 4.29 is close to 4.2. It is that "production traffic" is not a population and "the weekly average" is not a sample; both phrases describe whatever happened to arrive, scored by whatever the evaluator happened to be that week. A latency percentile with those defects is still computed from every request, so at worst it is imprecise. A quality figure with those defects is measuring a different thing every week, and there is no way after the fact to say which.

The request mix moves the score without touching the system#

Suppose the assistant handles three kinds of request, and each kind scores differently because each kind is differently hard. Nothing about that is a defect; a lookup against a single FAQ document genuinely is easier than reconciling four contract PDFs. What it means is that the weekly average is a weighted mean whose weights are set by traffic.

A marketing push that drives self-serve signups shifts the mix towards the easy class and lifts the reported score by 0.13 with the system untouched. A single enterprise customer onboarding with a document-heavy workload moves it the other way by a similar amount. Both movements are larger than any of the twelve weekly readings differ from each other, which means the observed stability is not evidence of a stable system — it is evidence that the mix happened not to shift much, or that the metric is too blunt to register when it did.

There is a second, quieter defect in taking a mean of a five-point judge score. An error budget is an arithmetic on events: so many bad events permitted per window, consumed at a rate you can watch. A mean has no events in it. Three catastrophically wrong answers scored 1 and forty perfectly good ones scored 4.5 average out to 4.26, which is above target, and the objective reports success on a week in which three users were actively misinformed. The SRE convention that survives contact with this is to define the indicator as a ratio of good events to valid events, so the quality SLI becomes the proportion of eligible requests whose judged verdict is acceptable, and the threshold that turns a score into a verdict is written down as part of the objective rather than left to whoever builds the dashboard.

The five things an objective has to name#

A usable quality objective specifies the population of eligible requests, the method by which a sample is drawn from that population, the evaluator and its exact version, the reference data the evaluator scores against, and the threshold over a stated window. Missing any one of them leaves a number that cannot be compared to itself across weeks.

Written as a spec that a job can read, rather than a sentence in a document, the same objective looks like this.

slo: answer_quality
population:
  routes: [chat.answer, chat.summarize]
  include: {auth: end_user, streaming: true}
  exclude:
    - traffic_source: [synthetic, loadtest, eval_replay]
    - status: [cancelled_before_first_token]
    - output_tokens: {lt: 20}       # confirmations, not answers
sample:
  method: deterministic_hash        # sha1(trace_id) % 10000 < rate_bp
  rate_bp: 200                      # 2.00% of eligible requests
  stratify_by: [tenant_tier, language]
  min_per_stratum_per_week: 200
evaluator:
  model: claude-sonnet-4-5-20250929
  temperature: 0
  template: rubric/groundedness_v7.md
  template_sha: 9f2c41ab
  sees: [question, retrieved_context, answer]
  output: {verdict: [pass, fail], failure_reason: enum, rationale: string}
objective:
  sli: passed_samples / scored_samples
  target: 0.95
  window: 28d rolling
  min_samples: 2000

The sees field is the one teams most often leave implicit and most often regret. A judge shown the retrieved context can rule on groundedness, meaning whether every claim in the answer is supported by the passages the system actually retrieved; the same judge shown only the question and the answer is ruling on plausibility, which is a different property that correlates with fluency and rewards confident invention. Two teams can run "the same" groundedness eval and disagree by fifteen points because one of them passes the context and the other does not.

Deterministic sampling matters for a less obvious reason. If the sampler uses a random draw at request time, the set of scored requests exists only in the scored-samples table and cannot be reconstructed; if it hashes the trace id, you can go back to any week and ask which requests would have been sampled at a different rate, which makes a retrospective re-score possible.

import hashlib

def is_sampled(trace_id: str, rate_bp: int, salt: str = "quality_slo_v1") -> bool:
    digest = hashlib.sha1(f"{salt}:{trace_id}".encode()).digest()
    bucket = int.from_bytes(digest[:4], "big") % 10_000
    return bucket < rate_bp

The salt is there so that raising the rate is a superset operation — every trace sampled at 200 basis points is still sampled at 400 — and so that the quality sampler and the trace sampler from the previous topic do not correlate. If both use the raw trace id modulo something, you can end up scoring only requests that were also fully traced, which is a population you did not choose.

The verdict rule and the target are two separate arguments#

Turning a judge's opinion into a pass or a fail requires a rule, and where that rule sits determines what the objective is about. Asking the judge for a binary verdict against named criteria puts the threshold inside the rubric, where it can be written in words a product owner can read and argue with. Asking for a 1-to-5 score and thresholding at 4 puts it outside, where it is a magic number in a config file and where it silently inherits whatever the judge believes the middle of a five-point scale means. The first is easier to defend and easier to change deliberately; the second is easier to build on top of an eval harness that already exists, which is why it is common.

The failure_reason enum earns its place in the same decision. A verdict with no reason gives you a rate and nothing to do about it, whereas a small closed set — unsupported_claim, missing_from_context, wrong_entity, refused_answerable, format_violation — turns the objective's failures into a distribution that can be read week to week. On the assistant above, unsupported_claim runs at about 60% of failures in a normal week, and a week in which missing_from_context overtakes it is a retrieval story rather than a generation story before anyone has opened a single trace. Keeping the set closed and small is what makes it comparable; a free-text reason from the judge is a different, less useful artefact that has to be clustered before it says anything.

The target is a separate negotiation and should be set from a measured baseline rather than chosen for its roundness. Score a few thousand samples, look at where the current system actually sits, and pick a target that is currently being met with a little room — 95% when the baseline is 96.3% — so that the first breach is a real change rather than a discovery that the system never met the number. A target set aspirationally at 99% on a system running at 96% produces an objective that is permanently red, which trains everyone to ignore it within a month and is worse than having no objective at all.

What has to exist before the rewritten objective can be reported#

The one-line version was reportable immediately, which is most of why it got written. The specified version needs machinery, and the honest way to present a rewrite in a planning meeting is with that cost attached.

Eligibility has to be computable, which means the trace record already carries route, auth type, traffic source, language and request class as first-class fields rather than as strings inside a log line. Sampling has to happen somewhere that sees every eligible request, so it belongs in the same place the trace is emitted. Scoring has to run off the request path — a queue consumer reading sampled traces, calling the judge, and writing verdicts — because a judge in front of production traffic adds its own latency and its own failure mode to the system whose health it reports. The scored-sample store has to keep the inputs, not just the verdict, so a sample can be re-scored later. And the evaluator needs a calibration set of a few hundred human-labelled examples, because the number the judge produces is only worth reporting to the extent it agrees with a person, and nobody knows whether it does until someone measures.

That last piece is what turns "three flat months" from reassuring into a question. A metric can sit still because the system is stable, or because the instrument cannot move. Distinguishing them takes a negative control: take 200 stored production answers that the judge passed, corrupt them in ways a person would unambiguously call wrong, and re-score.

A judge that still passes 94% of answers whose numbers have all been falsified is not measuring correctness, and an objective built on it will read 4.29 forever. Those five rows took an afternoon and about $9 of judge tokens to produce, and they are more informative about the quality metric than the twelve weekly readings it has emitted. Aparna Dhinkaran's Lessons from the Trenches: Building LLM Evals That Work IRL makes the same argument from a different direction — that the first thing to establish about an eval is whether it can distinguish outputs you already know to differ.

Where this sits relative to the eval suite you may already have#

A production quality SLO is not an offline eval suite, and teams that already run one sometimes assume the SLO is redundant. They answer different questions. The offline suite runs a fixed dataset against a candidate build before it ships and gates the deploy; its dataset is stable by design, which is exactly what makes it useless for detecting that live traffic has changed. The SLO scores live traffic continuously and cannot gate anything, because by the time it has an opinion the request is long since answered. A regression that the offline suite catches never reaches users; a regression the SLO catches is one the offline dataset did not contain, which is most of the interesting ones.

Nor does the SLO replace direct user signal. Thumbs-down rate, copy-to-clipboard rate, and the fraction of conversations that escalate to a human are cheap, carry no judge cost, and reflect what users actually thought. They are also sparse — a 0.4% thumbs-down rate on 8,000 weekly conversations is 32 events, which will not support a weekly objective — and they are biased towards users annoyed enough to click. Running both is normal; the judge-based SLI gives you volume and attributable failure reasons, the behavioural signal gives you ground truth to calibrate against.

The specified objective is now precise enough to be wrong in interesting ways. It depends on an evaluator that is a hosted model with a version, a template file somebody can edit, and a sampling filter expressed as a query — three moving parts sitting between the system and the number on the dashboard. When that number drops five points overnight with no deploy behind it, the first suspects are those three, and the next chapter is about running that triage in the right order.