Task success has to be checkable by two raters who never speak

Task success is the figure every other measurement in this topic is read against, and it is the one most often left as a sentence nobody can check. Agent evaluation — routinely shortened to evals — is the practice of running an agent over a fixed collection of tasks whose correct outcome somebody wrote down in advance, and scoring what comes back. The agent is the loop from topic 10.3, a model that reads a request, calls a tool, reads the result, calls another and keeps going until it decides it has finished. One pass of that loop over one task is a run; the collection of tasks together with their expected outcomes is the eval set; the written statement of what counts as a correct outcome is the rubric, the document that has to answer every question a scorer can ask; and the proportion of tasks whose end state was acceptable is the task success rate.

Agents need a practice of their own here because the two instruments a backend engineer reaches for first do not reach. A unit test compares an output against a fixed expected value, and an agent has no fixed output: it emits a sequence of tool calls that differs between identical runs, and it leaves its result in the world — a calendar entry, an email, a file — rather than in a return value. Reading the runs by hand covers a few dozen and then stops, while the prompt and the tool set change every week and each change puts the whole set back in the queue to be read again.

In practice this is met as a gate. The eval set runs in continuous integration on every change to the prompt, the tool descriptions or the model version, and the task success rate it prints is the number a release meeting argues over. Five further measurements sit around that one and occupy the rest of this topic: the quality of the path the agent took, how accurately it chose its tools, how many steps it needed, what each task cost, and whether it recovered when something it called failed. All of them are read next to the success rate, so a rate that two competent people compute differently is the most expensive vagueness the set can carry.

A definition that leaves the rater to decide#

Two engineers scored the same forty runs of a scheduling agent, working from the same one-line rubric, without talking to each other. They agreed on thirty-one and argued about nine. Run 118 was the one that took the longest to settle. The email thread had Ana asking for forty-five minutes with Priya and Tom, "Wednesday or Thursday afternoon", with Ana in Europe/Madrid; the agent booked Thursday at 14:00 CET with Priya and Tom on the invite, titled it "Q3 pricing review", and replied on the thread confirming. The last message in the thread, which arrived after Ana's original request, said "actually Wednesday works better".

One rater marked run 118 a success: a meeting exists, it is forty-five minutes, the right two people are on it, nobody had to chase anything. The other marked it a failure, because the user asked for Wednesday and got Thursday. Both readings are defensible against the rubric they were given, which was "the meeting got scheduled and the user is happy". Neither half of that sentence can be checked. "Got scheduled" does not say scheduled when, or with whom, or whether one event or three. "The user is happy" is a fact about a person nobody in the room can query.

It is worth being exact about what a disagreement of that kind proves, because everything the rest of the chapter does rests on it. A success definition is meant to behave like a function: hand it a run and the world that run left behind, and it returns a verdict. Both raters were handed the same run and the same world, and they returned different verdicts. Something other than the run and the world was therefore feeding into the result, and the only other thing in the room was the rater. The disagreement is evidence that the definition has a free variable in it, and it is evidence about where the variable sits: on run 118, on eight others, and nowhere else in the forty.

What it is not evidence about is which verdict was right. A disagreement says the definition failed to determine an answer, and determining the answer is a product decision rather than a tie-break. So the nine disagreements were not a rater problem to be fixed with training or a tie-breaker. Each of them was a place where the definition had no answer, and a rater supplied one. When two independent scorers reach different verdicts on the same run, the useful assumption is that the definition is underspecified at exactly that point, and the repair is to go and specify it.

The test only runs in one direction. Disagreement proves the definition left something undetermined, because two raters handed the same run and the same world could not have differed about anything the definition already settled. Agreement proves nothing of the sort: two raters agree when the definition determined the answer, and they agree when both supplied the same missing answer — the same reasonable default, the same reading of an ambiguous phrase — and from outside those two cases are indistinguishable. A rubric can be convicted by a disagreement and never acquitted by an agreement, which is why the repair below is to write missing facts down rather than to keep scoring until the arguments stop.

The two buckets are exhaustive by construction, which is worth noticing before anyone reads too much into them. Every run received exactly one pair of verdicts, and a pair either matches or it does not, so thirty-one and nine account for all forty and could not have failed to. That closure licenses one comparison and forbids another. It licenses reading the buckets against each other — roughly three runs in four settled themselves and the remaining quarter did not — which is a fact about this rubric on this set. It does not license a rate. Forty runs are the runs somebody chose to put in front of two people, and a disagreement rate measured on a hand-picked forty says nothing about what the same rubric would do on the next four hundred.

That framing turns the disagreements into a work queue, and the queue is shorter than the argument the two engineers had been engaged in. Take each contested run and ask what single fact, if it had been written down beforehand, would have made both raters produce the same verdict. For run 118 that fact is whether a later message in a thread supersedes an earlier one. For another of the nine — a run where the agent booked the meeting and never replied on the thread — the fact is whether confirming back to the requester is required or merely desirable. Nine disagreements yielded six distinct missing facts, because several runs were contesting the same gap, and six decisions is a morning's work rather than an open-ended research project.

An assertion is a question the end state can answer wrong#

The property that makes a success criterion checkable is that it reads the world after the run and returns a boolean without needing a person's interpretation. That means querying the systems the agent touched — the calendar, the mailbox, the ticket store — rather than reading the agent's own summary of what it did. An agent that reports "I've booked that for you" while the calendar API returned a 409 is a common enough failure that scoring the transcript instead of the end state builds a metric which agrees with the agent about the agent.

That last sentence is the step to slow down on, because it is what makes this a rule rather than a preference. Scoring is an activity performed on evidence, and evidence is only worth having when it could have come out otherwise. The agent's account of what it did is produced by the same policy that produced the actions, so one fault in that policy generates both the wrong action and the confident report of success — one cause, two symptoms. Cross-checking the two is therefore not two observations of the same event but one observation stated twice. Querying the calendar breaks the dependence, because the calendar was written by a system with no view of what the agent intended, so the missing event survives the agent's belief that it created one.

from dataclasses import dataclass
from datetime import timedelta
from typing import Literal

@dataclass
class Verdict:
    score: Literal["pass", "partial", "fail"]
    failed_required: list[str]
    failed_secondary: list[str]

REQUIRED = ["single_event", "attendees", "window", "duration", "invites_delivered"]
SECONDARY = ["title_mentions_subject", "thread_reply", "no_duplicate_holds"]

def score_schedule_meeting(task, world) -> Verdict:
    events = world.calendar.events_created_by(run_id=task.run_id)
    checks = {"single_event": len(events) == 1}
    if events:
        ev = events[0]
        expected = task.expected
        checks["attendees"] = set(ev.attendee_emails) >= expected.required_attendees
        checks["window"] = expected.window.contains(ev.start.astimezone(expected.tz))
        checks["duration"] = abs(ev.duration - expected.duration) <= timedelta(minutes=5)
        checks["invites_delivered"] = all(a.response != "not_invited" for a in ev.attendees)
        checks["title_mentions_subject"] = expected.subject_terms & set(ev.title.lower().split())
        checks["thread_reply"] = world.mail.agent_replies(task.thread_id) == 1
        checks["no_duplicate_holds"] = world.calendar.holds_created(run_id=task.run_id) == 0
    failed_r = [c for c in REQUIRED if not checks.get(c)]
    failed_s = [c for c in SECONDARY if not checks.get(c)]
    if failed_r:
        return Verdict("fail", failed_r, failed_s)
    return Verdict("partial" if failed_s else "pass", [], failed_s)

Run 118 fails window under this scorer, because the expected window derives from the last message in the thread that named a day, which is Wednesday. Both raters now return the same verdict, and more importantly, the argument they were having has moved to a place where it can be settled once: does a later message override an earlier one, or does an ambiguous thread mean both days are acceptable? That is a product question with an answer, and once the answer is written into how expected.window is built, no future rater has to re-derive it.

The scorer also runs in about 900 ms over forty runs against a fixture calendar, where the two humans spent roughly fifty minutes each. Speed is not the main argument for it, but it is what makes re-scoring the whole set after every definition change cheap enough that you actually do it.

Tolerances describe acceptable end states, not sloppy checking#

A tolerance sounds like a concession, and treating it that way produces the wrong ones. The right question is which set of end states genuinely satisfies the request, and a tolerance is how you write that set down when it is not a single value.

Two different things travel under the name. One is measurement slack on a continuous quantity: calendar clients round a forty-five minute event to the nearest five minutes in some views, so duration carries ±5 min because a forty-seven minute event is the same event to the user. The other is a genuine set of acceptable answers: "Wednesday afternoon" admits any start between 13:00 and 17:00 local where all required attendees are free, and there is no defensible way to pick one of those as the gold answer. The first is expressed as a numeric window around a target; the second is expressed as a predicate over the world, which is why window.contains(...) reads a range and attendees uses a superset test — extra optional attendees are acceptable, a missing required one is not.

One half of each of those checks is forced and the other half is chosen, and keeping the two apart is what lets a reviewer tell a loosened check from a corrected one. The shape of the check is forced by the shape of what counts as acceptable: a group of people admits a subset test and nothing else, because "the right people were invited" is a claim about membership, while a time admits an interval test, because "Wednesday afternoon" names a range rather than a point. The width is chosen — five minutes rather than one, an afternoon that ends at 17:00 rather than later — and a width is a product decision somebody can be asked to defend. A rubric review that cannot say which kind a given tolerance is has no way to argue about it.

That table is read across rather than down. Each row holds a clause and its replacement, and the quantity worth measuring is the distance between them: how much a rater had to invent to get from the left column to a verdict. "At the right time" and "with the right people" travel a long way, because each hides a whole rule about where the expectation comes from — which message names the window, whether an optional attendee counts. "For the right length" travels almost none, because the request stated a duration and the only open question was the width of the tolerance. The long rows are the rows the nine disagreements clustered on, and a rubric written entirely in the left column has left every one of those distances to be crossed at scoring time, by whoever happens to be scoring.

Deleting the last row is the part teams resist, because it feels like dropping the thing that actually matters. What it drops is a clause that contributed nothing to any score, since no rater could evaluate it and both silently substituted their own proxy. User satisfaction is worth measuring, and it is measured with instruments that can reach the user — thumbs, reply sentiment, escalation rate — on a different cadence from an eval that has to run on every pull request.

Why that clause looked harmless is worth stating in general terms, because most bad rubric lines have the same shape. A criterion nobody can evaluate contributes the same amount to every verdict, and that amount is nothing: it is a constant term in the score. The property it names varies across the whole set — some of those forty runs delighted somebody and some caused an afternoon of rework. So the clause sat perfectly still while the thing it was there to watch moved as much as anything in the eval, and a team reading the rubric saw a line about user satisfaction and believed the eval was watching it. The tell was available before any run was scored: the clause named a fact the scoring procedure had no way to query. A criterion that cannot fail and a criterion that is not wired to anything look identical from outside, which is the general form of this mistake and the reason it is worth hunting for deliberately.

Partial completion needs a rule written before you see the run#

The run that sends invites to three of four attendees, because the fourth address bounced, will appear in your set. So will the one that books the meeting correctly and never replies on the thread. If the definition has no position on these, each rater invents one, and the invented positions differ in the direction of whatever that rater was worried about that week.

Three rules are available and they are not equivalent. Strict binary treats any failed assertion as a failure, which is simple to defend and tends to floor the reported rate low enough that it stops being useful for detecting small regressions. Weighted scoring gives each assertion a fraction and averages, producing a number like 0.87 for a single run, and it is the tempting option because it looks precise; the cost is that a success rate built from averaged fractions cannot be read as "how often does this work", and a change from 0.87 to 0.84 no longer corresponds to any run you can go and look at. The rule in the scorer above splits assertions into required and secondary, calls a run pass when everything passes, partial when the required set holds and something secondary does not, and fail otherwise.

The reason that third rule survives contact with a dashboard is that partial runs are reported as their own rate rather than folded into success. A release that reports 78% pass, 14% partial, 8% fail carries information that a single 85% does not, and it makes the classification argument visible: if partial is climbing while pass falls, something is degrading in exactly the assertions you decided were secondary, and you get to reconsider whether they should have been.

Those three labels partition the runs, and the partition is what makes the three rates readable side by side. The scorer returns on the first exit it reaches: a failed required assertion returns fail, otherwise a failed secondary assertion returns partial, otherwise pass. Every scored run takes exactly one of those exits and no run can take two, so the three counts add to the number of runs scored and the three rates add to one. That is what licenses reading a release's three figures as a decomposition of one set rather than as three unrelated measurements, and it is why a rise in the partial rate has to have come out of one of the other two. The residual is just as sharp: the boundary between the three is drawn by which assertion names sit in REQUIRED and which sit in SECONDARY, so moving one name from the second list to the first moves runs from partial to fail with nothing whatsoever having changed about the agent. A three-figure report that does not travel with those two lists cannot be compared against last month's.

There is a fourth category that the three rules all handle badly, and it is worth naming rather than forcing into one of them. Some runs cannot be scored at all, because the input did not contain enough information to determine a correct end state — a thread where nobody named a duration, a day, or a required attendee, and any of several bookings would have been defensible. Marking those unscoreable and reporting the count keeps them out of both the numerator and the denominator, which is more honest than a coin flip and has the side effect of measuring how much of your eval set is ambiguous. When that count climbs above five percent or so, the set itself needs attention, since a definition cannot be tested against inputs that admit no answer.

An unscoreable count sits outside that partition rather than inside it, which is both the point of it and its cost. The three rates now add to one over the scoreable runs rather than over the set, so the denominator moves whenever the count does, and two releases carrying different unscoreable counts are being compared on two different denominators. Reporting the count next to the rates is what keeps that visible. Folding those runs into fail instead would hold one denominator steady and would charge the agent for an input that no policy could have answered, which is the trade being made and is worth making deliberately rather than by leaving the category out.

A judge can settle a leaf, and cannot hold the structure#

The shortcut everybody reaches for at some point is to hand the whole thing to a model: show it the task, the transcript and the end state, and ask whether the agent succeeded. It is genuinely fast to build, it produces a number the same afternoon, and it appears to solve the ambiguity problem by removing the humans who were disagreeing.

What it actually does is relocate the ambiguity. A judge given "did this succeed?" and a one-line rubric has to invent the same six missing decisions the human raters were inventing, and it invents them with less consistency than a person: on the original forty runs, the same judge prompt run twice with the examples in a different order disagreed with itself on seven of them, including four of the nine the humans had argued about. A judge is a rater, and a rater given an underspecified definition produces underspecified verdicts no matter what it is made of.

Where a judge does belong is at the leaves. Some assertions genuinely require reading natural language — whether the reply on the thread states the time that was actually booked, whether the event title refers to the subject under discussion — and no amount of string matching handles those well. The structure stays in code, with the required and secondary sets, the tolerances and the partial rule written explicitly, and a judge is called for the one or two checks that need it, with a narrow prompt scoped to a single question and a calibration sample behind it. That arrangement keeps the definition inspectable, keeps most of the scoring free and instantaneous, and confines the model's variance to fields where a human would also have had to exercise judgement.

An acceptable end state is a floor#

A success definition of this shape is a floor. It says the end state is acceptable, and it deliberately says nothing about whether it is good. An event booked at 07:30 on a Wednesday, technically inside "morning" and technically a slot where everyone's calendar is free, passes every assertion above and would annoy three people. Catching that needs a preference model or a human, and it belongs in a separate rubric with a separate cadence, because pinning it to the same gate makes the gate flaky.

The definition is also silent about everything that happened before the end state. A run that reaches exactly this calendar entry in six tool calls and a run that reaches it in forty-one, having written a file to a shared drive along the way that nobody asked for, are indistinguishable to score_schedule_meeting — both return pass, because both leave the same calendar behind. That indifference is correct for this instrument and dangerous as a complete picture of the system, since production repeats the path thousands of times a day while you only ever inspected the answer.

A definition of task success is finished when nothing is left in it for a rater to supply: when the run and the world it left behind determine the verdict, and the person holding the rubric contributes nothing to it. Two raters who never speak are the cheapest instrument for finding what is still left, and every disagreement they produce names one thing — not a rater to retrain and not a verdict to arbitrate, but a fact somebody has to decide and write down. Forty runs yielded six such facts and then a seventh, and each of them is now a line of code that no future scorer has to re-derive. What that buys is a figure two competent people compute the same way, which is the least a number can be asked for before five other measurements are read against it. Scoring the path is the next thing to build.