Build versus buy for an eval harness

An eval harness is the runner in the middle of an evaluation: the code that walks a list of cases, renders each one into a prompt, calls the model, hands the answer to a grader, and adds up the verdicts. Every hosted eval platform contains one, and it is not the part you are paying them for. A competent engineer writes a working version in an afternoon, which is what makes the build-versus-buy argument here unusually sharp: on the day it is demonstrated, the bought thing looks exactly like the thing you would build.

The honest way to have that argument is to write the harness, run it, and then list what a bought tool does that the harness does not, with a price against each item. This chapter is that list. It is five items long, none of them is exotic, and the total is large enough to change a decision — while three specific conditions still land on keeping the harness anyway.

The core is genuinely small, and it is worth looking at before arguing about it, because the argument is not that the code is bad.

def run_suite(cases: list[EvalCase], grader: Grader) -> RunResult:
    results = []
    for case in cases:
        prompt = render_prompt(case)
        completion = client.responses.create(
            model=MODEL, input=prompt, temperature=0.2, max_output_tokens=600,
        )
        verdict = grader.score(case, completion.output_text)
        results.append(
            CaseResult(case_id=case.id, output=completion.output_text,
                       passed=verdict.passed, rationale=verdict.rationale)
        )
    passed = sum(r.passed for r in results)
    return RunResult(results=results, score=passed / len(results))

That loop is correct. It is also a description of the first run on a small suite, and every capability it lacks is one that first appears on the second run, or at ten times the case count, or when a person other than the author has to look at the output.

What the loop costs once the suite grows#

A returns suite that starts at 200 cases reaches 1,000 within a quarter, which is the normal trajectory once a team starts turning support escalations into cases. At that size the sequential loop takes 1,000 × 3.4 s, or about 57 minutes per run, and nobody waits 57 minutes for a signal on a branch. Adding concurrency looks like a ThreadPoolExecutor and three lines, and then the provider's rate limit arrives: at 1,000 cases and roughly 2,300 tokens each, a run consumes 2.3 M tokens, so an organisation limit of 200,000 tokens per minute puts a hard floor of about twelve minutes on the run no matter how many workers you start. Exceeding it returns 429s — the HTTP status a provider sends to say you are over your quota and should come back shortly — and the naive harness has no idea what a 429 is. What it does with one depends entirely on the except clause somebody adds under deadline pressure, and the tempting one marks the case failed, which is how a green suite becomes 0.71 and a morning gets spent bisecting a prompt that never changed.

Both of those durations are one multiplication and one division, and the second is the one worth understanding, because it is the reason the obvious fix stops working. The serial figure is the easy one: a thousand cases at three and a half seconds each is three and a half thousand seconds, which is a little under an hour. Concurrency attacks exactly that, by having many cases in flight instead of one, and for a while it works. Then the second calculation takes over. A run has to move 2.3 million tokens through the provider, and the provider will accept 200,000 of them a minute across your whole organisation. Dividing the work by the permitted rate gives a little over eleven minutes, and that quotient does not contain the word "workers" anywhere. Starting more of them changes when the tokens arrive, not how many are permitted, so past a certain point extra concurrency buys nothing except the 429s the paragraph above described.

That is a shape worth recognising because it recurs: parallelism removes a constraint on how much can be in flight and never removes a constraint on total throughput. When the binding limit is a rate, the only remaining levers are sending fewer tokens, sending them less often, or asking for a larger quota.

The reason that particular except clause is so destructive is that it changes what the number means without changing what the number looks like. A score is supposed to be the share of cases the system got right. Marking an unanswered case as failed silently redefines it as the share of cases the system got right and the network delivered, and those two quantities move for entirely different reasons. Worse, the second one is not stable across runs, because how many 429s a run collects depends on what else your organisation was doing at the time. A gate watching that number is now partly a gate on other teams' traffic, and nothing in the output says so.

Handling this properly means a token-bucket limiter — a counter that refills at your permitted rate and that every request has to draw from before it is allowed out — metering requests against the same budget the provider is metering, honouring Retry-After when it is present, with jittered exponential backoff and a cap on attempts. It is a well-understood piece of engineering and it is roughly two to three days to write and a week of real traffic to trust, which is not much, and it is also not zero, and it is the first of five such items.

Caching is the one with a bill attached. Changing a grader rubric does not change any model output, and the loop re-pays for all 1,000 generations anyway. At mid-tier pricing of around $2.50 per million input tokens and $10 per million output tokens, one full run is about $4.75 of input and $4.00 of output, call it $8.75. Twelve engineers iterating on rubrics and prompts trigger perhaps fifteen runs a day between them, which is $130 a day, or roughly $2,800 a month — and on one team's own accounting about two thirds of those runs changed only the grader, so two thirds of that spend bought nothing.

Following that bill through its multiplications makes the size of the waste legible. A run moves the 2.3 M tokens from the paragraph above, split between the prompt going in and the answer coming out; at $2.50 per million the input side comes to $4.75 and at $10 per million the smaller output side comes to $4.00, which is the $8.75 a run. Fifteen runs a day is a shade over $130. A month of working days at that rate is roughly $2,800. Two thirds of those runs changed only the grader, and a grader change cannot change a model output, because the output was produced before any grader looked at it. So roughly two thirds of $2,800 was spent regenerating strings that already existed and were identical to the ones being paid for again.

A cache keyed on the SHA of the rendered prompt plus the model id plus the sampling settings removes that waste, and it has to be keyed on all three or it will serve a stale output after a model change and produce a silent, confident, wrong trend line.

The rule behind those three components is general and it is easy to get wrong in the same way twice. A cache key has to name everything the cached value depends on. The value here is a model output, and an output depends on the text that was sent, on which model received it, and on the sampling settings that governed how it was produced. Leave any one of those out of the key and two genuinely different values compete for one slot, so the first one written is the one everybody gets. Dropping the model id is the dangerous omission rather than the obvious one, because a model upgrade is exactly the moment somebody wants a before-and-after comparison, and a key that ignores the model returns the "before" for both sides of it.

Errors are not failures, and the loop cannot tell#

The distinction that matters most in a gate is between a case the model got wrong and a case that never got an answer. They arrive at the same except block and mean opposite things: one is a regression, the other is infrastructure. The state machine a serious runner implements looks like this, and the important property is that two of the terminal states are neither pass nor fail.

Excluding errors from the denominator is the correct behaviour and it needs a guard, because a run where 180 of 1,000 cases errored and the remaining 820 scored 0.87 is not a run that should report 0.87 with no further comment. The rule that works is to fail the run itself above an error threshold — 2% is a reasonable starting point — and to report the error count and its breakdown next to the score always. Getting this right is about three days including the reporting.

The review queue and the variance test#

The remaining two are the ones that make the comparison lopsided in a way that lines of code do not capture.

A review queue is where human labels come from, and 1,000 cases labelled by two annotators at ninety seconds each is fifty hours of work per full pass — a thousand cases judged twice is two thousand judgements, and two thousand judgements at a minute and a half apiece is three thousand minutes. Making those fifty hours usable needs assignment so two people do not label the same batch by accident, progress tracking so a partial pass is resumable, a view that shows the case and the raw output side by side without the reviewer having to hunt, the ability to record a category from a fixed taxonomy rather than free text, adjudication when two annotators disagree, and an audit trail of who changed what. Built well enough that a support colleague will use it without a training session, that is four to six weeks of an engineer, and it keeps needing attention because the people using it are not the people who wrote it.

Variance handling is the other, and it is the one that decides whether the numbers mean anything. The run before a prompt change scored 0.88 and the run after scored 0.86, and the loop reports both as bare numbers with no way to answer whether anything happened. The test that answers it is a paired comparison over per-case results: of the 1,000 cases, 931 gave the same verdict both times, 19 flipped from fail to pass, and 31 flipped from pass to fail. McNemar's test on those 50 discordant pairs — it asks only whether the flips in one direction outnumber the flips in the other by more than chance would produce, ignoring the cases that did not move — gives a p-value around 0.12, so the two-point drop is inside what this suite produces by chance, and the correct action is to change nothing.

Why the test ignores the cases that did not move is the part worth holding onto, because it is what separates a paired comparison from a comparison of two averages. A case that passed both times is consistent with the change having done nothing and equally consistent with the change having done something that did not reach this case. It cannot tell the two apart, so it contributes nothing either way. All the evidence is in the cases that moved. There are 50 of those, and the question the test asks about them is whether they moved lopsidedly. Suppose the change really did nothing, so that every one of those 50 flipped for reasons unconnected with it: each would then be as likely to have flipped one way as the other, and the expected split is even. What was observed was 19 against 31. A split that uneven, or worse, turns up often when fifty coins are tossed — often enough that it happens roughly one time in eight, which is what the p-value near 0.12 is reporting. The suite is not saying the change was harmless. It is saying this suite, at this size, cannot distinguish it from nothing, which is a different and more useful statement.

Without that machinery a team reverts good changes and ships bad ones with equal confidence, several times a quarter.

The total is somewhere around eight to ten engineer-weeks to reach parity on the capabilities a hosted tool ships on day one. That is the number the argument is actually about.

Three conditions that keep the harness#

The condition that most often decides it is not price. If your data cannot leave your network — health records, or a customer contract that names the sub-processors you are permitted to use — then the shortlist of hosted options collapses, and the remaining question is whether a self-hostable platform is worth operating. Best AI evals products for self-hosted / on-prem deployments is a reasonable survey of that narrower field, and running one of those in your own cluster is its own ongoing cost that has to be compared against the eight to ten weeks above rather than against zero.

The second condition is case shape. If an eval means driving a simulated customer through six turns against a tool sandbox with seeded fixtures, most platforms model that awkwardly or not at all, and you will spend more effort fighting the dataset schema than you would writing the runner. The harness wins here because it is the only thing that can express what you are measuring.

The third is scale and ownership together. A suite of 300 cases with one grader family, owned by two or three engineers who both understand it, does not exercise the review-queue axis at all and barely exercises the variance axis. The harness is genuinely sufficient, and it stops being sufficient at a predictable moment — when the first non-engineer needs to label something, or when the suite crosses roughly a thousand cases.

Against those three conditions sits a cost that never appears in the pull request. A harness that closes all five gaps is no longer 200 lines; it is two or three thousand lines of concurrency, retry logic, caching and a small web app, written by one person, with no external users to keep its interfaces honest. The failure mode is ordinary and slow: the author changes teams, the limiter breaks when the provider changes its quota headers, the cache serves stale outputs for a week before anyone notices, and the suite quietly stops being trusted. That risk is manageable with two owners and a test suite for the harness itself, and it belongs in the comparison alongside the eight to ten weeks, because it is the part that a vendor's on-call rota is actually being paid for.

It is also worth being precise about what "it already works in CI" establishes. It establishes that the happy path produces a number on the current suite at the current size, which is the one axis on which hand-written code always looks good, since it was written against exactly that path. It says nothing about the second identical run, about the run that starts while a provider is degraded, or about the run at 1,000 cases with four annotators feeding it. Those are the conditions under which the five items above stop being theoretical, and none of them are exercised by the evidence on offer.

So the reflex — same score, same failures, therefore the vendor adds nothing — is not wrong about the score and is measuring the wrong thing. What a bought tool sells is the second run, the tenth-scale run, and the run somebody else has to read, and those five capabilities cost eight to ten engineer-weeks to reproduce plus an ongoing owner. Write the list with prices on it before the decision, and the argument stops being about whether the code is impressive.

What none of this tells you is which tool to buy if you decide to buy one. Vendors describe all five of these capabilities in their documentation, in nearly identical language, and reading the documentation is not how you find out which of them is real.