The run record

An eval loop that stores a final answer per case is storing the model's prose about what it did. For a chat assistant that is most of what happened; for an agent that calls tools, spends tokens and takes time, it is a summary written by the component under test. The run record is the alternative: a flat object per case holding the transcript of that case's execution, kept in full, so that a later assertion can read what was sent rather than what was said about it.

This chapter is about what goes into that object and where each part is visible only once. An agent turn passes through three parties, and at every boundary a piece of evidence is either captured or destroyed by code doing something entirely reasonable — most of all where a JSON object from the provider becomes Python arguments to a function, because a coercion layer there can turn a wrong number into a right-looking one without raising anything.

What a run is made of, and where each part is visible once#

Here is what an eval store looks like when the loop was written to produce an answer, which is what most of them were written to produce:

select * from eval_results where case_id = 'refund_partial_damaged';

 case_id                 | input                        | answer                              | passed
-------------------------+------------------------------+-------------------------------------+--------
 refund_partial_damaged  | "the mug arrived cracked"    | "I've refunded $42.00 to order …"   | t

Four columns. The answer column contains the sentence the agent wrote about a tool call — the model's request that the harness run one named function with named arguments and hand the result back to it — and a sentence can be correct English about an incorrect action. The call itself, its name and its arguments and the order they arrived in and the result the handler returned, was constructed inside the harness, sent over the wire, executed, and then discarded.

Writing a better assertion later does not recover any of that, and the reason is a property of what was stored rather than a lack of ingenuity. The answer column holds the output of a function whose input was the entire run, and that function is many-to-one. A run in which the model asked for amount_cents: 4200 and a run in which it asked for amount_cents: 42 can both end with the sentence I've refunded $42.00 to order ORD-448192, because the sentence reports what the agent took itself to be doing rather than the bytes that left the harness. Two different runs, one stored string. An assertion is itself a function of what was stored, so an assertion over that column can only separate runs that the column already separates, and the separation an incident would need was gone before the row was written. That is why the capture has to come first: it is not that nobody thought of the right check, it is that the right check has nothing to read.

The material that goes missing is spread across three hops, and each hop carries data that exists there and nowhere else.

The annotation on the third hop is the one that decides most incidents. The provider returns a tool_use block whose input is a JSON object; the harness deserialises it and hands it to a function typed amount_cents: int; the function signature is satisfied and the refund goes out. Whatever the model actually wrote — the integer 42, the string "42", the float 42.0, or the integer 4200 — the handler saw an int and no error was raised. A record that stores only what the handler received has already lost the distinction between a model that emitted the wrong number and a coercion layer that turned a right one into a wrong one.

Losing that distinction costs more than a detail, because it makes one particular question undecidable from anywhere downstream. Two stages sit in series at that boundary. The model chooses a number and puts it in a JSON object; the dispatcher turns that object into arguments the handler will accept. What the payments API receives — and therefore what any ledger, any production log and any customer complaint records — is the output of the second stage applied to the output of the first. A single observation of two stages in series cannot say which of them introduced a defect, for the same reason that a total says nothing about which of its two summands moved. Telling them apart needs an observation taken between the stages: the JSON exactly as it arrived, before it was deserialised, next to the arguments the handler was finally called with. That intermediate value exists inside the harness for the duration of one function call, and by default it is written nowhere.

{
  "type": "tool_use",
  "id": "toolu_01Qk9",
  "name": "issue_refund",
  "input": {"order_id": "ORD-448192", "amount_cents": 42}
}

That block, verbatim, is the primary evidence. Store the bytes the provider sent, not a normalised rendering of them, because normalisation is exactly the process under suspicion. Serialising the parsed object back to JSON before storing is not the same thing: json.dumps(json.loads(x)) has already lost "42" versus 42 if anything in between applied a model with a declared type, and it loses key order, duplicate keys and numeric precision in the general case.

Coercion is worth being concrete about, because the libraries in question are the ones most teams already use and their defaults are permissive by design. Pydantic in its default lax mode accepts "42" for an int field, accepts 42.0 because it is integral, and accepts True for an integer in some configurations; strict=True rejects all three. A hand-rolled dispatcher that calls int(value) is more permissive still, since int(" 42\n") is 42 and int(42.9) is 42. Every one of these is a reasonable choice for a production handler that would rather serve a request than 400 it, and every one of them destroys the evidence of what the model meant. Capturing both sides of that boundary costs one dictionary per call and turns "the model got it wrong" into a question with an answer.

The record#

A run record is a flat object per case with the transcript inside it. The shape below is deliberately unexciting; the value is in what it refuses to drop.

from dataclasses import dataclass, field

@dataclass
class ToolCall:
    call_id: str
    name: str
    arguments_raw: str        # exact JSON text from the provider
    arguments_parsed: dict    # after json.loads, before handler coercion
    arguments_received: dict  # what the handler actually saw
    result_raw: str | None
    error: str | None
    started_at_ms: int
    duration_ms: int

@dataclass
class LLMCall:
    model: str                # the model that served it, echoed by the API
    request_messages: list[dict]
    response_blocks: list[dict]
    stop_reason: str
    input_tokens: int
    output_tokens: int
    cache_read_input_tokens: int
    cache_write_input_tokens: int
    time_to_first_token_ms: int
    duration_ms: int
    retries: int
    retry_sleep_ms: int

@dataclass
class RunRecord:
    case_id: str
    suite_run_id: str
    git_sha: str
    prompt_version: str
    temperature: float
    tool_schema_hash: str
    llm_calls: list[LLMCall] = field(default_factory=list)
    tool_calls: list[ToolCall] = field(default_factory=list)
    final_message: str = ""
    wall_clock_ms: int = 0

Three fields there tend to get argued about in review, and each earns its place by answering a question that comes up during an outage rather than during normal operation. model is read from the response rather than from your config, because the two disagree exactly when it matters — during a fallback, an alias rollover, or a deploy that did not pin what you thought. tool_schema_hash is a digest of the tool definitions as sent, which is what lets you tell "the model called it wrong" from "we changed the signature under it". retry_sleep_ms is time your harness spent asleep inside a client library, and chapter 6, on why a timing assertion measures your own concurrency, is largely about why that number has to be separable from the rest.

The declared type of arguments_raw is doing work as well. It is a string rather than a dictionary because a dictionary is what you have after parsing, and the parse is one of the steps under suspicion. A dictionary built from {"amount_cents": "42"} does differ from one built from {"amount_cents": 42}, so that particular distinction would survive. Others do not: an object whose key appeared twice collapses to the same dictionary as one where it appeared once, and a number that passed through a parser configured to produce floats is by then indistinguishable from a number written as a float in the first place. Keeping the text costs a few hundred bytes per call and leaves the record able to answer questions nobody had thought of when the recorder was written.

Each stored field is load-bearing for something specific:

Read the right-hand column as the agenda for the next three chapters. None of those assertions can be written against a string.

Capturing it without rewriting the agent#

The instrumentation is smaller than it looks, because there are only two places to hook. Every piece of evidence in a run crosses one of two boundaries: it leaves or enters the process on its way to the provider, or it passes from the harness into a tool handler as a function call. There is no third place where evidence comes into existence, because the model's choices reach you as bytes over HTTP and the effects of those choices happen behind a function signature. Two wrappers therefore intercept all of it, and neither of them sits inside the agent's own control flow — which is what stops the instrumentation rotting the next time somebody refactors the loop.

Wrap the provider client so that every call appends an LLMCall, and wrap the tool dispatcher so that every invocation appends a ToolCall with both the parsed arguments and the arguments as the handler received them:

def record_tool_dispatch(record: RunRecord, block: dict, handler) -> str:
    started = time.monotonic_ns() // 1_000_000
    raw = json.dumps(block["input"], separators=(",", ":"))
    received, error, result = None, None, None
    try:
        received = coerce_to_signature(block["input"], handler)
        result = handler(**received)
    except Exception as exc:                # capture, then re-raise
        error = f"{type(exc).__name__}: {exc}"
        raise
    finally:
        record.tool_calls.append(ToolCall(
            call_id=block["id"], name=block["name"],
            arguments_raw=raw, arguments_parsed=block["input"],
            arguments_received=received or {}, result_raw=result,
            error=error, started_at_ms=started,
            duration_ms=time.monotonic_ns() // 1_000_000 - started))
    return result

The finally matters. A run that raised is the run you most want the record for, and a naive implementation that appends after a successful return keeps evidence for every case except the interesting ones. The same argument applies one level up: a case that times out, exhausts its retries, or is killed by the suite's own deadline should still write whatever it accumulated, which means the record is flushed by the runner rather than by the code path that was supposed to finish.

Two properties of the list are easy to lose and expensive to reconstruct. The first is order: tool_calls must preserve the sequence in which the provider emitted the blocks, because an assertion like "the agent looked up the order before refunding it" is a claim about order and nothing else, and a dictionary keyed by tool name flattens it away. The second is multiplicity — a model may emit two tool_use blocks in one assistant turn, and a harness that executes them concurrently will append them in completion order rather than emission order unless it takes care. Keeping call_id alongside the position lets you reconstruct the intended sequence from the provider's blocks even when execution was parallel, and it is what pairs each result back to the call that asked for it.

Storage is worth a number, because "record everything" is a phrase that gets budgeted at zero and then panics somebody. A support-agent case with six turns, two tool calls and a 1,200-token system prompt serialises to roughly 35–45 KB of JSON. Take the middle of that range and multiply by the 300 cases in a suite, which is the right operation only because each case is stored whole and shares nothing with its neighbours: a suite run is therefore around 12 MB. A team merging thirty pull requests a day and running the suite on each multiplies that by thirty again, giving about 360 MB a day, and thirty of those days come to 11 GB a month before compression.

Compression takes most of it back, and the arithmetic is worth finishing because the intuition tends to stop at the word "gigabytes". Gzip takes JSON transcripts down by roughly 85%, so what lands on disk is about a seventh of what was generated, and a seventh of eleven gigabytes is under two gigabytes a month. A retention rule of ninety days for the full record plus permanent retention of the summary row therefore keeps three such months live at once. That is a smaller line item than the model calls that produced it, and it is the reason to decide the policy now rather than after someone notices the bucket.

Two practical objections come up when this is proposed, and both are better decided up front than after the first audit. The first is personal data: a transcript of a support conversation contains a customer's name, email address and order history, and storing it for ninety days in an engineering bucket is a decision with a legal shape. Eval cases are usually synthetic or scrubbed at authoring time, which sidesteps most of it, and the discipline that keeps it sidestepped is to scrub at the point cases are created rather than at the point records are written — redaction applied to a stored record is exactly the truncation problem described below, and it will remove the argument value you needed. The second objection is streaming: when responses arrive as server-sent events there is no single JSON body to save, so the recorder has to assemble the blocks itself and keep the assembled result plus the timing of the first event. That assembly is a few dozen lines, and it is where time_to_first_token_ms comes from.

A record, a trace, and a log#

A transcript is not reproducibility. Recording what happened is different from being able to make it happen again, and the record above stops short of the second on purpose. Even with the same messages, model, temperature and tool schemas, a provider serving a mixture-of-experts model — one that routes each token through a subset of specialised sub-networks rather than all of them — under variable batching will not always return the same tokens, so a transcript is evidence about one run rather than a recipe for it. What the extra fields buy is attribution: when a case flips, prompt_version, git_sha, model and tool_schema_hash narrow the cause to something you changed or something the provider changed, which is usually the whole investigation.

That non-determinism has a mechanical cause, and knowing it matters because the obvious defence against it does not work. Setting temperature to zero makes the picking step take the highest-scoring candidate instead of drawing one at random, which removes the randomness your harness controls and none of the rest. The scores being picked from are sums of an enormous number of floating-point products, and floating-point addition is not associative — adding the same quantities in a different order can change the last bit of the answer. A provider computes those sums on hardware that splits the work across many parallel units and combines the partial results in whatever order the split produced, and the split depends on how many requests were batched together at that instant, which depends on other customers' traffic. Identical input can therefore yield scores that differ in their final bits. Almost always that changes nothing. When the two leading candidates are nearly tied it changes which one leads, and because the loop only ever moves forward, one fragment landing differently changes every fragment after it.

Attribution is the weaker property that survives all of that, and it is the one worth being precise about. A case's verdict is a function of a definite list of inputs: the case's own text, the prompt template, the code in the harness, the model that served it, the sampling settings, and the tool definitions as sent. Hold every item on that list fixed, vary one, and a change in the verdict belongs to the item you varied. That is the entire logic of an eval suite, and it is why the record carries prompt_version, git_sha, model, temperature and tool_schema_hash next to the transcript — each one is an item on that list, written down at the moment it was used rather than reconstructed afterwards from a config file that has since moved on.

A field that is not recorded does not make the investigation slightly harder. It removes a conclusion. Attributing a flip to your diff means ruling out every other item on the list, an item nobody wrote down cannot be ruled out, and rival explanations are not partial credit that accumulates into confidence — one of them left standing is enough to make the attribution unsound. Drop tool_schema_hash and "the model started calling it wrong" and "we changed the signature under it" both remain live, with different fixes and different owners. Drop model and a case served by a fallback during a provider incident looks exactly like a regression in your prompt. Drop temperature and a run that was sampled is indistinguishable from one that was not, which is the difference between a real flip and a coin landing the other way.

A run record is also not your production tracing, though the two are converging and it is worth knowing where. OpenTelemetry — the vendor-neutral tracing standard most observability tools now implement — has generative-AI semantic conventions that define span attributes for exactly this material, the model, the token counts, the tool name, the arguments, so an agent instrumented for production observability is already emitting most of a run record, and platforms that ingest traces will let you assert against them. If your service is traced, the cheapest capture is often to read your own spans back rather than to build a parallel recorder. The distinction that survives is one of purpose: a trace is sampled, retained briefly and read by a human during an outage, while a run record is complete, retained per suite run and read by a comparison operator.

It is not a log for humans either, which is a design constraint rather than an aesthetic one. Pretty-printing, truncating long fields to keep the terminal readable, or redacting arguments before storage will all silently disarm the assertions that follow — a record truncated at 500 characters cannot tell you what the fifth tool call did. Store the complete structure and render it prettily at read time.

Everything an eval suite can say about an agent is bounded by what its harness wrote down, and the default write is the one thing the agent authored about itself. Keeping the request bodies, the raw tool arguments on both sides of the coercion boundary, the usage figures and the timings turns each case from a verdict into evidence, at a storage cost well under the model calls that produced it. Instead of re-running cases with print statements to find out what was sent, you query.