Auditing and evaluating hosted tools

A hosted tool result is a lab report with the methods section removed. It states a finding, it is usually correct, and it is not something a second party can check, because the apparatus was disassembled when the turn ended. That is not an accusation. A response body is an inference API's output, designed to be fed back into the next request, and it is neither intended nor priced as an audit log; expecting it to serve as one is a category error that costs nothing until the day it costs everything.

A hosted tool in this chapter is the provider-executed kind from chapter 1: a capability the provider both defines and runs inside your turn — a web search, a document fetch, a search over a hosted file store, a sandboxed code run — with no handler of yours anywhere in the path. The comparison to a lab report is an analogy, and the place it breaks is the place the rest of the chapter lives. A report with its methods section removed still describes work somebody could in principle go back and inspect, because the apparatus existed somewhere and somebody stood next to it. A hosted tool run leaves nothing of that kind on your side of the boundary. The container was destroyed, the fetch happened on a network you have no access to, and the response body was the only trace that ever crossed into your process, so there is no second place to look rather than a second place that is inconvenient to reach.

This chapter is about two consequences of that property, and they arrive in different meetings. The first is an incident review that needs to know what a calculation actually did. The second is a nightly evaluation suite whose pass rate moved and whose owners want to know why. Both come down to the same question — which variables in this system have identifiers you hold — and both have the same fix, which is capturing whole request and response bodies and then splitting one suite into two.

What each hosted tool actually hands back#

The general rule is unhelpfully simple: you get what the provider chose to put in the response, at the time the response was produced. What makes it operationally tricky is that "what the provider chose" differs per tool, and the differences are not intuitive.

The reason a response body is shaped that way is not secrecy, and reading it as secrecy leads to the wrong fix. A response body exists to be fed back into the next request, so its contents are settled by a single question: what does the model need in order to continue this conversation? It needs the answer the tool produced. It does not need the HTTP status of a page fetch, the package versions in a sandbox, or which chunking model scored a corpus, because none of those changes what comes next. That principle predicts the shape of the table below rather than each cell of it, and what it predicts reliably is that the fields an incident review wants are the ones that fail the continuation test.

The code-execution row contains both a relief and a trap. The code is in the response — the server_tool_use block carries it verbatim — so a team that captured whole response bodies can read the exact Python that produced a wrong number and will probably find the bug in a minute. A team that logged the final answer and the tool-call ids has nothing, and no way to get it back, because the response body was the only place that code ever existed on your side. The tool-call id itself is not a handle: srvtoolu_01A9k… is an identifier inside a response body, and no endpoint takes it and returns what happened.

That distinction is worth stating slowly, because the identifiers in this system all look alike and do not behave alike. A string is a handle only if something will dereference it — only if there is an endpoint that accepts it and returns the thing it names. A conversation id is a handle in that sense, and chapter 3 was largely about what the provider does and does not return when you dereference one. srvtoolu_01A9k… is a label: it distinguishes one tool block from another inside a body you are already holding, and there is nothing to hand it to. A log line carrying it therefore records that a tool ran and records nothing about what the tool did, and the two are indistinguishable until the day somebody asks.

Even with the code, the environment is not there. If a failure depended on a library version — a currency-rounding helper whose behaviour changed, a date parser that started interpreting 03/04 differently — the code alone does not tell you, and the container that ran it was destroyed. Container reuse makes this worse in an interesting way: a container carried across requests holds files and installed packages from earlier turns, so the same code can produce different results depending on what a previous, unrelated conversation left on disk. That is a variable with no identifier in your logs at all unless you were tracking the container id yourself.

The mechanism under that sentence is worth having in full, because it is what makes an unreproducible result possible rather than merely unlucky. A container is a filesystem with a process running on it. If the provider keeps it alive between requests, files written during an earlier turn are still on that filesystem and packages installed during an earlier turn are still importable. Code that opens a path it did not create, or imports a name it did not install, then simply succeeds, with no error to log, because from the code's point of view the file and the module are present. The same source therefore has two available behaviours — one on a fresh container, one on a warm one — and nothing in the code or in the request distinguishes which it got.

Whether a given provider reuses containers, and which of the columns above hold for the tools you have switched on, are properties of their implementation rather than of your request, stated here as the tools are currently specified. The step to make somebody demonstrate rather than assert in review is a captured pair of consecutive turns showing whether the container id is the same string across both, and a captured body showing which fields actually arrived on your own account.

The fix is unglamorous and has to be in place beforehand. Capture the full response body, verbatim, for every request that carries a hosted tool; set the include flags that return tool results; record the model string the response reports rather than the alias you sent, along with the container id and every tool type version.

audit.write({
    "ts": now_iso(), "case": case.id,
    "conversation": conversation.id,
    "model_sent": MODEL, "model_served": resp.model,
    "tool_versions": [t["type"] for t in request["tools"]],
    "container": getattr(resp, "container", None) and resp.container.id,
    "request": request,                 # full body, prompts included
    "response": resp.model_dump(),      # full body, tool blocks included
    "usage": resp.usage.model_dump(),
})

Two of those fields carry more weight than their length suggests. model_sent and model_served are separate because the string you send may be an alias, and an alias is resolved by the provider at the moment the request arrives — so it names whichever version sits behind it today rather than the one that ran last Tuesday. A record of what you sent is a record of the name of a moving target. The container id is there for the reason just given: it is the only string that separates a fresh environment from a warm one, it appears in the response rather than in your request, and so it is either captured on the way past or it is gone.

That record is worth writing even though parts of it are unreadable. The encrypted web-search content cannot be inspected by you, but it can be passed back to the model, which means a captured conversation can be re-run against the same retrieved material rather than against whatever the web says today — the difference between reproducing a run and re-enacting it.

That pair of words is worth pinning down, because the rest of the chapter turns on it. Reproducing a run means presenting the model with the same input bytes it saw before, so that any difference in what comes back is attributable to the model or to sampling and to nothing else. Re-enacting means performing the same steps again against a world that has moved on, which answers the question of what would happen now and cannot settle what happened then. A captured body supports the first because the bytes are in your hand. A live re-run supports only the second, however carefully it is arranged.

The same opacity, arriving as a metrics regression#

The second meeting is about a nightly evaluation suite. Two hundred production conversations are replayed and graded every night; half of them run through hosted conversation state and a hosted code-execution tool. This week the pass rate fell from 78.5% to 72.5%, and the natural conclusion is that the provider shipped a model update. The proposed action is to pin the version string and re-run: if the pass rate recovers it was them, if it does not it was us.

The first problem with that plan is arithmetic. Six points on n=200 is twelve conversations. The standard error on a proportion near 0.75 at that sample size is about 3.1 points, so a six-point move is roughly two standard errors — enough to raise an eyebrow, not enough to name a cause, and well within the range a re-run on a different night can produce with nothing having changed.

Both of those figures are worth deriving rather than accepting, because everything the meeting decides rests on them. The fall is 78.5% minus 72.5%, which is six points of pass rate. One point of pass rate on a suite of 200 conversations is two conversations, because one percent of two hundred is two, so six points is twelve conversations that passed a week ago and fail now. Twelve is a small enough number that a reader should already be uneasy about naming a cause for it, and the standard error is what puts that unease on a footing.

A standard error is how far a measured proportion would wander from run to run purely because the suite happens to contain one particular set of two hundred conversations rather than some other set. Each conversation either passes or fails, and the suite's pass rate is the average of those individual outcomes, so the spread of that average is fixed by two things only: how close the underlying pass probability sits to a half, and how many outcomes are being averaged. The calculation is to multiply the pass share by the fail share, divide by the number of conversations, and take the square root. Near 0.75 the pass share times the fail share is a little under a fifth. Dividing that by 200 leaves something under a thousandth. The square root of that is about three hundredths of the scale, which is the 3.1 points quoted above.

Each of those steps is allowed because of a property the suite has by construction. Multiplying the two shares is legitimate because every conversation has exactly two possible outcomes, so one share determines the other. Dividing by the count is what converts the spread of a single outcome into the spread of an average over that many outcomes, and it is the reason a bigger suite is steadier. The square root is there because those first two operations produce a spread in squared units, and the root converts it back onto the scale of the pass rate itself. That is also why quadrupling the number of conversations only halves the wobble rather than quartering it — a fact worth carrying, because it prices any proposal to fix this by making the suite bigger.

The 3.1 points is a floor rather than an estimate, and the honest reading is the wider one. The calculation assumes the two hundred conversations fail independently of each other, and they do not: they share prompts, a corpus and a model, so one upstream change moves several of them together. Correlated failures widen the band and never narrow it, which is why the spread you measure by re-running is the number to trust and the derived one is only the sanity check that tells you whether measuring is worth the evening.

Two standard errors sounds like a lot and is not, and the reason is where this meeting usually goes wrong. A measured pass rate is not the system's pass rate. It is one sample from the system, in the same way that a single run of any sampled process is one sample rather than a measurement of the process, and it moves between runs whether or not anything was done to it. The question in front of a reviewer is therefore not whether the number moved but whether it moved further than it moves when nobody touches anything. Nothing inside a single pass rate answers that, and the only instrument that does is the same suite run again.

Before anyone hunts for a cause, run the suite three times against yesterday's configuration and see how wide the band is. Teams that skip this step spend a week attributing noise.

The second problem is that pinning the model controls one variable out of at least five, and the other four are exactly the ones hosted primitives make invisible.

What makes that fatal rather than untidy is a rule about experiments short enough to state in one sentence. A change in an outcome is attributable to a change in an input only if every other input held still, and an input holds still only when somebody holds an identifier that pins it. Both halves are load-bearing. An input nobody has named is not being held still by anyone's discipline, however careful they are, and an input whose identifier lives on the provider's side is held still only for as long as they leave it alone. Run the inputs to a hosted turn past that test and the model identifier is the one that passes.

The input to each turn is assembled by the provider, so for the hosted half of the suite the fixture is a conversation handle, and chapter 3 established what that means: the stored conversation can compact between one night's run and the next, and a fixture whose contents can change is not a fixture.

The retrieval corpus moves, because the vector store the agent searches is fed by a sync job, and a re-sync that changes a rate table changes the retrieved chunks with it — a content change, entirely your own doing, that arrives looking exactly like a model change. The scoring of that corpus moves too, and this one is nobody's doing: hosted file search embeds and chunks documents with the provider's models, and those are upgraded without appearing in any version string you send, so an identical corpus and an identical query can return different chunks with no parameter to pin. And the execution environment moves, since base image, package versions and container reuse all sit behind the code-execution tool, none of them versioned in your request, all of them capable of changing arithmetic.

Each of those can produce a six-point swing, and none of them moves the model identifier. Re-running with a pinned version and seeing no recovery tells you nothing about which one it was; it merely eliminates the variable that was easiest to name.

Two suites, because they answer different questions#

The way out is not better instrumentation of the hosted suite. It is admitting that one suite has been asked to answer two questions and splitting it.

A hermetic suite gates deploys and must be a function of your code alone, hermetic in the sense build systems use the word, which is that everything the run depends on is inside the repository and nothing reaches out to a network or a clock. Convert every fixture from a handle into content: take the captured request and response bodies from the audit record, extract the full message array including retrieved chunks and tool results, and replay that array against a stateless call with no hosted state and no hosted tools. Stub the tools with their recorded outputs. Pin the model to a dated identifier and hash the fixture set so a change to the inputs is a reviewable diff rather than a surprise. When this suite moves, it moves because a human changed a prompt, a schema, or a handler, which is the only signal a deploy gate can act on.

The conversion from a handle into content is the whole of that trick, and it is worth seeing why it works rather than taking it as a procedure. A handle is turned into an input by the provider at the moment the suite runs, which means the fixture's contents are whatever their store holds tonight — and a stored thread can compact underneath you, as chapter 3 established. Content committed to the repository is turned into an input by nobody: the bytes in the file are the bytes that go over the wire, and the only way they change is a commit with an author on it. A fixture is not something you name, it is something you hold, and the audit capture is the machinery that converts one into the other.

Stubbing the tools with their recorded outputs has a price that is better stated than discovered. A suite replaying recorded tool results is no longer testing the tools. It is testing everything downstream of them, on the assumption that each tool still behaves as it did on the day of capture — which is precisely the assumption the other suite exists to check. That is why the answer is two suites rather than one better one, and it is the part of this arrangement that people resist, because it means neither suite alone tells you the system is healthy.

A live suite runs the real thing — hosted state, hosted tools, today's corpus — and must not gate anything. Its job is to notice that the world changed, and the appropriate response to a fall in it is an investigation, not a rollback. Run it on a fixed schedule, three passes per night, and plot the band rather than the point.

Together they diagnose the argument that started this section. Hermetic suite steady and live suite down means something on the provider's side or in your corpus moved. Both down means your change. Hermetic down alone means someone changed a fixture. That distinction is available for the price of storing full request and response bodies, and it is unavailable at any price if the only durable record is a handle.

The reason that reading works is that the two suites share your code and differ in everything else, so the difference between them isolates a class of variable rather than a specific one. That is a weaker conclusion than a controlled experiment would give, and it is the strongest one available when several of the candidates have no identifier anybody could pin.

Two limits on the capture, and one question it does not answer#

The audit record is your record, produced by your process, which makes it excellent evidence in a conversation with a customer or a regulator about what your system did, and weak evidence in a dispute with the provider about what their system did. For that, the request id in every response is the only shared reference, and it belongs in the audit record for exactly that reason.

The asymmetry there is worth naming, because it is not about anyone's honesty. Evidence persuades in proportion to how much the party it is used against had a hand in producing it. Your capture was written by your process from bytes only you were holding, so a provider can dispute what it shows without accusing anybody of anything. A request id was minted on their side and appears in both sets of records, which makes it the one string that names the same event twice.

The capture is also subject to your own retention rules: it contains full prompts and full customer text, so it is personal data, which means the erasure fan-out from chapter 3 has one more target in it. An audit log that quietly becomes the one copy of a deleted customer's transcript is a problem you created while solving a different one.

The remaining question is whether running evaluations against hosted primitives is the right thing to be doing at all. For acceptance, yes — the hosted path is what customers get, and a suite that never exercises it is measuring a system you do not ship. As the regression suite your engineers block merges on, no, because a regression suite has to attribute failures to changes, and hosted primitives are specifically the components whose changes you cannot see. The instinct that the two must be the same suite is what makes this feel like an impossible problem rather than a two-line split in a CI config.

Everything in this chapter follows from one sentence: nothing you did not capture exists. That is true of the code a hosted sandbox ran, the chunks a hosted search retrieved, the container that carried state from an unrelated conversation, and the input a compacted thread presented to the model last Tuesday. The capture is cheap relative to what it buys, it has to be turned on before the day it is needed, and its second use — as the content that converts an evaluation fixture from a handle into something a deploy gate can trust — is the one that pays for it weekly rather than once a year.

That one sentence — nothing you did not capture exists — is a working rule rather than a literal fact, and the difference matters when somebody quotes it back at you in review. Some of what you did not capture does still exist for a while: a conversation can be read back, a file id can be resolved, a dashboard can be opened. What is not true is that any of it is guaranteed to be there when you need it, or to be in the shape it was in when it mattered. The scenario above is the honest version — the read-back returned something, and what it returned was a summary rather than the turns. A record that may or may not exist, and may or may not have been rewritten since, cannot be the thing an incident review rests on, which is why the rule is written as though the material were simply gone.

The dashboard is where that distinction is easiest to lose, and losing it is what decided the scenario a year before the Saturday. A dashboard is a rendering of what the provider retains under their own policy, which is not the same object as a copy you hold: its contents are theirs to shorten, summarise or age out, and you learn which has happened by needing it. Declining a capture because the dashboard shows the calls is a decision to depend on a record whose terms you did not set.

Everything in this topic has now been argued from evidence: the trace, the deletion receipts, the cost and latency measurements, the red-team results, the audit records. What has not been done is the decision. Handing state and execution to a provider is sometimes exactly right and sometimes indefensible, and the next chapter works two concrete cases — one where the answer is a quick yes, one where the analysis is the whole job — and ends where a decision like this has to end, which is written down.