The tool execution audit log

An audit log for agent tool execution is a different artifact from the conversation transcript, kept for a different purpose, on a different retention clock, in a store with different access controls. The transcript is what the model was sent and what it produced; the audit log is what your code did with that output — the call it actually made after normalising arguments and applying scope, who authorised it, how long it took, what came back, and how many times it was attempted.

The transcript looks like it should be enough, which is why teams keep discovering that it is not. It shows a tool call block requesting update_account(account_id="acct_7741", tier="free", effective="immediate") and the tool result the model was given, which was {"ok": true}. It does not show what the tool wrapper sent after it normalised the arguments, resolved effective="immediate" into a timestamp, and applied the tenant scope from chapter 2, on scoped allowlists. It does not show that the first attempt to that endpoint timed out at 9 seconds with the outcome unknown, that a retry went out with the same idempotency key, and that the downgrade therefore appears once in the billing system and twice in the webhook stream. It does not record who approved the call, because approval happened in a Slack interactive message handled by a different service. And it says nothing at all about calls that were denied by policy, because a denied call never reaches the model as a tool result in some architectures, and in others it arrives as a sentence the model summarises away in its next turn.

Asked why more retention would not have fixed that, the answer is the one worth carrying out of this chapter, and it is about layers rather than about volume. A transcript is a record of an interface: what was handed to the model and what the model handed back. The retry happened underneath that interface. The tool wrapper timed out, decided on its own to send the request again, got a result, and returned one tool result upwards, because returning two would have made no sense to the caller. So the model was told {"ok": true} once, and one is the number the transcript honestly records. The count of attempts is not a fact that was dropped from the transcript for want of space; it is a fact that never crossed the boundary the transcript observes. Keeping that transcript for a year instead of ninety days produces a longer record of the same silence. This is the general shape of the failure, and it generalises past logging: when an instrument is placed at the wrong layer, its resolution and its retention are both irrelevant, because neither of them is what is missing.

What a record has to carry#

The unit of the log is one tool invocation attempt, not one tool call, because retries are attempts and they are exactly what the transcript loses. A record for a single attempt looks like this:

{
  "event": "tool.attempt",
  "ts": "2026-02-26T14:19:07.412Z",
  "run_id": "run_01HQ9F3M2K",
  "turn": 17,
  "tool_call_id": "toolu_01A9c8Xk",
  "attempt": 2,
  "tool": "billing.update_account",
  "tenant_id": "tnt_northwind",
  "actor": {"kind": "user", "id": "u_8812", "surface": "support_console"},
  "on_behalf_of": "acct_7741",
  "approval": {
    "proposal_id": "prop_8b31f0c4a2d7",
    "approved_by": "u_2290",
    "approved_at": "2026-02-26T14:18:51.900Z",
    "plan_digest": "9f2c1a4b7e03"
  },
  "policy": {"decision": "allow", "rule": "billing.update_account@support_console"},
  "args_ref": {"schema": "v3", "digest": "sha256:41b0…", "blob": "s3://tool-args/2026/02/26/…"},
  "deadline_ms": 9000,
  "downstream": {"request_id": "req_9fd2c1", "endpoint": "PATCH /v2/accounts/acct_7741"},
  "outcome": "timeout",
  "duration_ms": 9004,
  "error": {"class": "ToolTimeout", "state": "unknown"},
  "idempotency_key": "idem_run01HQ9F3M2K_turn17"
}

Every field in there is doing a specific job, and the way to keep a log honest under storage pressure is to be able to say which question each one answers and what breaks without it.

Holding a schema to that table is a stronger test than it looks, and it cuts both ways. A field for which nobody can write the middle column is one somebody added for a reason that has since evaporated, and it will be defended anyway on the grounds that it might be useful. A question with no field under it is an investigation that will fail later, at the worst possible moment, with no warning beforehand. Both faults are invisible while the system is healthy, which is why the table belongs in the repository next to the record's schema rather than in a design document nobody opens.

The record that most often goes missing is the denial. Deployments log successful tool calls diligently and drop denied ones, on the reasoning that nothing happened, and this is the wrong instinct in both directions. A burst of denials against another tenant's resources is the strongest signal you have that something is driving the agent somewhere it should not go, and it is the signal that arrives before the incident rather than after. Denials are also the thing you read when a legitimate workflow silently stopped working, which is the far more common ticket. The same applies to BudgetExhausted from chapter 3, on task deadlines, and to a refused commit from chapter 1, on pinned proposals: those are decisions the system made, and a log of only the actions it took cannot explain the actions it did not.

When privacy will not let you log the arguments#

lookup_customer(email=…, dob=…, last_four=…) is a tool whose arguments are the thing legal will not let you copy into a log store that a hundred engineers can query. The instinct to log everything and sort redaction out later produces the most complete log you will ever have and also the one that gets deleted six weeks after the privacy review notices it, which leaves you with nothing. It also does something less obvious: it duplicates customer data into a system whose access controls are looser than those on the database it came from, so the log becomes an easier target than the thing it was watching.

What survives review is a representation that preserves the properties you need for an investigation without carrying the values themselves. Four techniques cover most of it.

Log the argument shape: which parameters were present, their types, string lengths, and a digest of the JSON schema version. That alone answers "was this call malformed" and "did the agent start passing a field it never used before".

Log keyed pseudonyms rather than values. A keyed hash — an HMAC, or hash-based message authentication code, which mixes a secret key into the hash so the output cannot be reproduced without it — computed over the value with a per-tenant key held in a key management service (KMS) that performs operations with the key without ever handing it out, gives you a token that is stable — so you can see that the same email was looked up 340 times in eleven minutes — while being useless to anyone who cannot invoke the key. This is materially different from a plain SHA-256, which for a low-entropy field like a date of birth or a postcode is trivially reversed by enumeration.

The word doing the work in that last sentence is low-entropy, and it is worth unpacking, because "we hashed it" is one of the most confidently held wrong beliefs in this area. A hash is one-way in the sense that you cannot invert the function. It is not one-way in the sense that you cannot guess the input, and guessing is trivial when the set of possible inputs is small: every date of birth in a plausible range is a few tens of thousands of values, and every postcode in a country is a list you can download. An attacker with the log hashes each candidate once and compares, which turns the whole field back into plaintext at the cost of a few seconds. A keyed hash breaks that because the candidate list is no longer enough — you also need the key, and the key never leaves the key management service, so the enumeration has to be run through an API that counts and logs every call. The property that changed is not the strength of the hash function. It is who is able to compute it, which is the same substitution the rest of this topic keeps making: replace a check anyone can perform with one that requires an authority the attacker does not hold.

Log the internal identifier and not the human one. customer_id: cus_4419 is a pointer into a system that already has access controls and its own audit trail; email: maria@… is the data. Investigations almost always work fine with the pointer, and where they do not, the pointer plus a documented, logged lookup is the correct escalation path.

Log the result's shape rather than the result: row counts, whether the set was empty, and the classification tags of the fields returned, which record that a field held personally identifiable information without recording what it held. "Returned 1 record containing fields tagged pii.contact and pii.financial" is enough to answer what a breach touched.

import hmac, hashlib, json

REDACT = {"email": "pseudonym", "dob": "pseudonym", "last_four": "drop",
          "customer_id": "keep", "limit": "keep"}

def safe_args(args: dict, tenant_key: bytes) -> dict:
    out, dropped = {}, []
    for name, value in args.items():
        rule = REDACT.get(name, "drop")          # unknown fields default to drop
        if rule == "keep":
            out[name] = value
        elif rule == "pseudonym":
            digest = hmac.new(tenant_key, str(value).encode(), hashlib.sha256)
            out[name] = f"pt_{digest.hexdigest()[:16]}"
        else:
            dropped.append(name)
    return {"present": sorted(args), "dropped": dropped, "values": out,
            "shape_digest": hashlib.sha256(
                json.dumps({k: type(v).__name__ for k, v in sorted(args.items())}).encode()
            ).hexdigest()[:12]}

The default in that function is drop, for the same deny-by-default reason chapter 2 gave: a new argument added by a developer next quarter must not start flowing into the log because nobody remembered to classify it.

The honest cost is that redaction removes answers permanently. If you pseudonymise the search string, you will not be able to tell six months later whether an agent was searching for a specific named individual, and there are investigations where that is the question. The usual resolution is tiering rather than a single choice: full-fidelity arguments in a restricted store with a 7-day retention and access gated behind a break-glass approval — an emergency path an engineer has to claim explicitly, outside the normal permissions — that is itself logged, and the redacted record in the general store for 400 days. That way the first hours of an incident have the detail, and the long tail has the correlation.

Correlation, indexing and sampling#

A complete log that takes a week to query is a log you will read after the incident is over, and most of the difference is decided before anything goes wrong.

Correlation has to be established at write time, not reconstructed later. That means the run_id is generated at the entry point and threaded through the model client, the tool wrapper, the approval service and the outbound HTTP headers, and that the dependency's own request identifier is captured back into the record. Without the last part, joining your log to the billing service's log is a manual exercise over timestamps, and timestamps across services with 200ms of clock skew do not join reliably when a run makes six calls in four seconds.

The store has to support the shape of the question. The first-hour questions are narrow and known in advance: everything for one run_id, everything for one tenant_id in a window, every outcome != "ok" for one tool, every attempt with approval.approved_by set. Those want indexed structured fields, which means the log should be a table or a structured event stream rather than lines of text in a general log aggregator where args_ref has been flattened into a string. Teams that keep this in the same place as their application logs usually discover during the incident that the retention on that index is 14 days and the session in question was 15 days ago.

Sampling must never apply to these events. Distributed tracing at 1% is entirely reasonable for latency work and is a disaster for audit, because the one call that mattered is the one that was not sampled. Volume is not the problem it sounds like: an agent making six tool calls per task at 50,000 tasks a day produces 300,000 records, which at roughly 1.2KB each is about 360MB a day before compression.

Both of those multiplications are worth performing rather than accepting, because the volume argument is the one that gets sampling introduced and it does not survive contact with the numbers. Six calls in each of fifty thousand tasks is three hundred thousand records, and three hundred thousand records at a bit over a kilobyte apiece is a few hundred megabytes — a day's audit for an agent fleet fits inside what a single service's debug logging produces before lunch. So the volume is not the reason anyone samples these events. The reason is that they were emitted through a tracing pipeline that samples by default, and the sampling arrived with the pipeline rather than with a decision. That distinction matters when you argue the case, because "we cannot afford to keep it all" is answerable with arithmetic and "we inherited a sampler" is answerable by moving the sink.

Traces, integrity and the drill#

It is tempting to solve all of this by pointing at the tracing system, since spans — the timed records each service emits for its own part of one request — already carry timing, an identifier that threads through services, and attributes you can attach anything to. Emitting both from the same instrumentation point is sensible and saves duplicated plumbing, and treating them as the same system is not, because they disagree on every operational property that matters. Traces are sampled and these events cannot be; traces are retained for days and these need months; traces are readable by anyone with a dashboard and redacted arguments still need tighter access than that; and trace attribute schemas change whenever someone adds a field, whereas an audit record's schema is a contract that an investigation two years from now has to be able to parse. Version the record explicitly, keep a written meaning for each field, and treat removing one as a breaking change rather than a cleanup.

Integrity matters because the thing you are auditing may be the thing with the credentials. If the agent's runtime can write to the audit store with the same identity it uses for its work, then a sufficiently compromised agent can edit the record of what it did. The log should be append-only from the agent's perspective, written through a collector the agent cannot address directly, and ideally hash-chained so that a deletion is detectable rather than invisible:

def append(record: dict, prev_hash: str) -> dict:
    record["prev"] = prev_hash
    body = json.dumps(record, sort_keys=True, separators=(",", ":")).encode()
    record["hash"] = hashlib.sha256(prev_hash.encode() + body).hexdigest()
    return record          # store to a bucket with object lock / WORM retention

That chain is worth being precise about, because on its own it proves less than it appears to. Each record carries the hash of the one before it, so removing a record from the middle leaves the following record pointing at something that is no longer there, and altering one changes its hash and breaks every link after it. What that buys is not immutability. It is that tampering is no longer a local edit: to hide one record an attacker has to recompute every record written since, which they can do if they can write the whole store. The comment on the last line is therefore load-bearing rather than decorative. The chain converts "edit one row" into "rewrite the tail", and the object lock — or a head hash copied somewhere the agent's identity cannot reach, which serves the same purpose — is what makes rewriting the tail impossible. Stated as a condition on the claim: hash chaining makes deletion detectable only where something outside the store fixes a point the chain has to agree with.

Finally, someone has to have read it before the incident. A saved query, a runbook page with the four filters above written out, and a quarterly exercise where an engineer is asked to reconstruct a real session from the log and time themselves, is what converts a schema into a capability. The exercise reliably finds the missing field, and it finds it on a Tuesday rather than at 02:00.

One record per attempt, correlated at write time, never sampled, carrying the decisions as well as the actions: that is a modest amount of engineering, and it is the difference between an incident review that produces facts and one that produces a narrative assembled from what the model claimed it was doing.

What the log cannot do is tell you about the failure before it becomes an incident. Every field described here is written while something happens in production, and the interesting failures — an approval that goes stale between the yes and the write, a timeout that leaves a bulk operation half applied, an agent that finds a path around the confirmation entirely — are cheaper to discover in a test that forces them on purpose.