Reproducing a past model input

An assistant's behaviour on a given turn is a function of the array it was given, so the question "what did we actually send" is the first one asked in a dispute, an incident review, or an evaluation of a change. Teams with a messages table generally believe they can answer it: the turns are all there with timestamps, nothing was deleted, and the assembly function is fifteen lines. Load the rows, run assembly, print the array.

That first sentence is worth unpacking before the rest of the chapter leans on it. The endpoint keeps nothing between calls, which is the property this topic opened on, so everything that conditioned a reply — the order number, the tone the assistant had settled into, the figure it quoted — arrived inside that one request body. There was nowhere else for it to arrive from. A question about why the assistant said what it said therefore has exactly one place to be answered, and an array that is close to the one sent is not close to an answer: two arrays differing by a single omitted number are two different inputs, and the model saw one of them.

The belief that a messages table is enough rests on assembly being a pure function of that table — a computation whose output depends only on its stated inputs, so that the same rows always yield the same array. It stopped being one the moment any of the compression strategies of chapter 7 were adopted, and it was never quite one even before that.

Assembly is not a function of your messages table#

Whether a past array can be rebuilt is not obvious in either direction, and the conditions under which it could be are the thing to settle before looking at any particular input. Rebuilding means running assembly now and getting the bytes that went out then. That works when two properties hold together: the inputs must be unchanged since the call, and assembly must be deterministic, meaning the same inputs produce the same output on every run. Neither property is sufficient on its own. A deterministic function over inputs that have moved returns a different answer faithfully, and a function with a sampling step in it returns a different answer from inputs that never moved at all.

A messages table satisfies the first property better than almost anything else in the system, which is why the belief that it is the whole input is so durable. Rows are appended rather than rewritten, each carries a sequence number that fixes its position, and the read is an ordered scan with a fixed predicate. Nothing about that arrangement is wrong. The error is in how many inputs there are.

Six inputs feed that array and only one of them lives in the messages table. The assembly code revision changes whenever anyone tunes the verbatim window or moves a block within the array, and both are ordinary maintenance rather than rare events. The window and budget config changes with it. A summary produced by gpt-4o-mini at temperature 0.3 cannot be regenerated: re-running the same summarisation prompt over the same turns today yields different prose, and it would yield different prose even at temperature 0, because provider-side batching and hardware differences make bit-identical sampling something no hosted API promises. An extraction schema gains fields as the product learns what it needs. A retrieval index is rebuilt whenever the embedding model is upgraded, so the top-eight turns for a given query are not the top-eight turns they were. The system prompt lives in a config file with its own commit history.

The first two of those attack the second property from underneath, by making "assembly" ambiguous. Determinism is a property of a fixed function, and assembly is a function only relative to one revision of its source. Edit the source and there is a second function, equally deterministic and answering a different question, and the array it emits carries no marker saying which of the two produced it. The window and budget config breaks the same property from the other side. It is data rather than code, so it does not travel with the repository, and a record that pins the assembly revision but not the configuration has pinned half the function. "We can rebuild it, the code is in git" is the sentence to press on, and the question that presses on it is which configuration that code read on the day in question.

The summary is the input that no amount of discipline recovers, and the reason is worth taking slowly, because the obvious remedy sounds decisive and is not. A summary comes out of a model call, and a model call ends by picking one candidate fragment from a list of scores. Setting temperature to 0 makes that picking step deterministic: the highest-scoring candidate wins every time, with no random draw involved. What the argument leaves out is where the scores came from. They are the result of an enormous number of floating-point additions, and floating-point addition is not associative — adding the same quantities in a different order can change the last bit of the sum. A provider batches your request with whatever else arrived at the same moment, and the shape of that batch settles the order those additions happen in. Two calls with identical inputs can therefore produce scores whose last bits differ, and where two candidates were nearly tied, a last bit is enough to change which one leads.

One changed fragment does not stay one changed fragment. The loop feeds each chosen fragment back in as part of the input to the next step, so every fragment after the first divergence is scored against a different prefix than it would have been. A summary that differs in one word early can differ in which facts it carries by the end, and which facts it carries is precisely what a dispute turns on. Temperature 0 is a real setting that removes a real source of variation. It is not a promise of byte-identical output, and only the second claim would support a reconstruction.

The remaining three inputs fail the other property. Each is still deterministic, and none of them is still the same. An extraction schema that has gained fields extracts facts today for which March's call had no slot, so the pinned-facts block is wider than it was and carries material the model was never shown. A retrieval index rebuilt on a newer embedding model holds different vectors for the same turns. Nearest-neighbour order is a consequence of those vectors, so that reordering happens without the query or the turns changing at all. The system prompt is the one people assume is safe, because a config file under version control keeps every version it has ever had. Version control makes the text available rather than recording which version was deployed at a given minute, and those are different facts held in different systems.

Rows themselves also move. Any message soft-deleted since the call is excluded by today's query and was included in the original array; any message the user edited is now at a later revision. If the customer has exercised a deletion right in the intervening weeks, the rows are gone outright, and the array cannot be rebuilt even in principle.

Of those three, the soft-delete case deserves one more step, because there the query is not what changed. The read runs the same SQL today that it ran in March, including whatever predicate excludes deleted rows. What moved is the truth of that predicate for particular rows: a message visible then is marked deleted now, so the same text against the same table returns a smaller set. Nothing looks tampered with, and the array comes back short by exactly the messages somebody asked to have removed — which, in a dispute, are not a random sample of the conversation.

So the honest answer, with only a messages table, is that you can produce a plausible reconstruction and cannot produce the input. The gap between those two is not academic in a dispute where the question is whether the model was shown a £147.50 figure or a summary that omitted it. The reconstruction will contain today's summary of those turns, which may well include the number that the original summary dropped, and it would therefore support exactly the wrong conclusion.

What you can honestly hand over is worth stating so the answer is not simply "no". You have the turns themselves with timestamps, which establishes what the customer and the assistant said to each other, and that is often the substance of the dispute. You have whatever the assembly log from chapter 3 retained, which if sampling was in force is probably nothing for one specific conversation and if a trace flag was on is everything. You have the model's response, if responses are stored as assistant rows, which they are. What is unavailable is the mapping between them: which of those turns were in front of the model at the moment in question, and what compressed representation stood in for the rest. Saying that precisely, with the list of six inputs above, is a better answer than a reconstruction presented without caveats, because a reconstruction offered as the original is the kind of thing that gets discovered later by the other side.

Software has met this before, in build systems. A team that keeps its source and promises it can rebuild any release from it discovers, at the moment of an urgent rollback, that the rebuild pulls a patched transitive dependency and produces a different binary. The response was lockfiles and archived artefacts: pin the inputs, and keep the output, because "we can regenerate it" is a claim about a world in which nothing else moved. The outbound array is the artefact here.

That comparison is an analogy, and the place it breaks is the part of this problem with no engineering answer. A build with every input pinned is reproducible, because the compiler is a deterministic function: pinning the inputs is sufficient, and archiving the artefact is an optimisation on top. Here pinning is not sufficient. An array containing a generated summary cannot be regenerated from pinned inputs at all, for the sampling reason above, so archiving the output stops being an optimisation and becomes the only mechanism that works. The analogy is right about the diagnosis and understates the cure.

Persist what you sent#

The change is a record written per model call, containing the array and the parameters that produced it.

CREATE TABLE model_calls (
  id                 uuid PRIMARY KEY,
  conversation_id    uuid NOT NULL REFERENCES conversations (id),
  user_id            uuid NOT NULL,
  created_at         timestamptz NOT NULL DEFAULT now(),
  assembly_revision  text    NOT NULL,      -- git sha of the assembly module
  prompt_version     text    NOT NULL,
  model              text    NOT NULL,      -- pinned snapshot id, not an alias
  params             jsonb   NOT NULL,      -- temperature, max_tokens, tool names
  request_fingerprint char(64) NOT NULL,    -- sha256 of the canonical array
  request_uri        text,                  -- s3://…/2026/03/14/<id>.json.gz
  summary_id         bigint,                -- the exact summary row used
  retrieved_seqs     integer[],             -- what retrieval returned, if used
  input_tokens       integer,
  status             text NOT NULL,         -- 'sent' | 'ok' | 'error' | 'timeout'
  response_id        text,
  output_tokens      integer,
  latency_ms         integer
);

async def call_with_record(conn, conversation_id, user_id, messages, params):
    call_id = uuid4()
    body = json.dumps(messages, sort_keys=True, separators=(",", ":"))
    uri = await archive.put(f"{date.today():%Y/%m/%d}/{call_id}.json.gz", gzip.compress(body.encode()))

    await conn.execute(
        "INSERT INTO model_calls (id, conversation_id, user_id, assembly_revision,"
        " prompt_version, model, params, request_fingerprint, request_uri, input_tokens, status)"
        " VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,'sent')",
        call_id, conversation_id, user_id, ASSEMBLY_REVISION, PROMPT_VERSION,
        params["model"], json.dumps(params), sha256(body.encode()).hexdigest(),
        uri, count_tokens(messages),
    )
    try:
        resp = await client.chat.completions.create(messages=messages, **params)
    except Exception:
        await mark(conn, call_id, status="error")
        raise
    await mark(conn, call_id, status="ok", response_id=resp.id,
               output_tokens=resp.usage.completion_tokens)
    return resp

Writing the record before the call rather than after is what makes it useful during an incident. A call that times out at 30 seconds, or that the provider rejects, leaves a sent row with its full input attached, so the conversations that failed — which are disproportionately the ones people complain about — are exactly the ones you can examine. A record written only on success documents the calls that were fine.

The property doing that work is the commit, not the ordering of two lines of code. The insert is committed before the request leaves the process, so from that moment the input is durable independently of anything the call goes on to do, and the later mark only ever adds what happened next. A record written after the response arrives is written by code that runs only when a response arrives, which is why its coverage is exactly the set of calls that worked. That selection is not a sampling decision anybody made; it is a consequence of where the write sits. Rows that stay at sent carry their own signal, since each one is a call whose fate the handler never got to record, which is a different failure from a call that recorded an error.

Pinning the model to a snapshot identifier rather than a moving alias belongs in the same record for the same incident-time reason: gpt-4o-mini in March and gpt-4o-mini today may not be the same weights, and a model column containing an alias tells you nothing six weeks later.

An alias is a pointer the provider resolves at the moment of the call, and the column stores the pointer rather than what it resolved to. The same string in two rows six weeks apart can therefore denote two different sets of weights, and nothing in either row distinguishes the case where the model changed from the case where it did not. This is the second time in this chapter that a stable name has been read as a stable thing, after a path in version control read as a stable prompt. A snapshot identifier names one set of weights rather than whichever set is current, and that is the property which makes the column mean something later.

What the archive costs#

At 200,000 turns a month with an average serialised array of 12 KB, the raw volume is 2.4 GB a month, which gzips to roughly 600 MB. Object storage at $0.023 per GB-month makes the first month about 1.4 cents, and a year's accumulation of 7.2 GB compressed about 17 cents a month. Money is not the constraint. Keeping the same payloads as jsonb in the primary database is a different proposition — 29 GB a year in the transactional store, multiplied across a replica and daily backups, sitting on the same disk your chat queries depend on — which is the reason for the request_uri indirection rather than a column.

Every figure in that paragraph is one multiplication away from the one before it, and the conclusion they support — that money is not the constraint — is load-bearing enough to be worth checking rather than trusting. Two hundred thousand arrays at twelve kilobytes each come to two million four hundred thousand kilobytes, and a gigabyte is a million kilobytes, so that is 2.4 GB. The multiplication is allowed because the twelve is stated as an average: the sum of two hundred thousand individual sizes and two hundred thousand times their mean are the same quantity, by the definition of a mean. Gzip taking that to roughly 600 MB is a reduction to about a quarter, and that ratio is the one input to the calculation which is an assumption about the data rather than an arithmetic consequence — arrays of chat turns compress about that well because every object in them repeats the same handful of keys.

The bill then follows without any further assumption. Six hundred megabytes is six tenths of a gigabyte, and six tenths of $0.023 is 1.4 cents. A year's accumulation is twelve months of 600 MB, which is 7.2 GB, and 7.2 times $0.023 is about 17 cents a month — the standing cost of holding the whole year, rather than the cost of adding its twelfth month. The 29 GB is that same year computed without the compression step, since twelve months of 2.4 GB is a little under 29 GB, and uncompressed is the right basis there because a jsonb column is being compared against the disk a transactional store actually consumes.

The middle row is the interesting compromise and the one that gets overlooked. Storing the fingerprint, the assembly revision, the prompt version, the model snapshot, the summary row id and the retrieved sequence numbers costs a few hundred bytes and no message content at all — about 80 MB a month at this volume. It cannot produce the original array. What it can do is settle an argument about a reconstruction: rebuild the array using the recorded revision, prompt version and summary row, hash it, and compare. A match is proof; a mismatch tells you the reconstruction is not the thing, which is more than you knew before. For products where storing a second copy of conversation content is itself the problem — health, legal, anything under strict minimisation — this is often the right tier.

What makes that tier work at all is that its recorded fields turn the non-deterministic inputs into lookups. Checking out the recorded assembly_revision restores the function. Loading prompt_version restores the system message. Reading the row named by summary_id supplies the summary instead of generating one, and replaying retrieved_seqs supplies retrieval's answer instead of querying today's index. Each of those replaces a step that would otherwise produce today's answer with a stored value from that day, and once every such step has been replaced, assembly is deterministic again over inputs that have not moved — which is both of the properties named earlier, recovered by recording rather than by assumption. The comparison at the end holds for a separate reason: request_fingerprint is a digest of a canonically serialised array, keys sorted and no incidental whitespace, so two arrays agree on it only if they agree byte for byte.

That proof is conditional, and the conditions belong in writing before anyone relies on it. summary_id pins the summary only if summary rows are appended rather than updated in place, and the scenario above is exactly a case where the id would still have resolved while the text behind it would not have been March's. retrieved_seqs names turns rather than storing them, so it pins the selection and not the content. A message soft-deleted since the call is still excluded from the rebuild's read. The tier proves a reconstruction faithful when every referenced row is still what it was, and proves nothing when one of them is not — which is the correct behaviour, since that mismatch is itself the signal that something moved. The demonstration to ask for before accepting this tier as sufficient is a rebuild of a call from a month ago that reproduces its recorded fingerprint.

The last thing to check about that tier is that its two size figures agree. Four hundred bytes times two hundred thousand turns is eighty million bytes, the 80 MB a month quoted above, and eighty megabytes a month across two years is twenty-four such months — a little under 2 GB, which is the figure the retention paragraph below uses.

The real cost of the full archive is not storage but obligation. A second copy of message content needs the same access controls as the messages table, the same retention schedule, and the same deletion path, which means erasure requests have to reach the object store. Keying archived objects so that every object for a conversation and every object for a user can be enumerated without a full scan is a decision to make on day one; discovering later that you have 7 GB of gzipped conversations addressable only by call id, with a deletion request in hand, is an unpleasant week.

That enumeration difficulty is a property of how object stores are addressed rather than a policy failure. A store of this kind lists keys sharing a prefix cheaply and does not search their contents, so which questions you can answer later is fixed by the shape of the key you chose at the moment you wrote the object. The helper above builds its path from the date and the call id, which is the right shape for a lifecycle rule expiring things by age and the wrong shape for "every object belonging to this user". Answering that second question without a full scan means either putting the subject into the key or keeping the mapping in a table you can index, and model_calls already carries user_id alongside request_uri, so the second route is open here — provided the metadata row outlives the payload it points at.

Retention can be tiered along the same lines, because the two things the record is for have different lifespans. Full payloads earn their keep in the first thirty to ninety days, when they serve incident response, quality evaluation and support escalations, and they are the tier with the privacy weight. The four-hundred-byte metadata row is cheap enough to keep for as long as your dispute window runs — two years at this volume is under 2 GB in the primary database — and it is what lets you say, long after the payloads have expired, which assembly revision and which summary produced a given call. A lifecycle rule on the object store handles the expiry without anybody remembering to.

The archive pays for itself outside of legal requests, which is the argument that usually gets it prioritised. A stored set of real outbound arrays is an evaluation corpus that no synthetic fixture matches: you can replay last week's thousand hardest conversations against a candidate model and diff the responses before promoting it, and you can measure how often the assistant asked for information that was present in the array — the question chapter 3's product owner could not answer. It also shortens debugging from an argument to a lookup, since "what did we send" stops being a reconstruction exercise and becomes a SELECT on request_uri.

Two things this record does not give you are worth stating plainly. It is not reproducibility: sending the exact same array to the same model snapshot can return different text, so the archive is evidence of the input, and the output is evidenced only by having stored the response too. And it is not the same as the assembly log from chapter 3, which is sampled, short-retention and lives in a log pipeline whose access controls are usually looser; that log is for debugging this week's bug, while this record is for answering a question in six weeks. Conflating them tends to produce either a log pipeline full of customer data or an archive nobody can query quickly.

Of those two, the first repays one more step. The distinction underneath it is between evidence of an input and evidence of an output, and the two are established by different mechanisms. The input is under your control at the moment of the call, so it can be written down exactly. The output is produced by a step you do not run, so the only way to have it is to have kept the bytes that came back, and regenerating it is the thing the first half of this chapter says does not work. A stored response beside a reconstructed input is the worst available pairing, because it presents as a matched record of one call while only one half of it is.

The claim to give up is that stored messages are enough to reconstruct a past request, and the claim to replace it with is narrower and true: a request is reproducible only if something recorded it, or recorded enough of its inputs to prove a rebuild matches. Which of the two tiers you pick follows from what the answer is for. A fingerprint plus parameters settles arguments about faithfulness at almost no privacy cost, and a full archive answers the question directly at the cost of a second copy of customer content that now has to be governed like the first.

What all eight chapters have built is a set of things you can turn on. The messages table and the fresh assembly on every call, with the fingerprint that proves no process is holding state. The instrumented id list that turns "the model forgot" into a diff. A sequencing policy with a defined fate for the losing request and something for its tab to show. A headroom number logged per turn and a utilisation histogram on a dashboard. An overflow policy chosen against what it destroys rather than against what it saves. And a per-call record with a retention schedule someone has actually signed off. Each is small on its own; the reason to install them before you need them is that every one of them is cheapest on the day the conversation is created and most expensive on the day somebody asks what happened.