Message edits and tool effects

Edit-and-resend lets a user rewrite something they already said and get a different continuation from that point. It is a small feature and it is the one that breaks the most assumptions in a conversation store, because unlike regenerate it invalidates more than one message: everything downstream of the edit point was a response to text that no longer exists on the live path.

Two problems come out of that, and they need different machinery. The first is storage — a flat ordered list has no way to keep the displaced turns as a coherent alternative, so the write path deletes them, softly and with good intentions. The second is that some of those displaced turns did things. A tool call that looked up a delivery date can be abandoned with no consequence; a tool call that captured a payment cannot, and no amount of correctness in the message tree reaches the payment processor.

Tombstones and subtrees#

Why the displaced turns cannot simply stay where they are is worth one step, because it decides everything the write path does next. An assistant turn is produced from a request body containing every earlier message of the conversation verbatim. Change one of those earlier messages and the reply below it was produced from a body that no longer matches what is stored. The reply is still a real thing that was really said, and it has stopped being an answer to the conversation now on the path. Leaving it in place would render a thread in which the assistant responds to a question nobody asked.

A flat list has no way to say that quietly. A row's membership in the conversation is the whole of its relationship to it — the read path returns the rows for a conversation_id in seq order, so being in that result is what makes a row part of the thread, and there is no third state between present and gone. Taking a turn out of the live reading and taking it out altogether are therefore the same operation, and that collapse is what makes the statement below look reasonable.

On a (conversation_id, seq) list, the edited message needs position $2 + 1 and the old occupant has to get out of the way, which makes a soft delete — stamping a row with a deleted_at timestamp rather than issuing a DELETE, so the bytes remain and every reader agrees to skip them — the obvious write path:

UPDATE messages
SET    deleted_at = now()
WHERE  conversation_id = $1 AND seq > $2 AND deleted_at IS NULL;

INSERT INTO messages (conversation_id, seq, role, ...)
VALUES ($1, $2 + 1, 'user', ...);

Every read path filters on deleted_at IS NULL, so the conversation renders exactly as the product wants. It even feels reversible, since the rows are still there.

They are still there in the sense that the bytes exist. What is gone is their membership in anything. A tombstone — the marked-but-retained row — records that it was removed at a moment; it does not record what it was removed in favour of, which is the fact any later investigation turns on. That gap becomes concrete the second time the user edits: the second edit tombstones rows that were already tombstoned by the first, and now four rows carry deleted_at values fifteen seconds apart with no structure saying which edit produced which removal, which of the surviving rows belongs with which set, or whether a row deleted at 14:03 was part of the same alternative as one deleted at 14:03:02. Reconstructing the original conversation means sorting tombstones by timestamp and guessing, and the guess gets harder every time the user edits.

Soft deletion fails a replay for a reason worth stating directly, and it is the same gap seen from the other side. Replaying the original conversation needs the original path as a path — an identifiable, ordered chain that was the live conversation at a stated time — and a deleted_at column produces a set of rows with no chain and no statement about when they were live together.

The reason no amount of care with the column fixes this is that the missing facts are relations between rows and a column holds one fact about one row. Try to patch it and watch where the patches lead. Add a deleted_batch_id so rows removed by the same edit share a value, and the set problem goes away while the question of what replaced them remains open. Add a replaced_by column naming the message that took their place, and the tombstones now form a graph — at which point you are storing the edge that a parent_id stores, on the wrong rows, in a column whose meaning depends on a filter every reader has to remember. Each patch is individually reasonable and the destination is the model in the next section, arrived at by a route that leaves a deleted_at column behind to be misread later.

Edit is a branch on a user message#

A parent-linked message tree already has the write path for this; edit-and-resend is regenerate applied to a user message rather than an assistant one. The edited message is inserted as a sibling of the original, sharing its parent, with branch_reason = 'user_edit', and the head pointer moves to it. The rows after the edit point are not touched by any statement, because they are still children of the original message and the original message is still a child of its parent. They simply stop being on the path from the head.

Nothing in that write path is new; it is the same three statements a regenerate performs, with branch_reason set differently, which is the sign that the model was the right one. The interesting difference is scale. Regenerating orphans a single message, while editing three turns back orphans an entire subtree — in the diagram, four messages and the two tool calls inside them — and a subtree is precisely what a tombstone cannot represent, because it is a shape rather than a set.

That nothing has to be deleted is a structural property rather than a convenience, and it is the property the whole arrangement rests on. Every edge in this model points from a child to its parent, so no row's meaning depends on what points at it. A1 says it answers U1 and goes on saying so whatever else is inserted, and inserting U1b writes a new row and changes no field of any existing one. There is no operation called displacement in the model at all. What changes is which leaf the head names, and the subtree under U1 becomes unreachable from the head without being altered, filtered or flagged.

That has an honest cost, and it shows up later in this chapter rather than here. No column anywhere says the old subtree is dead, so any job that needs to find abandoned work — including the one that hunts for charges nobody is looking at — has to compute reachability from the current head instead of reading a flag. The traversal is the price of never writing down a fact that the next branch switch would falsify.

Both alternatives are now readable, which makes the product question answerable: the UI can offer "you previously asked …" with the original subtree behind it, and an analysis can compare the outcome of the original path against the edited one, controlled by construction because everything above the fork is identical.

One thing the tree does not give you on its own is when each path was live. The head pointer holds only the current selection, so a tree with three leaves does not say which leaf was selected at 14:03 on 9 March. An append-only log of head movements closes it:

CREATE TABLE conversation_head_log (
  conversation_id uuid NOT NULL REFERENCES conversations(id),
  head_message_id uuid NOT NULL REFERENCES messages(id),
  moved_at        timestamptz NOT NULL DEFAULT now(),
  moved_by        uuid,
  reason          text NOT NULL,
  PRIMARY KEY (conversation_id, moved_at)
);

Each row is one line, written in the same transaction as the head update, and the whole history of what the user was looking at falls out of an ordered scan. This is the table that turns "which conversation did the customer see" from an inference into a query.

The reason a log is required rather than a better pointer is that a column holds one value and an update destroys the one before it. A pointer can answer "what is selected" and can never answer "what was selected on 9 March", because the value that would have answered it was overwritten by the next click. Any question about the past needs a row per change, which is what an append-only log is. Writing that row inside the same transaction as the update is what makes the log complete rather than approximately complete: a separate statement afterwards leaves a window in which the head has moved and the log has not, and a log with gaps in it cannot be told apart from the log of a conversation that never branched.

An effect is not a message#

None of that reverses a payment, and being exact about why is what the rest of this chapter rests on. The tree is a record of what was said and in what order. A charge is a change in the world, made by a system that has never heard of your conversation schema and will not consult it. Moving a head pointer cannot reverse a card capture any more than deleting a sent email un-sends it — a comparison that holds on the one-way part and breaks immediately after, since a payment processor does expose an action that answers a capture and a mail server does not.

The chain is short and every link in it is checkable. The tool call left your process as a request to a service you do not own. That service recorded the outcome in its own store and told you about it. Your database has no write path into that store, so every operation available to you — a DELETE, a deleted_at, a head pointer moved anywhere at all — alters your record of the effect and reaches nothing else. The only remaining lever is a second request, which is a new event with a new identifier rather than an undoing of the first one, and a schema that wants to stay honest has to be able to name both.

Effects therefore belong in their own table, joined to the conversation through the identifier that already pairs a call with its result:

CREATE TABLE tool_effects (
  id              uuid PRIMARY KEY,
  tool_call_id    text NOT NULL UNIQUE
                    REFERENCES message_parts_tool_use(tool_call_id),
  conversation_id uuid NOT NULL,
  tool_name       text NOT NULL,
  idempotency_key text NOT NULL UNIQUE,
  state           text NOT NULL CHECK (state IN
                    ('pending','applied','failed','compensated')),
  external_ref    text,                 -- 'ch_3PkQ8sJ1n4Yh2P'
  amount_minor    bigint,
  currency        char(3),
  requested_at    timestamptz NOT NULL DEFAULT now(),
  settled_at      timestamptz,
  compensated_by  text                  -- 're_9dK2mQ...'
);

The foreign key target is the unique-call side of the pairing problem raised in the first chapter: the identifier appears on both the call and its result, so it cannot be unique across the parts table as a whole. Either a dedicated table holds the tool_use parts, or a view over the parts table is backed by a partial unique index on tool_call_id WHERE type = 'tool_use'. Either way it has to be a real constraint rather than a convention, because an effect row whose call cannot be located is precisely the artefact a finance team ends up staring at.

The tool_call_id foreign key is what makes the two records inseparable: every effect names the exact tool call in the exact message on the exact branch that caused it, so "which branch does this charge belong to" is a two-table join, and a finance investigation becomes a lookup on external_ref. What the ledger deliberately does not inherit from the conversation is its lifecycle. Its states move on evidence from the payment processor, not on anything the user does in the chat:

The compensated state names a second, opposite action recorded against the first — a refund, a cancellation, a reversing entry — because a settled capture cannot be withdrawn from the processor's history and can only be answered by another movement, which is why compensated_by holds the identifier of that movement rather than a boolean. The pending state is there because of ordering rather than tidiness. The row is written before the processor is called, in the same transaction that persists the tool_use part, so a crash between the call and the confirmation leaves a pending row naming an idempotency_key that a sweeper can retry or query against the processor. Write the ledger row after the call succeeds instead — which is the natural way to code it, since that is when you have the external_ref — and the crash window produces a charge with no local record at all. That window is a few hundred milliseconds wide per effectful call, which sounds negligible until a deploy restarts every pod during a busy hour and you are hand-matching processor exports against a database that never heard of them.

The ordering has a second half, and it fails in mirror image if it is got wrong. The ledger row must be committed before the request leaves, which also means the request must be issued after the commit rather than inside the transaction. A call made mid-transaction that is then rolled back produces a charge at the processor with no row anywhere, the same outcome as writing the row too late, reached from the opposite direction. With the row committed first, a crash leaves a pending row carrying the exact idempotency_key that was sent — the value the processor uses to recognise a request it has already handled — so a sweeper can re-send under that key and receive the original outcome rather than capturing a second time. The recovery path is safe to run precisely because it cannot cause the thing it is investigating.

The state most often misread is compensated, because it looks like a return to zero. A refund is a second movement recorded against the first, and both remain in the processor's history, which is why compensated_by holds the identifier of that second movement rather than a flag saying the first one is void. Anything that consumed the first movement before the second existed has already acted on it: a revenue figure may have been reported, an entitlement granted, a notification sent. Compensation is the best available answer to an effect that has happened, and it is a different thing from the effect not having happened. That gap is the reason the useful question below is how to find the affected charges rather than how to make them go away.

Abandoning a branch is not a transition on that diagram, and leaving it off is the design statement. A charge on an abandoned branch is still applied; what changes is that a reconciliation job can now find it, because "effects in state applied whose tool_call_id sits on a message that is not on any path to a live head" is an expressible query. Whether that condition should trigger an automatic refund is a business decision — sometimes yes, sometimes it should raise a ticket, sometimes the charge is correct because the customer really did buy the credits before changing the question — and the schema's job is to surface the set, not to decide.

Re-running an effectful call#

The second charge had a sibling problem of its own. When the user resends an edited message, the agent loop runs again on the new branch, and if the model decides to call purchase_credits a second time, the customer is charged twice for what they experienced as rewording a sentence.

An idempotency key is the standard tool: a value you attach to a request so the downstream service recognises a repeat of one it has already handled and returns the original outcome instead of doing the work twice, which is how a payment processor stops a retried capture from becoming a second charge. The mechanism is well understood; what branching makes hard is choosing what the key should be scoped to, and none of the candidates is obviously right. Key on tool_call_id and every branch is a fresh key, so every edit can re-charge. Key on (conversation_id, tool_name, arguments) and the new branch deduplicates against the old branch's charge, which is right when the edit was a typo fix and wrong when the customer genuinely wants a second purchase — and the schema cannot tell those apart, because the difference is in the customer's intent. Key on (branch_root, tool_name, arguments), taking the fork point as the scope, and you get a defensible middle: edits below a fork share a key, separate top-level requests do not. Every one of those is a product decision with real money on the wrong side of it, and the useful thing the schema contributes is that idempotency_key is a stored column, so whichever rule you choose is visible in the data and auditable afterwards instead of living in a helper function.

The belief worth dislodging is that having idempotency keys at all settles this. They were introduced to make retries safe and they do exactly that, which is what makes them feel like a general answer. A retry and a re-run on a new branch arrive at the processor as the same event: a second request for the same operation, from the same customer, for the same amount. On the customer's side they are opposites, since one is your infrastructure trying again after a timeout and the other is a person asking for something. Nothing distinguishing them ever crosses the wire, so no downstream service can make the distinction on your behalf. The scope you choose for the key is your product's statement about which of the two a re-run is, and storing the key in a column is what keeps that statement legible to whoever reads the ledger a year later.

There is a matching limit on how far branching should be allowed to go. Some tools should not be re-run on a branch at all, and marking them at the tool-definition level — an effectful boolean that makes the agent loop pause and ask rather than silently re-execute — is cheaper than any reconciliation you can build downstream. Retrieval, search and calculation are free to replay. Anything that moves money, sends a message to a third party, or mutates a record outside your service is not, and a schema that stores the branch faithfully while the loop happily re-charges the card has solved the audit problem and left the customer problem intact.

Both halves of that division want narrowing before anyone writes the boolean into a tool registry. Replayable means a re-run leaves nothing outside your service that somebody has to reconcile, and it does not mean free — a metered search endpoint is billed on every call, whether or not the branch it ran on survives. And effectfulness is not always a property of the tool: the same endpoint can be harmless with one set of arguments and expensive with another, so a flag at the tool-definition level is a deliberately conservative approximation. It will pause on calls that needed no pausing, which is the direction of error to prefer, and a product that finds itself pausing constantly has learned that the distinction belongs in the arguments rather than in the registry.

The division to carry away is that a conversation store is authoritative about what was said and about which alternative was live at a given moment, and it is authoritative about nothing else. Editing a message is a branch, because the displaced turns are a subtree and a subtree needs a shape to live in. Which branch was on screen at 14:03 is a log entry, because a single mutable pointer cannot answer a question about the past. And what a tool did to the world is a ledger row with its own states, its own external reference and its own reconciliation path, joined to the conversation by the tool call identifier and governed by the processor rather than by the user's last click.

Between the tree, the head log, the generation records and the effects ledger, a conversation now leaves marks in four places, and the abandoned branches keep their tool payloads and their attachments indefinitely because no user action ever removes them. That is fine until someone asks for their data to be deleted, at which point the question stops being what you have kept and becomes where all the copies are.