Deciding where conversation state lives

Two tickets can pose the same architectural question and deserve opposite answers. An internal assistant that answers questions about a company's own engineering documentation, wanted by Friday, is a straightforward yes to hosted state and hosted tools. Patient intake for a healthcare client — people upload a referral letter, a medication list, a photograph of a rash, and describe symptoms across several turns before a clinician sees anything — is a no, and the analysis that gets you there is most of the work.

What separates them is not sensitivity in the abstract, and not a company policy about vendors. It is a small number of properties that can be checked in an afternoon, plus one distinction that gets skipped: eligibility to process data under an agreement is a statement about permission, and the decision in front of you is about retention, residency, evidence and exit. A decision made this way has a cost attached — the seam that keeps the choice reversible is real work, and it can be priced — and it has an ending, which is not a verdict but a written record naming the conditions that would change it. Without that record the same argument is held again in a year by people who cannot tell which facts have moved.

Three properties that make the answer quick#

The internal assistant is a yes, and it is worth being precise about why, because "it is internal" is the wrong reason and will mislead you on the next ticket.

Handing conversation state to a provider is a transfer of custody rather than a change of storage medium, and that is what makes a short list of properties sufficient rather than arbitrary. Once somebody else holds the bytes, three things can go wrong and only three: a party outside your team can require an operation on the data that you are no longer positioned to perform, somebody can need the exact input a past run used and find that only the provider ever held it, and you can decide to take the data back and discover what that costs. Each of the three properties below asks whether one of those three is live for this workload. A ticket that answers no to all three is not being waved through because it is unimportant. It is being waved through because there is no remaining mechanism by which custody can hurt it.

The data has no obligation attached. It is a corpus of your own engineering documentation and the conversations are colleagues asking how the deploy pipeline works. No customer data enters it, no regulator has an interest in it, and nobody will ever exercise a right of erasure over a question about Kubernetes.

An obligation is easier to reason about by what it forces than by who imposes it. Every obligation that can attach to conversation data resolves into one of three operations — erase this and show that you did, produce this on demand for as long as the clock runs, or keep it inside these borders — and each of the three is an operation on bytes, performed by whoever is holding them. While you hold them, all three are ordinary work against a store you already run. Once the provider holds them, each becomes a call you make into someone else's system plus a receipt you write yourself, which is the fan-out that chapter 3 traced across conversations, files and vector stores. Where nobody can compel any of the three, none of that machinery has to exist at all, and the storage question collapses into a preference.

The data has no future value. Nobody will replay these conversations, bisect a regression against them, or produce them in a dispute. A transcript that nobody will ever need to reconstruct is a transcript whose replay fidelity is worth nothing, which removes at a stroke the loss that chapters 3 and 7 spent their time on.

That loss is worth restating mechanically, because unlike the other two it cannot be bought back once it has happened. Under hosted state the input the model actually processed was assembled inside the provider, out of a stored thread plus whatever hosted retrieval wrote into it, and no copy of that assembled input ever passed through your process. Reconstructing it therefore means asking the provider for it, which can only succeed inside their retention window and only to the extent that nothing on their side has been compacted since. A workload nobody will ever ask about is a workload where that failure is real and costless. It is still happening on every turn; it is simply happening somewhere it does not matter.

The exit stays cheap. Two hundred conversations a week, none of which matter after the day they happen, means that switching later costs a rewrite of a small integration and the loss of nothing. Compare that to the same decision at a million conversations of accumulated customer history, where the exit involves an export, a schema translation, and a conversation with legal about what "deleted" meant.

What makes an exit expensive is not the size of the data, and the belief that it is has misled more than one migration plan. A transcript is text and text is small, so the instinct is that moving it later is a script somebody writes in an afternoon. The expensive parts are the ones that are not text: attachments sitting as individual objects in a file store you do not operate, an index built over them by machinery that is not yours, and the identifiers in your own rows pointing at objects that will not exist after the move. Every one of those has to be rewritten or rebuilt, and then a reconciliation has to run across all of them to establish that nothing was dropped. At two hundred disposable conversations a week none of that exists yet, and that absence — not the volume of text — is the whole of the difference between the two tickets.

Those three properties are the test, and they are what "internal prototype where speed matters" is a proxy for. Ship it on hosted state, hosted file search over the docs, hosted code execution if it helps, and spend the saved days on something else. The one discipline the yes requires is an expiry, because prototypes acquire users and all three properties are statements about the present. Write the decision down with the condition that reverses it — first external user, first customer document uploaded, ten thousand stored conversations, whichever comes first — because the failure mode here is not choosing wrong today, it is nobody noticing when the facts changed.

Eligibility is the entry ticket#

The healthcare ticket arrives with the compliance question apparently pre-answered: the provider is eligible for regulated health data under a signed business associate agreement, the security team has the paperwork, and the storage and document-handling weeks are therefore free.

Eligibility means the arrangement is permissible. It is a statement about whether you are allowed to send the data, and it settles that question completely. It does not answer where the data rests, how long it is kept, whether you can prove it was deleted, or whether you can produce the record in five years. Those are four separate facts, each governed by a different clause or a different configuration, and an agreement that permits the processing is silent on all of them.

The reason this step gets skipped is not carelessness, and it is worth naming, because the same shape recurs on every vendor decision after this one. The agreement is the hardest artefact in the chain to obtain, it is the one a security review explicitly asks for, and it is the only document in the sequence that carries a signature. A signature reads as a decision having been made. What was decided, though, is whether the bytes may travel, and every question that follows in this chapter is about what can be done with them once they have arrived. Permission and custody are different axes, and a document drafted to grant permission has no reason to mention custody at all.

Two of those four questions have to be answered in writing before the decision can be made at all, and each has an answer that ends the discussion.

The first is whether you can get a retention and deletion configuration you can evidence, with the features you need surviving under it. This one most often resolves itself without judgement, because the two requirements collide. A short- or zero-retention arrangement — zero data retention, ZDR in most vendors' documentation, meaning the provider undertakes not to persist request or response content at all — is exactly what a health workload wants, and provider-hosted conversation state is a storage product: the conversation object is retained data, and a zero-retention configuration either excludes it or is not a zero-retention configuration. The same tension shows up around specific features, since providers document capabilities that are unavailable below a retention floor and reject requests from an organisation configured under it. If the answer comes back as "hosted conversation state is not available under the retention terms your client requires", the decision is made and no architectural argument is needed.

That collision is structural rather than a gap in one vendor's current product, and the difference decides whether waiting for a roadmap is sensible. A zero-retention configuration is a commitment not to persist request or response content. Provider-hosted conversation state is a feature whose entire function is to persist that content and read it back on the next turn, which is what the trace in chapter 2 showed happening, down to retrieved text being written permanently into the stored thread. One is the negation of the other applied to the same object, so no release can offer both for the same conversation. What can move is the scope of the promise rather than its direction: a contracted window short enough that the stored thread is genuinely transient is a different offer from zero retention, and it is the offer worth asking for by name.

The second is whether you can produce the record later, for as long as the obligation runs. Clinical and legal retention obligations for intake material are measured in years — commonly six to ten, depending on jurisdiction and whether a minor is involved. Provider retention windows are measured in days to weeks. Those two numbers do not overlap, which means that for any period beyond the provider's window the only copy that can exist is yours. The moment you accept that, you are writing transcripts and documents into your own store anyway, and the weeks of work you were told you could skip have quietly reappeared. What is left to save is the context-assembly code, which is days rather than weeks.

Putting the obligation and the window on one scale makes that non-overlap harder to wave through with a caveat. Six years is a little over two thousand days, and a retention window measured in weeks is a few tens of days, so the obligation outlives the window by something in the hundreds. For all but the opening sliver of the period during which somebody can ask you to produce the record, the provider is holding nothing whatsoever. The conclusion is not that the risk is high. It is that for almost the entire life of the obligation, the record exists only if you wrote it down yourself, which turns a storage preference into a requirement.

An answer that flips this to yes exists and is worth stating so the analysis does not look like a foregone conclusion. If the client's obligation is satisfied by a structured clinical summary rather than the raw conversation — often the case, since what goes into the medical record is the intake summary, not the chat log — and the provider will contract to a retention window short enough that the transcript is genuinely transient, then hosted state is holding data with a defined, short life while your store holds the durable artefact. That is a coherent design, and it is one nobody described in the original estimate.

Which of the two situations you are in — the default no, or the flip to yes just described — is not an engineering fact, and this is where the analysis is genuinely hard rather than merely long. Whether the durable clinical record is the intake summary or the whole conversation is a question for the client's clinicians and whoever owns their records, and no amount of reading vendor documentation settles it. Nothing in the architecture can be decided until that answer exists in writing, which is an uncomfortable position to defend in a planning meeting. The days it takes to obtain it are still the cheapest days in the project, because the alternative is discovering the answer after the storage weeks have already been cut from the estimate.

Failure recovery is the row that gets least attention and behaves worst. With application-owned state, a provider outage is a routing decision — the transcript is in your database, so the next turn can go to a second provider with a translated request and the conversation continues. With hosted state, an in-flight conversation is pinned to the provider that holds it; there is no failing over mid-thread, because the thread is not yours to move. For an intake flow a clinician is waiting on, that is the difference between degraded and stopped.

"Pinned to the provider" sounds like a policy somebody set, and it is worth following through as a fact about what you are holding. What you hold is an opaque handle, meaningful only inside the system that issued it. The turns behind that handle were assembled there, and anything hosted retrieval added to them was written there too. A second provider cannot be handed a conversation it did not assemble and cannot read one it never stored, so failing over means rebuilding the conversation out of bytes — and the bytes are precisely what the design chose not to keep. Failover under hosted state therefore does not degrade. It is unavailable, and it is unavailable for the same reason replay is.

Volume is the row that makes the switching cost stop being theoretical. Intake documents at roughly 1,800 patients a month, averaging four attachments of about 2 MB, accumulate around 170 GB a year in the provider's file store. Moving that later is a download of every object through an export path, a re-upload somewhere else, a re-index, and a reconciliation that proves nothing was dropped; at realistic throughput that is days of wall-clock time and a project with an owner, not a migration script. The equivalent move from an object store you already operate is a bucket replication. Nothing about the first version is impossible; it is simply a cost that grows every month the decision is not revisited, which is exactly the property that makes it easy to defer forever.

That volume figure is worth deriving rather than accepting, because the half of it that governs the exit is not the half anybody quotes. One thousand eight hundred patients a month at four attachments each is seven thousand two hundred objects a month, and the multiplication is permitted because the attachment count is an average per patient over the same month the patient count is measured in. At about two megabytes apiece those objects come to a little over fourteen thousand megabytes, which is roughly fourteen gigabytes a month, and twelve such months is near enough the hundred and seventy gigabytes in the sentence above. The object count, not the total size, is what sets the price of leaving. A provider's file store is built to hand back one object at a time inside a turn, so an export is one request per object, with a round trip that does not amortise across the set. That per-object floor is exactly what the bucket replication above avoids, because a store you operate exposes its objects as a set that can be copied wholesale rather than one handle at a time.

Keeping the option you did not take#

Declining hosted state does not mean writing the same integration twice, and the way to keep both doors open costs about thirty lines. Put context assembly behind an interface, keep tool declarations in configuration rather than inline, and make the audit capture from chapter 7 identical on both paths so that a comparison between them is possible later.

class ContextStore(Protocol):
    def turn_payload(self, case, text: str) -> dict: ...
    def record(self, case, request: dict, response) -> None: ...

class OwnedStore:
    def turn_payload(self, case, text):
        msgs = Message.for_case(case).ordered()      # our table, our order
        return {"messages": [*msgs, {"role": "user", "content": text}]}

class HostedStore:
    def turn_payload(self, case, text):
        return {"conversation": case.provider_conversation_id,
                "input": [{"role": "user", "content": text}]}

A seam here means one place where two implementations of the same job are allowed to differ while everything around them stays the same, and the reason a seam is available at this particular point is a property of the request rather than a trick of design. Either way, a turn ends in exactly one request body sent to a model that holds nothing between calls. Everything upstream of that body is the same in both designs — the same user text, the same case it belongs to — and everything downstream is the same as well, since the response is read and captured identically. The only thing that differs is where the earlier turns come from, and that difference lives entirely inside the construction of the body. A difference confined to one function is a difference an interface can hold, which is why the seam costs thirty lines rather than a second integration.

The seam is worth having for three reasons that have nothing to do with indecision. It makes chapter 4's comparison a configuration flag rather than a branch, so the cost and latency question can be re-measured when volumes change. It makes chapter 7's hermetic evaluation suite trivial, because the owned implementation is what fixtures replay through. And it means the reversal condition attached to the internal assistant's yes is a switch someone can actually flip on the day the first customer document is uploaded, rather than a rewrite nobody has time for.

What the interface does not cover has to be said in the same breath, because a seam that is oversold becomes the reason nobody prices the exit again. Only the assembly of the request body sits behind it. Uploaded documents held in a provider's file store, the index built over them, and the identifiers threading your own rows to those objects all sit outside it, and those are exactly the assets whose migration cost grows with volume. A reviewer told that the choice stays reversible should therefore ask to see the export path for the files and the plan for rebuilding the index, rather than the two ContextStore implementations, because the switch is only cheap for the part that was cheap in the first place.

The two conditions under reverses_if in the memo above are what make the document worth writing. A decision recorded without them is a verdict, and verdicts rot: the provider ships the retention tier in June, nobody revisits it, and the team carries a cost for two years because the original reasoning was never written in a form that could be checked against new facts. The same applies in the other direction, which is the more common failure — a prototype's yes stays in force long after the first external user arrives, because nothing in the record said what would end it.

Deciding where conversation state lives is three questions and a memo. Is an obligation attached to this data, will anyone ever need to reproduce a past run, and what does the exit cost at twelve months of accumulated volume? A no to all three is a quick yes to hosted state with an expiry condition attached. A yes to any of them means the analysis is the work, and it starts by separating what a vendor agreement permits from what it guarantees, because those are different documents answering different questions.

That is the shape of the work now. The eleven-line pull request from chapter 1 turns out to contain two decisions with different failure modes, and both of them are answerable — not by a policy about vendors, but by a trace you captured, a deletion you evidenced, a measurement you ran under a stated method, a red-team payload you reproduced, and a memo that says what would change your mind. The next feature that arrives with a provider primitive attached is a chance to run that sequence for real, on a system you own, before the decision hardens into something nobody remembers making.