The operational commitment of self-hosting

Consider a decision at the scale where it stops being reversible: the default model for a product shipping to 200 enterprise customers, embedded deeply enough in prompts, evaluation sets, output schemas and customer contracts that changing it is realistically an eighteen-month proposition. Somebody in the room says: self-host an open-weight model, because it is cheaper at scale and we avoid vendor lock-in. That sentence is two claims joined by "and", both plausible, and neither of them supported by anything said so far.

The first claim is a quantity, and chapter three showed how to compute it — self-hosting is cheaper above a break-even volume set by the hourly rate of hardware you can actually get, the throughput the model achieves at the batch size your latency target permits, and the fraction of the day the replica is busy. If that number has not been computed for this workload, "cheaper at scale" is an assertion about a threshold nobody has located, and the threshold may sit above the traffic this product will ever see. The second claim is a category error rather than an arithmetic gap, and it is what the rest of this chapter is about.

The error is worth naming precisely before it is argued, because "avoids vendor lock-in" sounds like a statement about who you depend on and is actually a statement about what it would cost to stop. Lock-in is not a count of dependencies. If it were, every architecture diagram with fewer boxes on it would be less locked in than one with more, which is not how anybody who has tried to leave one experiences it. Lock-in is the cost of exit, and exit cost is what has to be compared. A hosted API is one dependency and a self-hosted platform is a different one, so the sentence "we avoid vendor lock-in" is true about the vendor and says nothing at all until somebody prices the replacement.

What the eighteen months actually obliges you to#

Running your own inference means owning, permanently and at production quality, a set of responsibilities a hosted provider currently performs invisibly. Accelerator capacity has to be procured, which for reserved instances means a one-year commitment negotiated before you know your traffic, and for on-demand means competing for a supply that is periodically unavailable in the region you need. Serving expertise has to exist in-house: batch scheduling, key-value cache management, quantisation choices and their quality consequences, tensor parallelism topology, and the ability to diagnose why throughput dropped 30% after an engine upgrade. Engine upgrades themselves arrive every few weeks, carry real performance gains and occasional regressions, and are your responsibility to qualify.

The upgrade cycle for the model is the obligation that surprises teams most. A new open-weight release worth adopting appears every few months, and adopting it means re-running the frozen evaluation, re-tuning prompts fitted to the old model's behaviour, re-measuring throughput at your latency target, re-sizing the fleet because the new architecture has different memory characteristics, and rolling it out across an estate serving 200 customers who each have an opinion about change. A hosted provider does most of that work behind a version string. Doing it yourself is a recurring project with an owner, not a task.

On-call is the last piece and the one that changes the shape of a team. A GPU node that fails at three in the morning does not fail over the way a stateless web replica does, because the replacement has to pull 40 to 140 GB of weights and load them into device memory before it can serve, which is minutes rather than seconds.

Both halves of that comparison follow from what each replica has to have in place before it can answer. A stateless web replica starts from a container image measured in hundreds of megabytes, and once the process is up there is nothing further to warm: the first request is served like every later one. A model replica starts from a weight file of the size chapter one computed, which has to cross the network from object storage and then cross again into the accelerator's memory, and both of those are bandwidth-limited transfers of tens to hundreds of gigabytes. Neither transfer can be skipped, because the model cannot produce a token without every weight resident. The recovery time is therefore set by a file size rather than by process start-up, and no amount of health-check tuning shortens it.

What that does to the architecture is the part with a bill attached. When replacement is fast, spare capacity can be conjured on demand and a failed instance costs a few seconds of elevated error rate. When replacement takes minutes, capacity has to already exist and already be warm at the moment of failure, which is the redundancy replica in the previous chapter's table of costs sitting outside the break-even. Someone has to be paid to be awake for that, and the rotation needs enough people in it to be sustainable, which for a 24/7 commitment is rarely fewer than four or five engineers with real familiarity with the stack.

The last two rows carry the argument. Exiting a hosted API means porting prompts, re-running the frozen evaluation set of chapter five against a replacement, and migrating traffic — expensive, bounded, and something a small team can do in a quarter. Exiting a self-hosted platform means telling the people you hired to run it that the capability is being wound down, breaking or absorbing GPU reservations with months left on them, and rebuilding the operational knowledge if you ever want it back. The API is a contract you can leave; the platform is an organisation you have built.

Why one of those is harder than the other comes down to what each obligation is made of. A contract's obligations are written down and end on a date the contract names, so exiting is a project with a defined scope: port the prompts, re-run the frozen set, cut the traffic over, stop paying on the renewal boundary. A capability's obligations are people and commitments, and neither has a termination clause you can invoke. The reservations run to their term whether or not the workload does. The knowledge that made the platform work — which batch size the latency target permits, why throughput dropped after the last engine upgrade, what the recovery runbook actually does — lives in the heads of the engineers who built it, and it does not survive their reassignment even when they do. That is why the exit costs in the table are measured in different units: weeks for the contract, and quarters for the organisation.

What 200 customers change legally#

At this scale the gates of chapter one get re-examined with different stakes, because obligations now flow outward to customers rather than only inward to your own compliance. Adding a model provider makes them a subprocessor, a third party that handles your customers' data on your behalf, which under most enterprise data-processing agreements requires notification and gives some customers a window to object — a real workflow across 200 accounts, and a recurring one if you switch providers later.

The step people miss in that chain is which data is involved. It is natural to picture "our customers' data" as the rows in your database and to conclude that a model call, which sends a question and gets a paragraph back, is not a transfer of it. The prompt is the transfer. Whatever the retrieval step pasted in, whatever a tool call returned, whatever of the conversation history survived truncation — all of it is assembled into the request body and sent to the provider, which is exactly the processing the agreement is about. That is why the trigger is adding the provider rather than adding a feature, and why the notification obligation does not depend on how small the feature seems. Data residency commitments have to be met by the provider's actual processing regions, not their headquarters. Copyright indemnification for model outputs is offered by several closed providers on paid tiers and by essentially no open-weight publisher, which means self-hosting moves that liability onto you at exactly the moment you start signing enterprise contracts that ask about it.

Pulling in the other direction, some enterprise segments require deployment inside their own environment, and that requirement cannot be met by any hosted API at all. If a meaningful share of your revenue is regulated or air-gapped, an open-weight model is not the cheaper option, it is the only option — and the licence clause that governs it is redistribution, since shipping weights into a customer's infrastructure is distribution regardless of how the deployment is packaged. What makes packaging irrelevant is that distribution is a fact about where the file ends up rather than about the mechanism that put it there. A container image with the weights baked in, a virtual appliance, a chart the customer applies in their own cluster, and a signed download all leave a copy of the file on hardware you do not control and cannot recall. Each of those feels like a deployment decision and each is the same legal event. A custom community licence forbidding redistribution disqualifies a model for that segment however well it scores. Certification scope matters similarly: a provider's SOC 2 report, the independent audit of security controls that enterprise buyers ask for by name, covers their service, and self-hosting moves the inference stack inside your own audit boundary, which is more work but also removes an external dependency from the report you hand to customers.

A default and the triggers that reverse it#

Answering with a default and a set of triggers is what a defensible position looks like here, because it does not pick a side on principle. It states a default, gives the evidence for it, and names in advance the observations that would reverse it — the same pre-commitment discipline as the evaluation specification, applied to a decision that will be revisited rather than run once.

The dual-run state is what makes the transition survivable, because it is the only way to learn the operational cost before depending on it. Running the open-weight candidate on a slice of real traffic for a quarter — shadow mode first, where the candidate receives copies of live requests and its answers are logged rather than served to anyone, then a percentage of real requests — produces the utilisation figure the break-even needs, exposes the failure modes the previous chapter looked for, and tells you honestly how many engineer-hours a month the stack consumes. Skipping it means committing to a cost estimate built entirely from list prices and optimism.

The utilisation figure is the one of those three that can only be obtained this way, and the reason is worth following. Utilisation is the fraction of the hours you pay for that have work in them, so it is a property of how your requests arrive in time rather than of the hardware or the model. No specification sheet contains it, no benchmark produces it, and an average request rate does not imply it, because the same daily total spread evenly and the same total arriving inside a working day give the same average and completely different occupancies. The only source of a real arrival pattern is real traffic. Shadow mode supplies exactly that and nothing else: the candidate receives copies of live requests, so it experiences the true timing, while its answers go to a log rather than to a customer, so nothing depends on them being right. That is what makes it possible to measure the term before committing to the estimate the term dominates.

The triggers themselves belong in the decision record, in the same repository as the evaluation specification, with numbers rather than adjectives:

decision: default model provider, product v3
default: hosted API (model-a), behind an internal task-level interface
authored: 2026-05-14
break_even_self_host: 5.5M output tokens/month at 80% utilisation  # ch.3 method
current_volume: 1.3M output tokens/month

flip_to_self_hosted_if:
  - sustained volume > 8M output tokens/month for 2 consecutive months
  - a customer segment worth > 15% of ARR requires in-VPC or air-gapped deployment
  - hosted price rises > 40%, or our tier's rate limit caps us below peak demand
  - an open-weight candidate lands within 0.02 macro-F1 on eval/ticket-routing
    AND we have >= 4 engineers able to hold a serving rotation

flip_back_to_hosted_if:
  - GPU cost per 1M output tokens exceeds hosted price for 2 quarters
  - serving stack consumes > 0.75 FTE/month sustained
  - we miss two consecutive model generations because re-qualification stalled

review: 2026-11-14, and on any trigger
second_source: model-c-open-weight kept qualified against the same frozen eval

The second-source line is the cheapest insurance available. Keeping one alternative continuously qualified against the frozen evaluation costs a few hours and a few dollars per quarter, and it converts a provider outage, a price change or a deprecation notice from a crisis into a switch you have already tested. It also keeps the internal interface honest, since an abstraction that has only ever had one implementation behind it is usually not an abstraction. The mechanism there is that an interface's assumptions are invisible until something violates them. A wrapper written against one provider will quietly encode that provider's parameter names, its error shapes, its tolerance for a particular prompt format and its behaviour when a field is omitted, and nothing in the code will look wrong, because the only implementation behind it agrees with every one of those assumptions. Standing up a second one is what makes them show up as compile errors and failing tests rather than as a discovery made during an outage. That interface should stay thin and task-shaped — one function per job the product does — rather than a general wrapper that flattens every provider to a lowest common denominator and forfeits the features you are paying for.

The risks the default accepts#

Defaulting to the hosted API means accepting a specific set of risks rather than avoiding risk, and naming them is part of defending the choice. A provider can deprecate the version you qualified on with a few months' notice, which forces a re-evaluation on their schedule. The behaviour of a model behind a stable version string can shift slightly with serving-side changes, and a prompt fitted tightly to one model's habits is sensitive to that in ways your monitoring may not catch unless you run the frozen evaluation on a schedule rather than only at decision time. An outage at the provider is an outage in your product, with no remediation available to you beyond a fallback you built earlier. Pricing can move, and rate limits are granted rather than owned. These are real, and the reason they do not overturn the default is that each has a bounded response — a second source kept qualified, a scheduled regression run, a degraded mode — whereas the operational obligations in the table above have no equivalent shortcut.

Three beliefs push a team past that default without anyone examining them, and each is tempting for a reason. The first is the one this chapter opened on: running the model yourself removes a dependency. It replaces one, and the replacement is the harder of the two to leave. The second is that an existing container platform means the serving problem is already solved, which is tempting because the deployment mechanics genuinely are familiar — and the mechanics were never the difficulty. What is new is the part that has no analogue in a stateless service: recovery bounded by a file size, capacity that has to be warm before it is needed, and a performance envelope that moves with kernel versions. The third is that the model upgrade cycle can be deferred. It can, and deferring it is a decision to fall behind on a schedule set by other people, which is the cost the flip-back triggers in the record above are written to catch.

Nothing here argues that self-hosting is the wrong answer. It is the right answer at high sustained volume, for regulated deployment shapes, where a fine-tuned model is a differentiator, and where the serving capability already exists because something else in the company needs it. Many mature systems end up in neither pure state: a small self-hosted model absorbs the high-volume, latency-sensitive, well-understood fraction of traffic — classification, reranking, extraction — while a hosted frontier model handles the long tail that actually needs the capability, with a router in between and a cost model that tracks both. That arrangement is more work than either pure option and it is often what the break-even arithmetic recommends once you separate the traffic instead of averaging it.

The commitment, then, is the thing to price rather than the licence or the hourly rate. An open-weight deployment is a capability with a payroll, an on-call rotation, a procurement relationship and an exit cost measured in quarters, and it is chosen well when someone has written down what would make it worth having and what would make it worth abandoning. A hosted API is a supplier, with the narrower risks a supplier carries and the shorter exit a contract allows.

What these seven chapters leave you with is a sequence you can run rather than a ranking to memorise: gate the candidates on licence, acceptable use and serving feasibility; price them with the split, the tier and the serving configuration made explicit; convert self-hosting into a break-even volume; read the model cards for what they omit; write the deciding test before running it; bake off for variance, concurrency, failure modes and terms; and record the default with the triggers that would reverse it. The specific models in your shortlist will be obsolete within a year, and every artifact that procedure produces — the frozen evaluation set, the cost model, the decision record — will still be there to point at the next three.