Open-weight models

Choosing the model a product runs on is a decision with four columns — what a candidate is permitted to do, what it costs, how well it does your task, and what operating it commits you to — and this topic fills them in that order. The first column can remove a candidate before any of the others are worth computing, and it turns on a distinction the industry's vocabulary routinely blurs.

A trained language model is, physically, a file: some tens or hundreds of billions of numbers called weights, which are what a training run produced and what a server loads into memory in order to answer a request. Whether anyone outside the organisation that trained those numbers is allowed to hold that file is the largest single division in the landscape of available models. Most of the models a working engineer has heard of are closed, meaning the weights stay on the provider's own infrastructure and you reach the model through a hosted API, sending prompts over the network and paying per token. An open-weight model is one whose publisher has put that file up for public download, so anyone can take a copy, run it on hardware they control, look inside it, and change it. The phrase is regularly read as one fact settling availability, licensing and deployment together, when it asserts only the first of those; everything else is decided by separate documents and separate engineering facts, each of which can fail independently. Calling a model open-weight tells you the parameters can be obtained, and what "open" is not promising is everything after that.

Why those questions come apart is worth stating as a chain, because it is the chain the rest of the chapter walks. A weight file is inert data. Nothing inside it checks who is reading it, no server is contacted when it is loaded, and copying it costs a disk write. No technical mechanism anywhere in the stack can therefore stop you doing anything at all with a file you already hold. What stands in the way instead is a document stating what you are permitted to do, and a document is not enforced by the software. It is enforced by a court, months later, on the basis of what it says rather than what your deployment does. That is why the first column of a model comparison is read rather than run, and why a candidate can pass every test you are able to execute and still be unusable.

Why a publisher gives the file away#

Weight releases exist because there are requirements a hosted API cannot satisfy at any price. Data that a customer's contract forbids from leaving their network cannot be sent to somebody else's endpoint; a model the provider may withdraw on ninety days' notice is a shaky foundation for a product sold with a five-year support commitment; a model you cannot modify cannot be fine-tuned on your own material and kept as an asset; and at steady, high volume, renting hardware by the hour can come out cheaper than paying by the token, which chapter three works out properly. Publishers give the file away for reasons of their own, taken up later in this chapter, and those reasons are commercial, so the releases arrive with conditions attached — which is what the rest of this chapter is concerned with.

How a publisher can afford to do that is the half of the arrangement that explains where the conditions land. The weights are the output of a training run, and the training run is the expensive part: the accelerator time, the data pipeline, the people who know how to make a run of that size converge at all. Publishing the file transfers the output and none of the capability that produced it, so the publisher still holds the only route to the next version. What it gains in exchange is distribution. Every team that builds on those weights writes tooling for that architecture, fine-tunes against that tokenizer, and reports results against that family, which makes the next release cheaper for everyone to adopt than a competitor's would be. What it stands to lose is a much larger company taking the same file and reselling it as a hosted service. The licence is the instrument that separates those two outcomes, which is why the conditions cluster where they do — on who may resell, above what size, and under whose name — rather than on anything technical.

In practice the term reaches you in four places. A model repository page — Hugging Face hosts most of them — carries the weight files, a licence tag and a model card describing the release. An inference marketplace lists the same open weights served by several independent hosts at several different prices, which chapter two unpicks. A vendor's slide compares their open-weight release against a closed frontier model on a public benchmark, which chapter four teaches you to read sceptically. And an enterprise customer's security review asks whether your system can run inside their own environment, a question only an open-weight deployment can answer yes to.

Four gates, and only the first one is the licence#

Four checks, decided by four different parties at four different moments, is the reason no single answer to "can we use this" exists. The grant is written by the publisher's lawyers on the day the model is released and applies to the file. The acceptable-use policy is a separate document, revised after release, and applies to whatever your product is doing this quarter. Training-data terms are usually not written at all, because nobody has committed to anything. And the serving question is settled by whichever inference engine and accelerator you can obtain, which no lawyer was consulted about. Passing any one of the four tells you nothing about the other three, so each has to be checked on its own, and the order below is cheapest first.

The grant is what the licence gives you: the right to use, modify and distribute the weights, usually subject to conditions. Genuinely permissive grants exist — Apache 2.0 and MIT are used by several major families and impose almost nothing beyond attribution and patent terms. Custom community licences are common too, and they carry conditions that a standard open-source licence never would: a user-count threshold above which you must negotiate separately, a requirement to display "Built with X" in your product, a rule that derivative models carry the original family name, or a prohibition on using the outputs to train a competing model.

The acceptable-use policy is a second document, frequently incorporated by reference from a single sentence in the licence, and it restricts what the model may be used for rather than what you may do with the file. Domain prohibitions in these policies cluster around exactly the areas that generate enterprise demand: medical and legal advice, credit and employment decisions, biometric identification, critical infrastructure. A licence that says "commercial use permitted" and an acceptable-use policy that says "not for automated legal decisions" are not in conflict; the second narrows the first, and both are binding.

Training-data terms are the third gate, and the one that most often has no document at all. Most open-weight releases describe their training corpus in a paragraph — "publicly available web data, code and licensed sources" — without a manifest, which means you cannot establish provenance for the material the model reproduces. That matters where a customer contract or a regulatory regime requires you to state where your system's knowledge comes from, and it matters commercially because closed API providers routinely offer copyright indemnification for outputs while open-weight publishers almost never do. Downloading weights does not transfer any warranty about what went into them.

The fourth gate is operational and has nothing to do with law. A set of weights is a serving problem before it is a product feature: the architecture must be supported by an inference engine you can actually run, at a numerical precision that fits the accelerators you can get, with a tokenizer and chat template your stack applies correctly, and at the context length you need rather than the one on the card. An inference engine is the server software that loads the weights and turns incoming requests into tokens — vLLM and llama.cpp are two of the common ones — and the chat template is the markup it wraps a conversation in so the model sees the format it was trained on.

The last row is the one that gets read backwards. A permissive licence on weights you have already downloaded is durable in a way an API contract is not, because nobody can reach into your object store and delete the file. That durability applies to the copy you hold and not to the relationship: the publisher decides whether there is a next version, whether it keeps the same licence, and whether the security fix you need is backported to the release you standardised on.

Three further beliefs in this area are tempting for good reasons and wrong in ways that cost money. The first is that a permissive licence tag settles the question, which is tempting because in ordinary software it very nearly does — an Apache 2.0 library does not come with a second document narrowing what you may build with it. Model releases broke that convention, and the scenario below is what that costs. The second is that because the weights run inside your own network, what the model is used for is nobody else's business. Where the file runs has no bearing on an acceptable-use policy, which binds the use rather than the deployment, and an air-gapped installation breaches a domain prohibition exactly as a hosted one does. The third is that the context length on a model card is a property of the file. It is a property of a configuration, and the configuration your serving stack reads may enable less than the card advertises, which is the reason the fourth gate is an experiment rather than a document.

Confirming the grant before anyone benchmarks anything#

The check is cheap and mechanical, which is the argument for doing it before a candidate enters a comparison rather than after it wins one. Repository metadata will tell you whether the weights are gated and what licence the publisher declared, and the declared tag is a starting point rather than an answer.

$ curl -s https://huggingface.co/api/models/some-org/some-model-34b \
    | jq '{gated, license: .cardData.license, name: .cardData.license_name,
           link: .cardData.license_link, extra: .cardData.extra_gated_prompt}'
{
  "gated": "manual",
  "license": "other",
  "name": "some-org-community-license-1.1",
  "link": "LICENSE.md",
  "extra": "You agree to the Acceptable Use Policy and will not redistribute the weights."
}

Three of those five fields are already decisions. "gated": "manual" means a human at the publisher approves each requester, so your CI cannot pull the weights with an anonymous token and your legal entity — not an engineer's personal account — should be the one that accepts the terms. "license": "other" means no standard licence applies and the text in LICENSE.md governs. The gate prompt names an acceptable-use policy and forbids redistribution, which rules out shipping the weights inside an on-premise appliance no matter what the rest of the licence says.

Then read the two files. The clauses worth extracting into a decision record are the ones that bind engineering choices later: any threshold tied to your user count or revenue, any naming or attribution obligation, any restriction on training other models from outputs, any prohibited domain that overlaps your product, and the governing law and termination terms. A record like this takes twenty minutes and survives the person who wrote it:

candidate: some-model-34b
licence: some-org-community-license-1.1 (custom, not OSI)
commercial_use: permitted below 100M monthly active users
redistribution: forbidden — weights may not leave our infrastructure
derivative_naming: model name must begin with "SomeModel-"
outputs_train_other_models: forbidden
aup_prohibited_domains: [legal advice, medical diagnosis, credit decisions]
overlap_with_our_product: legal advice — BLOCKING
indemnity: none offered
accepted_by: Acme Ltd (legal entity account), 2026-03-04

Whether the weights will run at all#

The operational gate fails more often than the legal one, and it fails later, because it only shows up when someone tries to serve the thing. The first question it asks is arithmetic, and the arithmetic has one input. A parameter is one number, and a numerical precision is how many bits are spent storing each of them; bfloat16, the format most models are trained and published in, spends sixteen bits, and sixteen bits is two bytes. The size of the file on disk is therefore the parameter count multiplied by two and nothing else, because there is one number per parameter and every one of them costs the same. So a 70-billion-parameter model at bfloat16 has weights of about 140 GB before any activation or cache memory: seventy billion numbers at two bytes each is a hundred and forty billion bytes, and a hundred and forty billion bytes is a hundred and forty gigabytes.

All of that has to sit in the accelerator's own memory rather than in the host's RAM, which is what turns a file size into a hardware constraint. Producing a single token requires reading every weight in the model, so weights parked across the bus would have to be fetched across it for each token, at a bandwidth an order of magnitude below the card's own. That does not fit on a single 80 GB accelerator, so the realistic options are two of them with tensor parallelism, or a 4-bit quantised build at roughly 40 GB that fits on one at some quality cost you now have to measure. Quantisation stores each weight at lower numerical precision than the sixteen bits it was trained at, which shrinks the file and speeds up generation at some cost in accuracy, and it recurs throughout this topic because it moves cost, latency and quality together. Four bits is half a byte, a quarter of what bfloat16 spent per parameter, so the same parameter count divides down to about a quarter of the original footprint. That lands a little under the forty on the card, because a build of this kind usually leaves some tensors — and the scaling factors recording how each block of weights was compressed — at their original precision.

The weights are the floor rather than the total. The KV cache adds to this per concurrent request and scales with context length. That cache holds the attention state a request has built up for the tokens it has already seen, which the model re-reads to produce each new token, and at 128k tokens of context it can run to tens of gigabytes on its own, which is how a deployment that passed a single-request smoke test falls over at eight concurrent users. Two properties of that cache do the damage jointly. It grows with the number of tokens in a request, because there is one entry per token seen. And it is held per request rather than shared, because the tokens it describes belong to that request alone. A smoke test sends one short prompt, allocates one small cache, and reports that the model fits. Eight users sending long ones allocate eight large caches out of whatever memory the weights left behind, and what the weights left behind does not grow to accommodate them.

Fitting is only the first question, because a configuration that loads is not automatically a configuration you can serve traffic on. Tensor parallelism is the arrangement that makes a model larger than one card servable at all: each layer's weight matrices are cut into pieces, one piece per accelerator, and every card computes a partial version of that layer's output from the piece it holds. Splitting a model across two accelerators that way means every token generated crosses the interconnect several times, so the same two cards give very different throughput in a machine with a fast direct link between them than in one where they talk over PCIe.

The reason that traffic is unavoidable follows from what a partial result is. No single card holds enough of the layer to produce a finished output, so the partials have to be summed across cards before the next layer can begin, and that sum is a message on the interconnect. There is one such exchange per split layer, the layers run strictly in sequence because each consumes the previous one's output, and the whole sequence runs again for every token emitted. The interconnect therefore sits on the critical path of every token rather than being touched once when the weights are loaded, which is why the link between the cards shows up in throughput and not merely in start-up time. Renting hosts with that interconnect, in the region where your data is allowed to live, with enough of them to survive a node failure, is a procurement question that arrives long before it arrives as a bill. A candidate that needs two 80 GB cards per replica and three replicas for redundancy is asking for six scarce accelerators before it has served a single customer request.

Architecture support is the other common wall, and the reason it is a wall rather than a delay is that an inference engine is not a general interpreter of weight files. A checkpoint is numbers with names attached; what turns those numbers into a reply is code that knows the exact sequence of operations this architecture performs, written and tuned for the accelerator. A published release that changes that sequence changes what the code has to do, and no amount of correct metadata in the file supplies the missing implementation. New releases frequently land before the open inference engines can serve them, and the gap is measured in weeks. The usual offenders are mixture-of-experts models, which hold many parameter blocks but activate only a few of them per token and so need routing logic the engine must implement, and models shipping a novel attention variant that no engine has kernels for yet.

$ vllm serve some-org/some-model-34b --tensor-parallel-size 2
ValueError: Model architectures ['SomeModelForCausalLM'] are not supported for now.
Supported architectures: ['AquilaForCausalLM', 'BaichuanForCausalLM', ...]

That message means the candidate is not deployable today regardless of its benchmark scores, and the honest entry in your comparison is "blocked on engine support, revisit when released" rather than a quality number. The same category of problem includes chat templates that the serving stack applies differently from the reference implementation, which silently degrades instruction-following, and advertised context lengths that the published configuration does not actually enable.

Open weights, open source and open data#

Open-weight is not open-source in the sense the Open Source Initiative defines, because that definition forbids restrictions on field of use, and most custom model licences contain exactly such restrictions. It is also not open-data: the training material, the filtering rules and the human-preference data used after pretraining are almost never released, so you cannot reproduce the model, audit what it memorised, or answer a customer asking whether their industry's copyrighted material was involved. Reproducibility and inspectability are separate properties from availability, and the word "open" gets used for all three.

The looseness of the vocabulary has a history worth knowing, because it explains why the labels are unreliable rather than merely imprecise. Before 2023, published research models were mostly small enough that licensing was an afterthought, and the ones that mattered commercially were reachable only through an API. Large-scale weight releases changed the stakes on both sides at once: publishers wanted the ecosystem, the papers and the downstream tooling that come from wide distribution, and also wanted a way to stop their largest competitors from serving the model as a product. Custom community licences are the compromise those two goals produced, and "open source" became the marketing term applied to the result because it was the phrase people already recognised. The Open Source Initiative eventually published a separate definition for open-source AI precisely because the existing one was being claimed by artifacts that did not meet it. Treat the label on a repository page as a hint about which document to read, never as the conclusion.

Open weights also do not, by themselves, remove a dependency. They remove one kind — the provider who can deprecate an endpoint on ninety days' notice — and replace it with a different one, since the next version, the safety fixes and the tooling ecosystem still come from the publisher. What you gain is a copy nobody can revoke and the option to run it where your data already lives; what you take on is everything the provider was doing on your behalf, which chapter three prices and chapter seven treats as an organisational commitment.

None of this is an argument against open-weight candidates. It is an argument about ordering. A licence review, an acceptable-use check against your actual product, a redistribution decision and a serving feasibility test cost about a day between legal and engineering, and any one of them can eliminate a candidate outright — which makes them worth spending before the evaluation rather than after it, when a fine-tune and three weeks of prompt work are already attached to the answer. The four gates are independent, they are checkable from public metadata and two documents, and none of them becomes easier to check later.

Running that day up front means the comparison table only contains models you are permitted and able to ship. The first column that table then asks you to fill in is what each candidate costs, a figure that turns out to be considerably less straightforward than the number printed on a pricing page.