Comparison
Open-source vs proprietary LLMs: the cost math nobody shows
Self-hosting looks cheaper per token and usually isn't. Where the crossover actually is, and the three reasons that beat cost either way.
Open-source vs proprietary LLMs is usually argued on price, and the argument is usually wrong in both directions. Open weights are not free — you rent the GPU whether or not anyone sends a request. Proprietary APIs are not obviously expensive — you pay only for tokens actually used. The crossover exists, and it’s further out than most teams assume.
The arithmetic
An API bills per token. Ten thousand requests a day at a few thousand tokens each lands somewhere in the low hundreds of dollars a month for a mid-tier model — and if traffic halves, so does the bill.
Self-hosting bills per hour. A GPU capable of serving a competent open model runs roughly one to two dollars an hour on demand, which is $700–1,500 a month before anyone uses it. You need more than one for redundancy. Reserved capacity cuts the rate and removes the ability to scale down.
So the crossover isn’t a token price comparison; it’s utilisation. A GPU at 80% sustained load is excellent value. The same GPU at 5% — which is what bursty internal traffic looks like — costs many times the API equivalent for identical work.
The honest rule: self-hosting wins on cost when you have high, steady, predictable volume. It loses badly when volume is low or spiky, and most products start there.
The costs that don’t appear in the comparison
Per-token maths omits the parts that dominate for small teams.
Engineering time. Inference serving, batching, quantisation, autoscaling, upgrades. Someone owns this, and their time is the largest line item at any realistic scale.
Idle capacity. You provision for peak and pay for it at 3am.
Model upgrades. A provider improves models under the same endpoint. You do an evaluation, a migration and a rollout.
Reliability. Providers run multi-region infrastructure with published uptime. Matching that yourself means redundant GPUs in more than one place.
Against that, the API side has its own omitted costs: rate limits you’ll engineer around, pricing you don’t control, and deprecation timelines you don’t set.
Side by side
| Criterion | Open weights, self-hosted | Proprietary API |
|---|---|---|
| Cost shape | Fixed per hour | Variable per token |
| Cheapest at | High, steady volume | Low or spiky volume |
| Scale to zero | No | Yes |
| Data residency | Fully yours | Provider’s terms |
| Frontier capability | Trails | Leads |
| Model stability | You decide | Provider deprecates |
| Fine-tuning | Full control | Where offered |
| Ops burden | Substantial | None |
| Latency floor | Your network | Provider’s, plus internet |
The three reasons that beat cost
Most teams that self-host successfully didn’t do it to save money.
Data can’t leave. Healthcare, defence, some financial and public-sector work. When the requirement is that no customer data crosses your boundary, the cost comparison is irrelevant — one option is disqualified.
The model must not change. A provider can deprecate a version and shift behaviour under you. If you’ve certified a system’s outputs, holding fixed weights is the only way to keep that certification meaningful.
Deep customisation. Full fine-tuning on domain data, or modifications no API exposes. If your edge comes from a model nobody else can call, you need the weights.
Every one of these is a hard requirement rather than a preference. That’s the tell: if your reason is “it should be cheaper”, re-run the utilisation maths first.
Capability is a moving target
Frontier proprietary models lead on the hardest reasoning, and open weights have closed the gap on the ordinary work faster than most expected. Both statements keep being true as the numbers move.
What matters for a product is that most tasks don’t need frontier capability. Classification, extraction, routing, summarisation and formatting run fine on smaller open models. Complex multi-step reasoning with tool use is where the frontier still earns its price.
Which points at the answer most mature systems reach.
The hybrid nearly everyone lands on
Route by task, not by ideology. Cheap high-volume work goes to a small model — self-hosted if volume justifies it, a cheap API tier if not. Hard reasoning goes to a frontier model. Anything touching data that can’t leave goes to a local model regardless of cost.
This requires a routing layer that abstracts the provider, which is worth building early for a different reason: it’s also your insurance against a price change or a deprecation. Teams with one hardcoded provider discover the cost of that coupling at the worst moment.
How to choose
Self-host when data residency is a hard requirement, you need frozen weights, you need deep customisation, or you have sustained high volume with a team to run it.
Use an API when volume is low or unpredictable, you want frontier capability, or your engineers should be building your product rather than an inference platform. This is the correct default, and staying there is not a failure to mature.
Do both when your workload splits cleanly — and it usually does, because the volume is in the easy tasks.
Before deciding on cost: estimate your actual GPU utilisation. If it’s under 40%, the API is cheaper, and it isn’t close.
See also: where LLM spend actually goes and when fine-tuning is the right lever.