The Vibe Father
The single most useful thing to understand about inference economics is that self-hosting costs are almost entirely fixed and API costs are almost entirely variable. A lightly used self-hosted model is extremely expensive per token. A heavily used one is cheap. The API is the exact mirror image.
Everything else — GPU choice, quantisation, batching — is a refinement on that one structural fact. This walks through where the break-even sits in 2026, how to work out which side of it you are on, and the costs people leave out of the comparison.
Related the Anthropic price sheet and what running a frontier open-weight model actually costs.
The numbers you need
| Item | 2026 figure |
|---|---|
| H100 SXM5, on-demand | ~$2.54 per GPU-hour, about $60.85 per day |
| H100 rental range | ~$1.50 to $10 per GPU-hour by provider and commitment |
| H200 range | ~$1 to $14 per GPU-hour |
| Per-token floor, one H100 at full utilisation | ~$1.76 per million tokens |
| Well-utilised self-hosted TCO | ~$0.40 per million tokens |
Note the enormous spread on rental rates. The difference between $1.50 and $10 per GPU-hour is not a rounding error — it is the difference between a viable business case and a disaster. Committed or reserved capacity is dramatically cheaper than on-demand, which means your break-even depends heavily on how confident you are in your own demand forecast.
Where the break-even sits
| You are replacing | Break-even, roughly |
|---|---|
| Premium API (frontier tier) | 5–10M tokens per month |
| Mid-tier API | Tens of millions per month |
| Budget API (small/cheap models) | 50–100M tokens per month |
These are order-of-magnitude figures, not quotes. The direction is what matters the cheaper the API you are replacing, the more volume you need before self-hosting makes sense. That is intuitive once stated and routinely ignored in practice, because self-hosting proposals tend to compare against the most expensive model in the stack rather than the one that would actually do the work.
The trap is comparing your self-hosted open model against frontier API pricing when the honest comparison is against a small, cheap hosted model of similar capability. Do that and the break-even often moves out of reach entirely.
The costs that get left out
Most self-hosting business cases compare GPU rental against API spend and stop. The gap between that and reality is where projects go wrong.
| Cost | Frequently omitted? | Notes |
|---|---|---|
| GPU rental or amortised purchase | No | The number everyone models |
| Idle time | Yes | You pay for 24 hours, you use a fraction |
| Redundancy | Yes | One node is a single point of failure |
| Engineering time | Yes | Serving stacks need real ownership |
| On-call | Yes | Inference becomes your outage |
| Model upgrades | Yes | Re-benchmark, re-deploy, re-tune every release |
| Storage and networking | Sometimes | Multi-terabyte weights move slowly |
Idle time is the biggest single distortion. Engineering workloads are bursty — heavy during working hours, near-zero overnight and at weekends. If your traffic is concentrated in roughly a third of the week, your effective per-token cost is roughly three times your utilisation-adjusted model. Anyone quoting a self-hosted cost per token without stating an assumed utilisation is quoting a best case that will not occur.
The utilisation curve, plainly
| Utilisation | Effective cost profile |
|---|---|
| Under 20% | Very expensive per token — API wins easily |
| 20–50% | Still usually worse than API |
| 50–80% | Competitive if volume is high |
| Over 80% | Cheapest per token — but no headroom for spikes |
The awkward part is that high utilisation and good latency are in tension. Running at 90% utilisation gives you the best per-token economics and the worst tail latency, because a queue forms the moment demand moves. Teams that optimise purely for cost tend to discover this in the form of complaints about the agent being slow.
The frontier model wrinkle
Everything above assumes a model you can serve on a manageable number of GPUs. At the frontier, the maths changes shape entirely.
Kimi K3, whose open weights publish on July 27, needs roughly 1.4TB of fast memory in four-bit precision — about eighteen H100-class GPUs for the weights alone, before any context. At on-demand rates that is over $30,000 a month for one always-on deployment, against hosted pricing of $3 in and $15 out per million tokens.
To break even at $30,000 a month against $15 output pricing you would need to be generating on the order of two billion output tokens monthly, continuously, with the deployment near full utilisation. Very few organisations are in that regime, and the ones that are already know it.
| Model class | Self-hosting realistic? |
|---|---|
| Small (7B–30B) | Yes — often on a single GPU |
| Mid (30B–120B) | Sometimes, with real volume |
| Large (200B+) | Rarely outside specialist infrastructure |
| Frontier open weights (1T+) | Essentially no, outside data-centre operators |
Quantisation changes the arithmetic
Precision is the lever that most directly moves hardware requirements, and it is the first thing a serious self-hosting plan tunes.
| Precision | Memory vs 16-bit | Typical trade |
|---|---|---|
| 16-bit | Baseline | Reference quality, maximum cost |
| 8-bit | ~Half | Usually negligible quality loss |
| 4-bit (MXFP4 and similar) | ~Quarter | Measurable but often acceptable |
Kimi K3 illustrates the scale of the effect, roughly 5.6TB at 16-bit against about 1.4TB in four-bit. That is the difference between roughly seventy H100-class GPUs and roughly eighteen — a four-fold change in capital before any other decision.
The caution is that quantisation quality loss is task-dependent and does not show up uniformly. A four-bit model may match its 16-bit self on straightforward generation and degrade noticeably on long-context reasoning or precise numerical work. Benchmark the quantised build you intend to run, not the published figures for the full-precision original.
A worked comparison
Take a team running an agent that consumes 40M tokens a month, split roughly 30M input and 10M output.
| Option | Rough monthly cost | Notes |
|---|---|---|
| Frontier API at $5/$25 | ~$400 | Before caching or batch discounts |
| Same, with caching on stable context | Materially lower | Cached input drops up to 90% |
| Mid-tier API at $2/$10 | ~$160 | If capability suffices |
| Self-hosted, one H100 at 30% utilisation | ~$1,800 | Rental alone, before people |
At 40M tokens a month, self-hosting is several times more expensive than the frontier API and an order of magnitude worse than a mid-tier one — and that is before engineering time, redundancy or on-call. This is the situation most engineering teams are actually in, which is why most self-hosting proposals should not survive contact with a spreadsheet.
The arithmetic flips when volume rises by one to two orders of magnitude and utilisation stays high. If you are not there, the honest answer is that you are buying something other than savings.
The reasons that are not about cost
Plenty of legitimate self-hosting decisions are not cost decisions at all, and it is worth being honest about which argument you are actually making.
- Data residency or air-gap requirements. If the data cannot leave, price is not the deciding variable.
- Regulatory constraints that no vendor arrangement satisfies.
- Latency floors that a network round trip cannot meet.
- Continuity guarantees — a model that cannot be deprecated or rate-limited under you.
- Fine-tuning on proprietary data you will not send anywhere.
Each of these can justify paying more per token. What causes trouble is presenting one of them as a cost saving, because then the project gets measured on a metric it was never going to hit.
The latency argument, briefly
Self-hosting is sometimes justified on latency, and the claim deserves scrutiny. Removing a network round trip to a provider saves tens of milliseconds. A reasoning model thinking before it answers costs seconds to tens of seconds.
If your latency problem is the network, self-hosting helps. If it is the model reasoning — which at high effort it almost always is — self-hosting changes nothing, and you will have bought infrastructure to solve a problem that lived somewhere else entirely. Measure where the time actually goes before you make it an infrastructure decision.
A decision procedure
- Measure your actual monthly tokens, split input and output. Guessing is where most of the error enters.
- Identify the model that would really do the work — not the most expensive one in your stack.
- Price that model's API cost at your real volume, after caching and batch discounts.
- Estimate honest utilisation for a self-hosted deployment. Be pessimistic, you will be right.
- Add the omitted costs — redundancy, engineering, on-call, upgrades.
- Compare. If the gap is under about 30%, stay on the API, the operational burden is not free.
- Re-check every six months. Both GPU rates and API prices are falling.
The hybrid that usually wins
For most engineering organisations the answer is neither pure option. It is a small self-hosted or local model for high-volume mechanical work, with API calls reserved for tasks that genuinely need frontier capability.
| Workload | Where it belongs |
|---|---|
| Classification, extraction, formatting | Small self-hosted or cheap API |
| Embeddings and bulk indexing | Self-hosted — steady, high volume |
| Routine features with tests | Mid-tier API |
| Hard debugging, long-horizon agents | Frontier API |
| Anything with residency constraints | Self-hosted regardless of cost |
| Spiky, unpredictable demand | API — you pay only for what you use |
| Steady overnight batch work | Self-hosted, or the Batch API at half price |
This gets you the fixed-cost advantage where volume is steady and predictable, and the variable-cost advantage where demand is spiky and capability requirements are high. It also means a provider outage degrades your throughput rather than stopping it.
Common questions
Is self-hosting an LLM cheaper than an API?
Only above a volume threshold that depends on which API you are replacing — roughly 5–10M tokens a month against premium APIs, 50–100M against budget ones, and only at high utilisation.
How much does an H100 cost per hour?
Roughly $2.54 on-demand for an H100 SXM5, with the broader market spanning about $1.50 to $10 per GPU-hour depending on provider and commitment.
What is the cheapest achievable cost per million tokens?
Around $0.40 for a well-utilised self-hosted deployment, against a floor near $1.76 for a single H100 at full utilisation on a larger model. Both assume utilisation most teams do not achieve.
Can I self-host a frontier open-weight model?
Realistically no. Kimi K3 needs about 1.4TB of fast memory in four-bit — roughly eighteen H100s before context, well over $30,000 a month.
What is the most common mistake in these comparisons?
Assuming full utilisation, and comparing an open model against frontier API pricing rather than against the cheaper hosted model that would actually do the job.
Does quantisation change the break-even?
Substantially. Four-bit precision needs roughly a quarter of the memory of 16-bit, which cuts GPU count and therefore fixed cost by a similar factor. Benchmark the quantised build you intend to serve, because quality loss is task-dependent and shows up unevenly — often on long-context reasoning first.
What utilisation should I assume?
Lower than you want to. Engineering workloads are bursty, concentrated in working hours, and near-zero at weekends. If traffic occupies about a third of the week, effective cost per token is roughly three times a naive full-utilisation estimate.
Is a hybrid setup worth the complexity?
Usually yes. A small self-hosted or cheap model for steady high-volume mechanical work, with frontier API calls reserved for hard tasks, captures the fixed-cost advantage where demand is predictable and the variable-cost advantage where it spikes. It also means a provider outage degrades throughput rather than stopping it.
Sources and further reading
- SitePoint, self-hosted LLM costs 2026
- Alpacked, self-hosted LLM costs, architecture and break-even
- TECHi. Kimi K3 inference economics
- Anthropic model pricing 2026
- Capability against price on the live board
A closing note on how these decisions actually get made. In practice the self-hosting question is rarely settled by a spreadsheet, it is settled by whoever is most confident in the room. The spreadsheet is still worth building, because it converts confidence into a number that can be checked in six months. Write down your assumed utilisation, your assumed volume and your assumed GPU rate, then revisit them once the deployment is real. The gap between what you assumed and what happened is the most valuable thing you will learn, and it will make the next infrastructure decision materially better. It is also the fastest way to find out whether your organisation is genuinely volume-constrained or simply attracted to the idea of owning its own inference — a distinction worth establishing before the hardware arrives rather than after.
