Self-Hosting LLMs vs Cloud APIs: Cost, Speed, Privacy 2026

The question used to be simple: can you even run a useful LLM locally? In 2026, the answer is definitively yes. Open-weight models like Llama 3.3, Qwen 3, DeepSeek R1, and Mistral Large rival proprietary models on many benchmarks. Consumer GPUs have enough VRAM to run 70B-parameter models. Tools like Ollama make local inference as easy as pulling a Docker image.

But "can" and "should" are different questions. Cloud APIs from OpenAI, Anthropic, and Google keep getting cheaper, faster, and more capable. The real decision in 2026 is not about possibility. It is about economics, performance requirements, and privacy constraints. (Those cloud options increasingly overlap: OpenAI's GPT-5.5 and GPT-5.4 now also run on Amazon Bedrock's OpenAI-compatible endpoint, and our Bedrock Responses API readiness check maps what changes.)

This guide works the actual numbers. Current vendor rates, what the hardware costs to buy and to run, and a payback calculation you can redo with your own figures. Where a number could not be traced to a primary source, we say so instead of printing it.

If you are building with AI coding tools specifically, our comparison of terminal AI coding agents covers which agents use local vs cloud inference under the hood. For a full pricing breakdown, see our AI coding tools pricing comparison.

How Cloud API Pricing Works in 2026#

Cloud LLM providers charge per token, typically quoted per million input tokens and per million output tokens. The spread between models is enormous, so "cloud API pricing" is not one number.

Current Pricing Snapshot#

Rates below are per 1 million tokens, read from each vendor's own pricing page on 2026-08-17: OpenAI API pricing and Anthropic's platform pricing docs. Note that anthropic.com/pricing now redirects to claude.com/pricing, and the consumer page there does not carry per-token rates. The docs page is the one to bookmark.

OpenAI, selected text models:

Anthropic:

Two things in those tables reset the whole self-hosting argument, and both arrived after the cheap-cloud/expensive-cloud split most comparisons still assume.

The first is the bottom of OpenAI's range. gpt-5.6-luna bills $0.20 per million input tokens and gpt-5-nano bills $0.05. Against the $0.40 that GPT-4.1 mini charges, the budget tier got two to eight times cheaper, and the budget tier is precisely what a self-hosted 8B model competes against. The second is Anthropic's note that Sonnet 5's introductory $2/$10 is now the standard price, and that the increase to $3/$15 scheduled for September 2026 will not happen.

Caching and batching cut further. Anthropic prices a cache hit at 0.1× base input and the Batch API at 50% off both directions, so the two stack to roughly 95% off for work that tolerates a queue. OpenAI's cached-input column above shows the same 10× ratio. We measured when the 24-hour prompt cache discount actually applies, because a cache you miss bills silently at full price, and a later run on GPT-5.6 found the 24-hour window replaced by 30 minutes.

The takeaway: under roughly 2 million tokens per day, cloud APIs beat any self-hosted setup on total cost. You pay nothing when idle, and idle is most of the time.

The True Cost of Self-Hosting#

Self-hosting sounds free after you buy the hardware. It is not. The real cost includes hardware amortization, electricity, cooling, maintenance time, and opportunity cost. Here is what the numbers actually look like.

Hardware Requirements by Model Size#

The table above assumes Q4 quantization. To estimate VRAM for a different precision or for fine-tuning with LoRA or QLoRA rather than inference, use our free LLM VRAM calculator. It shows the math and suggests a GPU that fits.

NVIDIA's own spec pages are worth reading before you budget, because the headline number people quote is VRAM and the number that bites is watts. The RTX 4090 carries 24 GB of GDDR6X at a 450W total graphics power. The RTX 5090 moves to 32 GB of GDDR7 on a 512-bit bus, which is the upgrade everyone talks about, and to 575W TGP with a 1,000W minimum system power requirement, which is the part that quietly reshapes your running cost. Token generation at this scale is bandwidth-bound rather than compute-bound, so the wider bus is where the speed comes from. At $2,000–$3,800 and patchy availability, buying one is still the hard part.

Running Costs#

Electricity gets underestimated more than any other line in this comparison, partly because the rate people use is out of date. The EIA's Electric Power Monthly puts the US residential average at 18.44 cents per kWh (May 2026, preliminary), with the all-sector average at 13.83 cents. A rig under your desk pays the residential rate.

Running a card at 350–450W around the clock costs 256–329 kWh a month. At 18.44 cents:

• RTX 4090, 24/7: ~$47–$61 per month
• RTX 4090, 8 hours/day: ~$16–$20 per month
• RTX 5090, 24/7 at its 575W ceiling: ~$77 per month

Add $5–$15/month for cooling in most setups. A dual-4090 rig serving 70B models around the clock runs $95–$122 in electricity alone, before the hardware.

What the Hardware Actually Costs per Month#

Assume a three-year amortization, 24/7 availability, and the EIA residential rate:

Those totals are arithmetic over three inputs, each of which you can check: a street price, NVIDIA's published TGP, and the EIA rate. We deliberately do not publish an effective cost-per-token for these rigs. Doing that requires a sustained throughput figure, throughput swings by an order of magnitude with batch size and serving engine, and every published number we found for it traced back to someone's blog rather than a reproducible harness. A cost-per-token built on an unverifiable denominator looks precise and misleads.

The hidden cost of idle hardware: a rig sized for 300 million tokens a day still bills you $104 a month when your workload is one million. The same workload costs a few dollars on a cloud API. Utilization, not the sticker price, is what decides this.

Self-Hosting Tools: Ollama vs vLLM vs llama.cpp#

Three tools dominate the self-hosting landscape in 2026. Each targets a different use case.

Ollama — The Developer's Default#

Ollama wraps llama.cpp in a Go-based server with a Docker-like experience. One command pulls and runs models with an OpenAI-compatible API endpoint.

Best for: Development, prototyping, personal use, privacy-focused workflows, air-gapped environments. For a complete setup walkthrough, see our Ollama + Open WebUI self-hosting guide.

Strengths:

• Zero-configuration setup. ollama run llama3.3 and you are running
• Automatic quantization and GPU detection
• OpenAI-compatible API (drop-in replacement in most SDKs)
• The 0.17 series (early 2026) added cloud model offloading, web search, multimodal support, streaming tool calls, and thinking models
• Native tool calling support for external API integration

Limitations:

• Caps at ~4 concurrent requests by default
• Single-stream throughput on an 8B model at Q4 lands in the low tens of tokens per second on consumer cards. Treat any specific figure you read, including ours, as unverified until you measure it on your own hardware, because quantization, context length, and driver version all move it
• Not designed for production-scale multi-user serving

vLLM — The Production Choice#

vLLM's PagedAttention engine manages GPU memory like an operating system manages RAM, paging model weights in and out to maximize throughput under concurrent load.

Best for: Production APIs, multi-user serving, high-throughput batch processing.

Strengths:

• Continuous batching interleaves token generation across concurrent requests instead of queueing them
• Red Hat benchmarked both engines on a single H200 with Llama 3.1 8B at FP16, from 1 to 64 concurrent users. At one user the two are comparable. At 64, vLLM "generated roughly 44 times more tokens per second than llama.cpp," and llama.cpp's P99 time-to-first-token passed three minutes while vLLM's stayed flat (Red Hat Developer)
• Speculative decoding support for faster generation
• OpenAI-compatible serving endpoint

Read that 44× carefully. It is a concurrency result on a data-center GPU, not a claim that vLLM is 44 times faster in general. At one user, the engine you pick barely matters.

Limitations:

• More complex setup than Ollama
• Higher minimum memory overhead
• Hardware support is broad but uneven. The vLLM docs list NVIDIA, AMD, and x86/ARM/PowerPC CPUs, plus plugins for TPU, Gaudi, and Apple Silicon. Older comparisons calling it NVIDIA-only are out of date, though the CPU and plugin paths are far less exercised than the CUDA one

llama.cpp — The Embedded Option#

The pure C/C++ inference engine with no external dependencies. It runs everywhere, from data center GPUs to Android phones.

Best for: Embedded applications, edge deployment, mobile devices, maximum hardware compatibility. For running models on dedicated NPUs and on-device accelerators, see our on-device AI inference guide.

Strengths:

• Runs on CPU, Apple Silicon, NVIDIA, AMD, and mobile devices
• Smallest footprint of the three
• 4-bit quantization enables Llama 3.2 3B on standard Android devices
• Direct library embedding into native applications

Limitations:

• No built-in serving layer (you add your own HTTP server)
• Lower throughput than vLLM under concurrent load
• More manual configuration required

Quick Decision Matrix#

For how these engines plug into the wider tooling ecosystem, see our tracking of the Model Context Protocol's growth, the standard most AI tools now use to reach external services. If you want a container-first local setup, compare Docker Model Runner vs Ollama. And if you end up running several backends at once, a gateway like LiteLLM is what sits in front of them.

Performance: Latency, Throughput, and Quality#

Latency#

Cloud APIs add network round-trip time, typically 50–200ms before the first token, depending on your location and the provider's infrastructure. Self-hosted models on local hardware start generating in 10–50ms.

For interactive applications where time-to-first-token matters (chat interfaces, code completion, real-time suggestions), local inference has a structural advantage. For batch processing or async workloads, the latency difference is irrelevant.

Throughput#

Single-user throughput is roughly comparable, and a quantized 8B model on a 4090-class card generally outruns what a cloud API streams back to you. We are not quoting a tokens-per-second figure for it. The numbers in circulation vary by several times over depending on quantization, context length, and serving engine, and we found no harness that reproduces any of them.

Multi-user throughput is where cloud APIs pull ahead. OpenAI and Anthropic run inference on massive GPU clusters with load balancing, auto-scaling, and request queuing. Replicating that with self-hosted infrastructure requires significant engineering investment in vLLM configuration, load balancing, and GPU fleet management.

Model Quality#

This is the part cost tables cannot answer. Strong open-weight releases (DeepSeek R1, Qwen 3 235B, Llama 3.1 405B) hold their own against mid-tier hosted models on published benchmarks. The frontier tier, Claude Opus 5 and GPT-5.6 Sol among them, still leads on hard reasoning, long-context work, and instruction following. Benchmark parity is also not the same as parity on your prompts, which is why the quality test belongs before the cost test rather than after it.

If your use case requires frontier-level intelligence, cloud APIs are your only option. If a 70B or 8B model handles your workload well, self-hosting becomes viable.

Privacy and Compliance#

This is often the strongest argument for self-hosting, and sometimes the only one that matters.

When Self-Hosting Is Required#

• Regulated industries: Healthcare (HIPAA), finance (SOX, PCI-DSS), and government (FedRAMP) may require that patient data, financial records, or classified information never leave your infrastructure.
• Data residency: Some jurisdictions require data processing to occur within national borders. Self-hosting on local infrastructure guarantees compliance.
• Air-gapped environments: Military, critical infrastructure, and some enterprise environments operate without internet access. Cloud APIs are not an option.
• Competitive sensitivity: If your prompts contain proprietary algorithms, trade secrets, or competitive intelligence, sending them to a third-party API introduces risk, even with data processing agreements in place.

Cloud API Privacy Guarantees#

Both OpenAI and Anthropic now offer enterprise tiers with zero-data-retention policies, SOC 2 Type II compliance, and contractual guarantees that API inputs are not used for training. For many organizations, these guarantees are sufficient.

The risk calculus is: do you trust a contractual guarantee, or do you need physical control? The answer depends on your threat model, not on technology.

Where inference runs is one half of the question. What the model is allowed to touch is the other, and our guide to running agent code in an isolated sandbox covers that side regardless of which backend generated the code. If your concern is spend and policy across several providers at once, see the OpenRouter guardrails and spend-governance run.

The Hybrid Approach: Best of Both Worlds#

The most cost-effective architecture in 2026 for many teams is hybrid: self-host for predictable baseline load, route to cloud APIs for overflow and frontier model access.

How It Works#

1. Baseline traffic (predictable, high-volume, latency-sensitive) routes to self-hosted models via Ollama or vLLM.
2. Overflow traffic (demand spikes beyond local GPU capacity) routes to cloud APIs automatically.
3. Frontier model requests (tasks requiring GPT-5 or Claude Opus-level reasoning) always route to cloud APIs.
4. Privacy-sensitive requests (containing PII, regulated data, or trade secrets) always route to self-hosted models.

Implementation Pattern#

An OpenAI-compatible router sits in front of both your local vLLM instance and cloud API endpoints. Since Ollama, vLLM, and most cloud providers all expose OpenAI-compatible APIs, your application code does not need to change. The router handles model selection based on request metadata, load, and policy rules.

The savings come from keeping expensive GPUs busy. Route steady, predictable traffic to the local models so the fixed monthly cost in the table above is spread across real work, and reserve the API for spikes and frontier calls so you keep that capability without paying for it around the clock. The arithmetic collapses the moment those cards sit idle, which is why a week of measured traffic should come before any hardware order.

Decision Framework: When to Self-Host vs Use Cloud APIs#

One framing note before the lists. Volume is the input everyone reaches for first, and the break-even math in the next section shows it is the weaker signal. What you are replacing moves the answer further than how much of it you send.

Use Cloud APIs When:#

• Your token volume is under 2 million per day
• The model you would be replacing is already a budget tier (gpt-5.6-luna, gpt-5-nano, GPT-4.1 mini). At those rates the hardware rarely pays for itself
• You need frontier capability, meaning Claude Opus or GPT-5.6 Sol class reasoning
• Your team lacks GPU infrastructure expertise
• Your workload is bursty and unpredictable
• Time-to-production matters more than per-token cost

Self-Host When:#

• You are replacing a mid-tier or frontier model (Sonnet, Opus, gpt-5.6-terra) and an open-weight model demonstrably holds quality on your prompts
• Privacy or compliance requirements mandate on-premise processing
• Latency to first token is critical (sub-50ms)
• You run an air-gapped or restricted network environment
• Your volume is sustained rather than spiky, so the GPUs stay busy
• You have (or can hire) infrastructure expertise to maintain the setup

Go Hybrid When:#

• You process 3–10 million tokens per day with variable load
• You need both frontier models and privacy guarantees
• Some of your traffic tolerates a smaller model and some genuinely does not
• You are building a product that serves multiple use cases with different requirements

Break-Even Analysis#

Most break-even tables answer the wrong question. They ask how much volume justifies buying a GPU. The stronger predictor, once you work the arithmetic against current rates, is which cloud model you are replacing.

Every figure below is reproducible. Assume an 80/20 input-to-output split, which is roughly what retrieval and support workloads look like, and a 30-day month. Cloud cost is (input tokens × input rate) + (output tokens × output rate) at the vendor rates cited earlier. Payback is hardware capex ÷ (monthly cloud cost − monthly electricity), because electricity is the cost you keep paying either way.

5 million tokens/day (120M input + 30M output per month), replaced by an 8B model on one RTX 4090 ($1,800 capex, $54/mo electricity):

10 million tokens/day (240M input + 60M output per month), replaced by a 70B model on 2× RTX 4090 ($3,600 capex, $108/mo electricity):

Look at the first table's top three rows. At 5 million tokens a day, a volume that comfortably clears the "you should self-host now" threshold most guides publish, replacing any of OpenAI's budget models with your own GPU never pays back inside the hardware's life. The cheap cloud tier got cheap enough that the electricity alone is competitive with the whole API bill.

Then look at the bottom rows of both tables. Replacing a frontier model pays for the hardware in weeks.

The catch sits in the word "replaced," and it is a quality claim rather than a cost one. Swapping GPT-4.1 mini for a local 8B is often a fair trade. Swapping Claude Opus 4.8 for a local 70B usually is not, and the payback column is meaningless if the local model cannot actually do the job. The honest sequence is to prove quality parity first on your own prompts, and only then run the money.

To put your own volume, split, and rates through this, use our free API vs self-hosting cost calculator. It reports monthly cost each way and the break-even token volume for the models you actually use.

Getting Started: Practical First Steps#

If you are exploring self-hosting for the first time:

1. Install Ollama (curl -fsSL https://ollama.com/install.sh | sh) and run a small model (ollama run llama3.2).
2. Test it against your actual workload, meaning your real prompts rather than benchmarks.
3. Compare output quality honestly against the cloud model you are considering replacing.
4. If quality is acceptable, measure your actual daily token volume for a week.
5. Run the break-even math with your numbers.

If you are scaling to production:

1. Deploy vLLM with your chosen model on GPU infrastructure.
2. Set up an OpenAI-compatible routing layer.
3. Start with cloud-only, then gradually shift predictable traffic to self-hosted.
4. Monitor quality, latency, and cost continuously.

If you are using AI tools in your development workflow, our Claude Code advanced workflow guide covers practical workflows for AI-assisted coding, whether the models run locally or in the cloud. For GPU server hosting, see our Hetzner Cloud AI GPU server guide.

When to Use This Analysis, and When to Skip It#

Use it when you are sizing a decision you have not made yet: someone has proposed buying GPUs, or proposed moving off them, and you need the arithmetic before the argument.

Use it when your bill is dominated by one model and one prompt shape. The tables above are most accurate when a single workload drives the spend, because that is when the 80/20 split assumption holds.

Skip it if your volume is under about 2 million tokens a day. Nothing here will change the answer. Stay on the API and spend the afternoon on something else.

Skip it if you have not yet proven an open-weight model can do the job. Payback periods for a swap you cannot actually make are entertainment.

Skip the cost framing entirely when the driver is regulatory. If HIPAA, FedRAMP, or a data-residency rule says the tokens cannot leave your infrastructure, the price comparison is irrelevant and you should be reading the privacy section instead.

Claim-to-Source Matrix#

Which numbers in this article we loaded from a primary page on 2026-08-17, and which are our own arithmetic on top of them.

What Effloow Added#

"Self-host or use the API" usually gets answered with a vibe and a threshold someone remembers from a conference talk. Three things here are ours rather than restated:

• A payback table whose every input is checkable. Vendor token rates, NVIDIA's published TGP, and the EIA electricity rate, combined by a formula printed above the table so you can redo it. No cost-per-token figure, because that would need a throughput denominator nobody publishes reproducibly.
• A correction to the standard advice. The finding that fell out of the arithmetic is that the model you are replacing predicts the answer better than your volume does. At 5 million tokens a day, replacing OpenAI's budget tier never pays back inside the hardware's life. Replacing a frontier model pays back in weeks at the same volume. Guides that key the decision on tokens-per-day get this backwards.
• An interactive cost calculator so the decision runs on your volume, split, and rates rather than our example.

Three corrections also came out of re-reading the primary sources: vLLM is not NVIDIA-only, the RTX 5090's 575W ceiling costs meaningfully more to run than its 32 GB headline suggests, and the electricity rate most comparisons use is several cents low.

If you need this run against your own prompts and your own bill rather than a worked example, that is what Effloow's Proof Studio produces. The services page covers scope.

The Bottom Line#

Self-hosting is mature. The tools are production-ready, the open-weight models are genuinely good, and the hardware is sitting on a shelf somewhere.

Cloud APIs got cheaper faster, though, and that is the part that has not filtered into most advice yet. Claude's 4.5-and-later generation cut Opus pricing by two thirds against Opus 4.1, and OpenAI's budget tier now starts at five cents per million input tokens. At low and moderate volumes the API wins on total cost, because you never pay for a card that is doing nothing.

So the sequence matters more than the verdict. Start on the API. Measure a real week of traffic and note which model the spend concentrates in. If it concentrates in a budget model, stop here, because the hardware will not pay for itself. If it concentrates in Sonnet or Opus, test whether an open-weight model holds quality on your actual prompts, and only if it does should you run the payback math.

Most teams land on a split rather than a side: local for the steady, privacy-bound, quality-tolerant traffic, and the API for spikes and for the reasoning that only a frontier model does well.