Self-hosted LLMs: The complete infrastructure guide in 2026

A self-hosted LLM is an open-weight language model that runs entirely inside your own data centre or a cloud environment you control. Your data never leaves your perimeter, you pay no per-token rate, and you decide which model version sits in production. What you trade for it is work: hardware, an inference runtime and a governance layer you build yourself.

In 2026 that is no longer a theoretical trade-off. Most production architectures have settled into a hybrid pattern: sensitive and high-volume workloads run internally, while the heaviest reasoning tasks stay on a commercial API. This guide covers what that takes in practice, from model selection and VRAM maths through runtimes, governance and the question of when self-hosting actually adds up.

Key takeaways

• A self-hosted LLM runs entirely on your own infrastructure, keeping proprietary and regulated data inside your security perimeter.
• The shift to private AI is driven by three forces: data sovereignty, lower cost at high volume, and deterministic latency.
• Open-weight models including DeepSeek V4, GLM-5.2, Qwen 3.5 and Gemma 4 are competitive with proprietary models across most enterprise benchmarks.
• Check the licence before you build. The Llama 4 community licence excludes EU-established companies, which rules the model out for most European organizations.
• Hardware planning starts with VRAM. Weights must stay resident in fast memory. INT8 halves that requirement against FP16, and Q4 brings it down to roughly a quarter.
• Production runtimes such as vLLM, SGLang and TensorRT-LLM deliver high throughput. Ollama and llama.cpp suit development and small teams.
• Budget 0.5 to 1 FTE of engineering for the first year. For most teams that is the largest cost, not the GPU.

Why organizations move to private AI infrastructure

Private AI infrastructure is the combination of compute hardware, orchestration software and network configuration dedicated exclusively to running local machine learning models. Four motives come up repeatedly.

Data sovereignty, GDPR and the EU AI Act

For organizations handling patient records, financial files, legal documents or government information, where data gets processed is not a detail. The moment you send prompts to a processor outside the EU, you take on a third-country transfer and the question of how the US CLOUD Act interacts with your GDPR obligations. A hyperscaler with EU data residency does not fully remove that question either, and a share of the European public and healthcare sector will not accept the residual risk.

Self-hosting removes the transfer entirely. No prompt, no document and no interaction log leaves your own perimeter. That materially simplifies your record of processing activities, your DPIA and the conversation with your data protection officer.

It does not remove your obligations. Since 2 August 2026 the Article 50 transparency obligations of the EU AI Act apply: you must disclose when a user is interacting with AI and mark AI-generated content in machine-readable form. Obligations for high-risk systems were pushed to December 2027 by the Digital Omnibus package, which buys you time to get documentation in order. GDPR and the EU AI Act do not replace each other: GDPR governs personal data, the AI Act governs the AI system itself, and plenty of deployments fall under both.

Security and governance move in-house

Data sovereignty is only one layer. Self-hosting moves the full security and compliance burden inside, so you build the controls a commercial provider would otherwise supply.

Auditability and logging are the foundation. Capture prompts, outputs and access events in tamper-evident logs, enforce role-based access control on your inference endpoints, and map that practice to a framework such as SOC 2 or ISO 27001. Because the model runs inside your own environment, the same audit trail serves both internal review and regulatory evidence.

Content filtering has to be added deliberately. Open-weight models ship without the guardrails built into hosted APIs. A production-grade self-hosted stack therefore wraps the model in classifiers or open source safety filters that screen inputs and outputs for harmful, non-compliant or data-leaking content before a response reaches the user.

Supply chain risk shifts to you. Review model licences for commercial use and redistribution, verify the provenance of downloaded weights, and scan container images for vulnerabilities before promoting them. Treat model artefacts with the same discipline you apply to software dependencies, because this is the gap that gets skipped most often.

When self-hosting actually costs less

A commercial API has a low entry barrier and scales linearly with usage. Self-hosting swaps that variable cost for a largely fixed one: hardware or reserved GPU capacity, plus operations. The break-even point sits higher than most teams expect.

For a workload of roughly 10 million tokens per day, a commercial API runs around 250 euros per month, and a cloud platform with EU data residency around 540 euros. The same workload on two rented L40S cards runs closer to 2,150 euros. Below 5 million tokens per day an API almost always wins. Between 5 and 50 million it is close, and above that self-hosting starts to pay, provided your utilisation is high and steady. These are European market figures, so treat them as an order of magnitude rather than a quote.

The item most often missing from the calculation is people. Budget a minimum of 0.5 to 1 FTE of engineering for the first year to build, monitor and update the stack. At most organizations that is the real cost driver. You buy a GPU once. You pay every month for the team handling model updates, quantization and incident response.

If you want to cut the bill before buying hardware, apply the caching, batching and routing patterns to your existing API workload first. That moves the break-even point and sharpens the business case for self-hosting.

Deterministic latency and execution control

Public API endpoints come with variable latency, rate limits and unexpected downtime during global peaks. Run the model yourself and you control the execution variables. Techniques such as prefix caching, speculative decoding and dedicated batching queues deliver predictable response times for applications where that predictability is business-critical, such as real-time customer contact or any process under a hard SLA.

The open-weight models worth running in 2026

The performance gap between closed frontier models and open weights has largely closed. What remains is a choice between architectures that fit different hardware, and a licensing question you are better off asking before you build rather than after.

Flagship reasoning and general-purpose models

• DeepSeek V4: the reasoning flagship, released in April 2026 with a native 1 million token context window. The Pro variant carries 1.6 trillion parameters with 49 billion active per token; the Flash variant 284 billion total with 13 billion active. Strong on algorithmic problem solving, formal maths and multi-step analysis. The MIT licence makes commercial use straightforward.
• Google Gemma 4: derived from Google’s Gemini research and released in April 2026 across five sizes, from edge variants of around 2 billion parameters to a 31 billion dense model. Every variant shares a 256K token context window and supports 140+ languages. The 31B model runs on a single server GPU, which makes it one of the more practical starting points for a first on-premise deployment.
• Qwen 3.5 and Mistral: two families that do well on performance per parameter. Qwen ships under Apache 2.0, which removes commercial-use friction entirely. Mistral is a European provider with Apache-licensed releases, which counts beyond the technical merits in procurement processes where European origin and data residency are explicit criteria.

Check the licence first: Llama 4 and the EU

Meta’s Llama was for years the default starting point for enterprise fine-tuning, backed by the broadest tooling ecosystem and the most documentation. In 2026 that is no longer an automatic choice for European organizations. The Llama 4 community licence excludes companies established in the EU from accepting its terms, which blocks commercial deployment from an EU entity.

In practice this means the following. If you want to keep the familiar Llama workflows, the older Llama 3.x models remain available. If you are building something new, pick Qwen, Mistral, DeepSeek V4 or Gemma 4, and get the licence reviewed before the first sprint rather than the week before launch. Gemma carries its own caveat: internal use is clear, but shipping a model you fine-tuned on proprietary data to external users warrants a legal view on whether Google’s restrictions carry over to the derivative.

Small models for edge and on-device workloads

Self-hosting is not only about flagship models. A growing class of compact models between 3 and 8 billion parameters delivers usable quality on narrow tasks while running on modest hardware: a single consumer GPU, a laptop or an edge device. Qwen3.5-4B, Phi-4-mini and the smaller Gemma variants all sit in this class.

These models suit on-device assistants, real-time classification, summarization and retrieval pipelines, where latency and cost matter more than frontier reasoning. They also serve as efficient draft models for speculative decoding alongside a larger system. For many enterprise workflows a well-tuned small model running locally is more practical than a giant model that demands data centre accelerators.

Specialized agentic and coding models

• GLM-5.2 (Zhipu): engineered for long-horizon agentic work and multi-step tool use. It uses sparse attention to keep processing overhead low during complex execution inside enterprise systems, and ranks at the front of SWE-bench-style coding tasks in 2026. Available under MIT.
• Kimi K2.6: a multimodal Mixture-of-Experts system of roughly 1 trillion parameters, configured to coordinate multiple agents. It activates only a subset of parameters per token, which optimizes throughput for combined vision and text analysis. Note the catch with MoE: all expert weights still have to fit in memory, so the memory bill does not shrink with the active parameter count.

Hardware: what you actually need

Hardware follows from the model and the quantization level, not the other way round. Start with how much VRAM you need and work back to the cards that provide it.

GPUs and memory tiers

The baseline requirement for efficient inference is available Video RAM (VRAM) and memory bandwidth. Because model weights must stay resident in fast VRAM during inference, your hardware choice directly caps the model size you can run.

Tier	Memory	What it runs	Typical use
Workstation	32 to 48 GB VRAM	Models up to roughly 30B at INT8, or 70B at Q4	A single RTX 5090 (32 GB) or two cards together. Development, internal tooling, small teams
Apple Silicon	64 to 256 GB unified memory	70B comfortably, up to the 200B class at Q4	Mac Studio as a quiet, power-efficient node for mid-size models. Lower throughput than a discrete GPU
Data centre node	80 to 192 GB VRAM	Dense flagship models in real time	Nvidia H100, H200 or B200. Production inference with many concurrent users
Scale-out cluster	Multiple nodes	Frontier MoE models of 400B and above	InfiniBand interconnect with tensor or pipeline parallelism across units

If you would rather not run your own server room, there is a middle path between hyperscaler and owned rack. Colocation, or reserved GPU capacity at a European provider such as OVHcloud, Scaleway or Hetzner, gives you data residency inside a chosen jurisdiction without building a data centre. For many healthcare, government and financial organizations that is the practical form sovereignty takes.

How to calculate your VRAM requirement

To estimate the absolute minimum VRAM for inference, use one formula:

Total VRAM = model weights + KV cache + overhead

Model weights = parameters × bytes per parameter. The byte count depends on the precision or quantization level:

Precision	Bytes per parameter	70B model	8B model
FP32	4	about 280 GB	about 32 GB
FP16 / BF16	2	about 140 GB	about 16 GB
INT8	1	about 70 GB	about 8 GB
INT4 (Q4)	about 0.56	about 39 GB	about 4.5 GB
INT2 (Q2)	about 0.33	about 23 GB	about 2.6 GB

KV cache. The KV cache holds context from previous tokens during generation. Its size grows with context length, batch size and model architecture. At modest batch sizes and short to medium context (4K to 8K tokens), plan for 1 to 2 GB. At large context windows of 32K to 128K tokens and beyond, the KV cache can expand to 8 GB or more.

Overhead. The CUDA runtime, kernels and internal activation buffers all consume memory. Rule of thumb: multiply your base model size by 1.2 to 1.3, or reserve a flat 1 to 1.5 GB. Expect real-world usage to run 10 to 20 percent above the bare calculation.

Quantization at Q4_K_M or Q5_K_M is safe for most production workloads. Quality loss is limited, while the memory requirement drops to roughly a quarter of FP16. Where it does matter: structured output, maths and retrieval over long context. Test that on your own workload rather than trusting aggregate benchmarks, particularly in medical or legal domains.

Enterprise inference runtimes and deployment stacks

The inference runtime is the software layer that turns raw open-weight files into a stable API. This choice determines your throughput, your memory behaviour and how many concurrent users you can serve.

High-throughput production runtimes

Runtime	Strength	When to pick it
vLLM	PagedAttention manages the KV cache dynamically and prevents fragmentation	The broad default for production inference with many concurrent requests
SGLang	RadixAttention and optimized MLA kernels, plus structured output (JSON schemas)	At high throughput, on DeepSeek architectures, and anywhere you enforce schema validation
TensorRT-LLM	Compiles models into optimized engines with fused kernels and FP8/INT4	When extracting peak performance from a dedicated Nvidia fleet is the priority
Text Generation Inference (TGI)	Tensor parallelism, optimized tokenizers and validation layers	If you already work inside the Hugging Face ecosystem and want a proven platform
Unsloth	Sharply reduced memory consumption for fine-tuning and inference	If you fine-tune models in-house on modest hardware before promoting them to production

The relationship between vLLM and SGLang has shifted in 2026. SGLang is no longer a runner-up but a co-leader: on H100 hardware it delivers roughly 29 percent more throughput than vLLM on Llama 3.1 8B, and on DeepSeek architectures the gap widens to more than three times, thanks to its native MLA kernels. If you are running DeepSeek, that alone justifies benchmarking both.

Development and small-team stacks

• Ollama: wraps the complexity into single commands. Widely used for internal proofs of concept, test environments and small teams on a local server.
• llama.cpp: the underlying C/C++ implementation that runs across a wide range of consumer and professional hardware, with manual control over CPU threads, context window and memory compression. Also the route if you deliberately want to run without a GPU.

How to approach the rollout

Putting a private model into core business systems calls for a phased approach. Skip it and you end up with a technically correct stack nobody uses.

Phase 1: strategic evaluation

Before you commit budget to your own compute, decide which workloads belong internally at all. Map your processes, estimate token volume per process, and test per workload whether self-hosting offers a real advantage over a commercial API. Workloads with sensitive data or high steady volume rise to the top; occasional and experimental work stays in the cloud. A durable AI strategy ties that technical choice to longer-term business goals.

Phase 2: optimization and application development

Once the infrastructure is in place, development teams extract performance with three techniques:

1. Retrieval-Augmented Generation (RAG). Connect the model to a local vector database so it answers from your own documentation without touching the underlying weights. See the explainer on RAG for how it works.
2. Quantization pipeline. Compress FP16 parameters to INT8 or INT4 so you get more throughput out of the servers you already run. Validate against your own evals afterwards.
3. Orchestration and interface. Put a layer such as Open WebUI or your own frontend in front of the model so it becomes part of the daily working environment instead of an endpoint nobody reaches. If you want the build steps from the ground up, the guide to building your own LLM application covers them.

Phase 3: long-term operational management

This is the phase business cases underestimate most. Open-weight models ship at pace, and every new generation calls for retesting, requantizing and a controlled rollout. Add monitoring on latency, memory pressure and output quality, plus an evaluation set that grows with your application. Without that discipline you end up a year later with a stale model in production that nobody dares touch.

If you want to work out which workloads belong internally and which do not, an AI assessment maps your current usage and risks.

Conclusion

Self-hosting an LLM in 2026 is no longer a research experiment but a workable production option for organizations that need control over their data, costs and performance. The gap between open weights and proprietary models has narrowed enough that families like DeepSeek, Qwen, GLM, Mistral and Gemma can anchor serious enterprise workloads, from frontier reasoning down to compact models at the edge.

Success depends less on the model you pick than on the discipline around it. The organizations that get the most from private AI infrastructure match hardware to real VRAM and quantization needs, choose an actively maintained runtime such as vLLM, SGLang or TensorRT-LLM, review the licence before they build, and treat security, governance and evaluation as first-class concerns.

So start small and specific. Pick one workload with clear value and a clear compliance argument, prove it against measurable criteria, and scale only once the operational discipline is in place. That beats buying a cluster and then going looking for a use case.

Frequently asked questions (FAQ)