Every message sent to ChatGPT or Claude leaves the building. For most people that trade is fine. For a growing number of home lab owners it is not, and a Proxmox server that already runs Plex, Home Assistant, and a Pi-hole is sitting on exactly the spare CPU, RAM, and (often) GPU that a local language model needs. One recent write-up from a self-hoster who ditched cloud AI for a local LLM on Proxmox summed up the appeal well: the hardware was already paid for, so the model became the thing that finally made the whole home lab worth it. We rounded up the seven apps worth running in a VM or an LXC container to get there, ranked on setup effort, hardware efficiency, and how close the daily experience gets to ChatGPT. These are the best apps for hosting a local LLM on Proxmox this year.
What to look for in a Proxmox LLM host
Not every local-LLM project is built to run headless on a hypervisor. A few things separate the ones worth installing from the ones that fight you.
- VM vs LXC container. An LXC container is lighter on RAM and disk and starts faster; a full VM is safer for GPU passthrough and isolates driver issues from the Proxmox host itself.
- GPU passthrough support. Anything running a model larger than a few billion parameters wants a dedicated GPU. Check whether the app plays well with an NVIDIA card passed through via vGPU or PCI passthrough, or whether it degrades gracefully on CPU-only hardware.
- An API-compatible endpoint. An OpenAI-compatible REST API is what lets other services on the network, like Home Assistant or a coding editor, talk to the model without custom glue code.
- A model manager. Pulling, swapping, and quantizing models from a command line or a simple UI beats hand-editing config files every time a new model drops.
- A web UI. A browser-based chat interface is what makes the server feel like a ChatGPT replacement rather than a backend service only other software can reach.
- Quantization support. GGUF and similar quantized formats are what let a 7B or 13B model fit in 8 to 16 GB of VRAM, or run on CPU at a usable speed.
Quick comparison
| App | Best for | Deployment | Free tier | GPU support |
|---|---|---|---|---|
| Ollama | Overall Proxmox host | Linux VM, LXC, Docker | Free, open-source | NVIDIA, AMD (ROCm), CPU |
| Open WebUI | Best UI on top of Ollama | Docker on same VM/LXC | Free, open-source | Inherits backend GPU |
| LM Studio | Model discovery on desktop | Desktop app (not headless) | Free | NVIDIA, Apple Silicon |
| text-generation-webui | Advanced and older model formats | Linux VM, Docker | Free, open-source | NVIDIA, AMD, CPU |
| vLLM | High-throughput, production serving | Linux VM, Docker | Free, open-source | NVIDIA (best), AMD (partial) |
| llama.cpp server | Leanest, GGUF-native | Linux VM, LXC, Docker | Free, open-source | NVIDIA, AMD, Apple Silicon, CPU |
| LocalAI | OpenAI-API drop-in replacement | Linux VM, LXC, Docker | Free, open-source | NVIDIA, CPU |
The 7 apps ranked
1. Ollama — Best overall
Ollama is the app most Proxmox home labbers land on first, and usually the one they stay with. Installation is a single shell command inside a Debian or Ubuntu VM (or a privileged LXC container with the right kernel modules), and pulling a model is as simple as ollama run llama3. It exposes an OpenAI-compatible API on port 11434 by default, which means Home Assistant, Continue.dev, and dozens of other tools connect to it without extra configuration.
Where it falls short: The bundled interface is a command-line chat, not a web UI, so most people pair it with Open WebUI or a similar frontend. GPU passthrough into an LXC container needs the NVIDIA driver matched exactly between host and container, which trips up first-timers more than the VM route does.
Pricing: Free, open-source.
Platforms: Linux VM, LXC, Docker, also native on Windows and macOS for local testing outside the lab.
Download: ollama.com
Bottom line: The default choice for a Proxmox LLM host, and the backend most of the other apps on this list are built to sit on top of.
2. Open WebUI — Best UI on top of Ollama
Open WebUI turns Ollama into something that actually looks and feels like ChatGPT: chat history, multiple conversations, model switching from a dropdown, document upload with retrieval-augmented answers, and multi-user logins with per-user permissions. It runs in its own Docker container right beside Ollama, so the two together fit comfortably in one LXC container or a small VM.
Family or household deployments benefit most here. Each person gets their own login, their own chat history, and an admin can set which models are available to whom.
Where it falls short: It is a frontend, not a model server, so it depends entirely on Ollama (or another OpenAI-compatible backend) already running underneath it. The permission and user-management screens take a few minutes to get comfortable with the first time.
Pricing: Free, open-source.
Platforms: Docker on a Linux VM or LXC container, alongside Ollama.
Download: github.com/open-webui/open-webui
Bottom line: The pick for anyone who wants the household to actually use the local LLM instead of just the person who set it up.
3. LM Studio — Best for model discovery
LM Studio is a polished desktop app for Windows, macOS, and Linux that makes browsing, downloading, and testing Hugging Face models painless, with a built-in catalog, RAM and VRAM estimates before you download, and a local server mode that exposes the same OpenAI-compatible API the other apps on this list use. It is less Proxmox-native than Ollama, since it is designed to run with a GUI on a workstation rather than headless in a container, but it earns a spot here because it is the fastest way to figure out which model actually fits your hardware before committing to a Proxmox deployment.
Some home labbers run LM Studio on a desktop machine purely for model research, then pull the winning model into Ollama on the Proxmox box for the always-on server.
Where it falls short: Running it headless inside a VM works but loses the point of the app, since the GUI is the main draw. It is not built for the kind of unattended, boot-and-forget operation a home lab server wants.
Pricing: Free.
Platforms: Windows, macOS, Linux desktop (not container-native).
Download: lmstudio.ai
Bottom line: Useful as a scouting tool before deployment, not as the Proxmox host itself.
4. text-generation-webui — Best for advanced and older model formats
text-generation-webui, often called “oobabooga” after its creator, is the veteran of this list and still the app with the broadest format support: GGUF, GPTQ, AWQ, and EXL2 all load through the same interface, along with LoRA fine-tune support and a notebook-style raw completion mode. For anyone running an older or more obscure model that newer tools have dropped support for, this is usually where it still works.
The web UI includes character cards, chat presets, and extensions for things like long-term memory and voice, features aimed more at hobbyist tinkering than at a clean daily-driver chat experience.
Where it falls short: The interface shows its age next to Open WebUI, and the sheer number of settings and extensions can overwhelm someone who just wants a working chat box. Updates occasionally break compatibility with specific quantization formats.
Pricing: Free, open-source.
Platforms: Linux VM, Docker, also runs on Windows and macOS.
Download: github.com/oobabooga/text-generation-webui
Bottom line: The right pick for tinkerers running unusual model formats or fine-tunes that other servers do not support.
5. vLLM — Best for high-throughput and production serving
vLLM is built for a different problem than most of this list: serving many concurrent requests fast, using PagedAttention to keep GPU memory efficient under load. On a Proxmox box with a proper GPU passed through, vLLM will out-throughput Ollama by a wide margin when several people or several apps are hitting the model at once, which matters for a home lab running an LLM behind multiple services (a coding assistant, a chat UI, and an automation pipeline, all at the same time).
It speaks the OpenAI API natively, so it slots into the same tools that talk to Ollama.
Where it falls short: Setup is heavier, GPU requirements are stricter, and single-user chat performance is not meaningfully better than the simpler options here. CPU-only or low-VRAM home labs get little benefit from it.
Pricing: Free, open-source.
Platforms: Linux VM with GPU passthrough, Docker.
Download: github.com/vllm-project/vllm
Bottom line: Overkill for a single-user replacement of ChatGPT, but the right tool once a home lab starts serving a local model to multiple apps or people at once.
6. llama.cpp server — Leanest, GGUF-native
llama.cpp is the C++ inference engine most of the apps on this list run on under the hood, and its own bundled server (llama-server) is the leanest way to expose a GGUF model over an OpenAI-compatible API without any extra layers. Resource usage is the lowest on this list, which makes it a strong fit for a small LXC container or a Proxmox host that also needs to run other services on the same hardware.
It includes a minimal built-in web chat interface, functional but plain, which is enough for basic use without adding Open WebUI on top.
Where it falls short: No model manager, no multi-user accounts, and configuration happens through command-line flags rather than a settings screen. It rewards someone comfortable with a terminal more than someone who wants a point-and-click setup.
Pricing: Free, open-source.
Platforms: Linux VM, LXC, Docker, also builds on Windows, macOS, and Apple Silicon.
Download: github.com/ggerganov/llama.cpp
Bottom line: The pick for squeezing a model onto minimal hardware or into a resource-constrained LXC container.
7. LocalAI — Best OpenAI-API drop-in
LocalAI aims to be a near-total drop-in replacement for the OpenAI API, covering not just chat completions but image generation, text-to-speech, and embeddings behind the same interface. For a home lab that already has apps or automations wired to call the OpenAI API, LocalAI lets that integration point stay unchanged while the actual inference happens locally on the Proxmox box.
It supports a wide range of backends, including llama.cpp, so the same server can run several different model formats depending on what a given request needs.
Where it falls short: The breadth of supported model types adds configuration complexity compared to a single-purpose server like Ollama, and the project moves fast enough that documentation occasionally lags behind the latest release.
Pricing: Free, open-source.
Platforms: Linux VM, LXC, Docker.
Download: localai.io
Bottom line: The right choice when existing tooling is already built against the OpenAI API and swapping the endpoint, not rewriting the integration, is the goal.
How to pick the right one
For a single person replacing ChatGPT for daily use, Ollama paired with Open WebUI covers nearly everything: a chat interface, model switching, and an API endpoint for anything else on the network. A household sharing one Proxmox box benefits from the same pairing, since Open WebUI’s per-user logins keep chat histories separate without needing multiple VMs.
A hobbyist who wants to poke at fine-tunes, character presets, or older quantization formats gets more mileage from text-generation-webui, even with its steeper learning curve. Someone still deciding which model fits their hardware should start with LM Studio on a desktop before committing a Proxmox VM to a specific setup.
A GPU-rich rig serving multiple apps or users at once is the one case where vLLM earns its extra setup cost, since its throughput advantage only shows up under concurrent load. A CPU-only box, or a Proxmox host with limited RAM to spare, is better served by llama.cpp server directly, which carries the least overhead of anything on this list. And a lab with existing automations already calling the OpenAI API is the one clear case for LocalAI, since it lets that integration point stay exactly as it is.
FAQ
Can Proxmox run LLMs without a GPU? Yes. Ollama, llama.cpp server, and LocalAI all run on CPU alone, and a quantized 7B or 8B model is usable for chat on a modern multi-core CPU, though responses come noticeably slower than on a GPU. Smaller quantized models (3B and under) stay responsive even on modest hardware.
Which local LLM is closest to ChatGPT quality? Model quality depends on which weights are loaded, not which server runs them, so any of these apps can run the same models. Larger open models in the 70B-plus range, run in a good quantization, come closest to ChatGPT-level responses, though they need substantial VRAM or a slow CPU fallback to run well.
Is Ollama free? Yes. Ollama is free and open-source, with no paid tier, subscription, or usage limits, since inference happens entirely on the hardware it runs on.
Should I use a VM or LXC container for Ollama? An LXC container is lighter and starts faster, which suits a CPU-only setup or a lab tight on RAM. A VM is the safer choice for GPU passthrough, since it isolates the NVIDIA driver stack from the Proxmox host and avoids the version-matching issues that come up when a container shares the host’s kernel and drivers.
What model runs on 16 GB VRAM? A 13B model in a 4-bit or 5-bit GGUF quantization fits comfortably in 16 GB of VRAM with room for a reasonable context window. Some well-optimized 34B quantizations also fit at a reduced context length, though 13B and under gives the most headroom for longer conversations.