Nobody needs a 24 GB card to run a useful local model in 2026. Quantised 7 to 14 billion parameter models fit in 8 to 12 GB of VRAM at speeds a used GTX 1080 or RTX 2080 turns in comfortably, and the difference between “cloud only” and “local most of the time” is now a driver install and a 4 GB download. The trick is the runtime. Some apps assume a fresh RTX 50-series card; others were built for exactly the older hardware sitting in a home server chassis.
We tested 8 desktop apps for running local AI on old NVIDIA GPUs, focused on Pascal (GTX 10-series and Quadro P), Turing (RTX 20-series and Quadro T), and early Ampere (RTX 30-series and workstation A-series). Every app on the list runs Llama 3.1, Qwen 2.5, Mistral 7B, or Gemma 2 on 8 GB of VRAM with a usable token-per-second rate. The one that fits your setup depends on whether you want a chat window, a service, or a low-level runtime.
What to look for in a local AI app for older GPUs
The hardware constraints matter more than the marketing copy.
- CUDA 11 support, not just CUDA 12 (Pascal caps at CUDA 12.6 on Linux, later on Windows).
- Quantisation formats that make 7B fit in 8 GB (GGUF Q4_K_M, Q5_K_M, or AWQ 4-bit).
- Layer offloading so a 13B model can spill excess layers to CPU without becoming useless.
- A one-click install that doesn’t require compiling PyTorch from source.
- An OpenAI-compatible API so any client can point at localhost without a custom shim.
- Model download that resumes and verifies checksums (dial-up-era paranoia still helps).
Quick comparison
| App | Best for | Windows / Linux | Free plan | Standout feature | Licence |
|---|---|---|---|---|---|
| Ollama | Headless daemon and scripts | Both | Free | One-line model pull, CLI-first | MIT |
| LM Studio | Everything-in-one chat | Both | Free | Model browser + chat + server | Proprietary |
| KoboldCPP | Single-executable runtime | Both | Free | No install, GGUF, TTS, images | AGPL 3.0 |
| text-generation-webui | Power-user tinkering | Both | Free | Every quantisation format supported | AGPL 3.0 |
| GPT4All | First model on a modest laptop | Both | Free | LocalDocs sidebar, low-VRAM presets | MIT-like |
| Jan | Fully open-source LM Studio | Both | Free | MCP support, OpenAI-compatible | Apache 2.0 |
| llama.cpp | Bare-metal speed | Both | Free | Fastest CPU + partial GPU offload | MIT |
| vLLM | Multi-user API server | Linux (WSL on Windows) | Free | Paged attention, best throughput | Apache 2.0 |
The apps
1. Ollama – best headless daemon on old NVIDIA cards
Ollama is the runtime most people install first because the whole workflow is ollama pull llama3.1:8b then ollama run llama3.1. On a GTX 1080 Ti (11 GB), Llama 3.1 8B Q4_K_M runs at 25 to 30 tokens per second; on an RTX 2080 Super (8 GB), the same model runs around 40. Ollama binds an OpenAI-compatible endpoint on localhost:11434, and any front end that speaks OpenAI (Open WebUI, Msty, LibreChat) plugs in with a base URL change.
Where it falls short: The chat client is a terminal. Ollama does not ship a GUI; graphical use means adding Open WebUI or another front end.
Pricing:
- Free: Everything, MIT-licensed
- Paid: None
Platforms: Windows, macOS, Linux
Download: ollama.com
Bottom line: The right pick when the goal is to serve a model to other tools and scripts, not chat in an app.
2. LM Studio – best all-in-one for a first install
LM Studio is the “download an EXE and be talking to a model in five minutes” option. The app bundles a Hugging Face browser, chat pane, and a toggle to expose an OpenAI-compatible server on localhost:1234. VRAM detection is accurate on older cards, and the model list flags which builds fit fully in your GPU versus which need CPU offload.
Where it falls short: The client is closed source. Commercial use requires filling in the LM Studio Team-licence form and waiting for a quote.
Pricing:
- Free: Personal use
- Paid: Team licence on request
Platforms: Windows, macOS, Linux
Download: lmstudio.ai
Bottom line: The friendliest first install on older NVIDIA hardware, with the fastest path to a working chat window.
3. KoboldCPP – best zero-install runtime
KoboldCPP is a single executable (koboldcpp.exe on Windows, a static binary on Linux) that runs GGUF models, Stable Diffusion image generation, and Whisper transcription without any Python environment. On a GTX 1660 Super (6 GB), a 7B Q4 model still fits with a few layers offloaded to CPU, and the same executable handles a quick image generation without switching apps.
Where it falls short: The UI is web-based (opens in a browser tab) and feels functional rather than polished. Feature depth is high, discoverability is low.
Pricing:
- Free: Everything, AGPL 3.0
- Paid: None
Platforms: Windows, macOS, Linux
Download: github.com/LostRuins/koboldcpp
Bottom line: The pick when the target machine is a laptop, a homelab, or a friend’s PC where you don’t want to install Python.
4. text-generation-webui – best for power users
text-generation-webui (oobabooga) supports every quantisation format that matters (GGUF, GPTQ, AWQ, exllamav2), lets you switch loaders on the fly, and exposes low-level generation knobs (rope scaling, mirostat, dynamic temperature) that other apps hide. On an RTX 3060 (12 GB), it happily runs a 14B model at Q4 with 20 to 30 tokens per second.
Where it falls short: The installer pulls a few gigabytes of Python dependencies. First launch takes 5 to 10 minutes. Not the tool for someone who wants to click one thing and chat.
Pricing:
- Free: Everything, AGPL 3.0
- Paid: None
Platforms: Windows, macOS, Linux
Download: github.com/oobabooga/text-generation-webui
Bottom line: Ideal for the user who wants to compare model quantisations and generation strategies on the same hardware.
5. GPT4All – best low-VRAM starting point
GPT4All has the lowest floor: the built-in model catalogue flags 3B and 4B models that run on 4 GB cards, and the LocalDocs sidebar handles small RAG jobs without a separate vector database. On a GTX 1060 (6 GB), Mistral 7B Instruct runs at a usable 15 to 20 tokens per second.
Where it falls short: The GUI is designed for individual use; there’s no built-in multi-user API. Model catalogue is smaller than LM Studio’s or Ollama’s.
Pricing:
- Free: Everything
- Paid: None; enterprise support is a paid add-on
Platforms: Windows, macOS, Linux
Download: gpt4all.io
Bottom line: The right first install on a laptop with 4 to 6 GB of VRAM.
6. Jan – best fully open-source LM Studio equivalent
Jan is what LM Studio would look like if the client itself was Apache 2.0. First-party model catalogue, OpenAI-compatible server on localhost:1337, Model Context Protocol support so Claude Desktop and Continue can hit a Jan-hosted model, and no telemetry.
Where it falls short: Younger project than LM Studio; the catalogue is smaller and some Hugging Face quantisations arrive later. Windows GPU acceleration on non-CUDA hardware still catching up.
Pricing:
- Free: Everything, Apache 2.0
- Paid: None
Platforms: Windows, macOS, Linux
Download: jan.ai
Bottom line: The open-source pick when you want an LM Studio-style app whose licence you can read in an afternoon.
7. llama.cpp – best bare-metal runtime
llama.cpp is the C++ project the rest of the ecosystem is built on. It runs everywhere, compiles in five minutes on Linux, and produces the fastest single-user throughput on older GPUs because there’s no Python abstraction between the driver and the model. On a GTX 1080 Ti with -ngl 33, it hits raw token rates the higher-level apps trail by a few percent.
Where it falls short: No installer, no GUI. You’re compiling and running command-line tools. First-time setup for a Windows user is more work than any other app on this list.
Pricing:
- Free: Everything, MIT
- Paid: None
Platforms: Windows, macOS, Linux
Download: github.com/ggml-org/llama.cpp
Bottom line: For developers who want to know exactly what happens between the model file and the first token.
8. vLLM – best multi-user API server on Linux
vLLM is the runtime for turning an old workstation into a small team’s inference server. Paged attention scales throughput close to linear with concurrent requests, and 4-bit AWQ quantised models let a 24 GB card serve two or three people typing at once without stalling. Runs on Linux natively; Windows users need WSL 2 with CUDA passthrough.
Where it falls short: Not a chat app. vLLM is an OpenAI-compatible server and expects clients to hit it. Small-model support is fine but the project’s optimisation focus is bigger deployments.
Pricing:
- Free: Everything, Apache 2.0
- Paid: None
Platforms: Linux (WSL on Windows)
Download: github.com/vllm-project/vllm
Bottom line: The pick when the goal is a private OpenAI clone that a handful of people share.
How to pick the right one
- If you want a chat window in the next 10 minutes: LM Studio or GPT4All.
- If you’re serving models to other tools: Ollama.
- If you want the same experience open-source: Jan.
- If your machine can’t install anything system-wide: KoboldCPP.
- If you want to squeeze the last few percent of performance: llama.cpp.
- If you want to A/B model builds and quantisations: text-generation-webui.
- If a small team is going to share the machine: vLLM on Linux.
FAQ
How old is too old for a GPU to run local AI?
Cards with 4 GB of VRAM and Compute Capability 6.0 or higher (Pascal onward) can run 3B and 4B models comfortably. Below 4 GB, the practical answer is CPU-only inference with a small model.
Does a GTX 1080 Ti still make sense in 2026?
Yes for local AI. It has 11 GB of VRAM, wide CUDA 12 support, and enough compute for 7B and 8B models at 25 to 30 tokens per second. Not enough for image generation at high resolutions.
Which quantisation format is best for old cards?
GGUF Q4_K_M or Q5_K_M for chat quality, AWQ 4-bit for maximum throughput when using vLLM. GPTQ still works but the format is quietly being retired.
Can I run these apps on AMD or Intel GPUs?
Ollama, LM Studio, KoboldCPP, and llama.cpp support Vulkan or ROCm on many AMD cards. Intel Arc is supported via SYCL in llama.cpp and OpenVINO in a few forks. This article focuses on NVIDIA because that’s where the older-card conversation lives.
Do I need to worry about telemetry?
Ollama, KoboldCPP, Jan, llama.cpp, GPT4All, and vLLM ship with no telemetry. LM Studio has telemetry toggles in settings; opt out on first run.
