XDA’s report that a mid-range GPU can subsidize a stack of paid AI subscriptions is on the money. An RTX 4070 Ti or a Radeon RX 7900 XT runs 13B and 34B parameter models at a speed that beats what most cloud services charge for as a single subscription tier. But choosing which model, quantization, and runtime to use on the hardware you already own requires actual numbers, not vibes.
The seven apps below cover two kinds of benchmarking. The first three are runtime-integrated benchmarks that report tokens-per-second and time-to-first-token for real workloads. The rest are third-party benchmark suites that give reproducible scores for comparing hardware. Every pick runs on Windows and Linux; two also work on macOS with Apple Silicon.
What to look for in a GPU AI benchmark
Not every AI benchmark tells you what you need to know. Check for these five things:
- Reports tokens-per-second at multiple context lengths (128, 2048, 8192)
- Measures both prompt processing (prefill) and generation separately
- Supports the quantization you actually plan to use (Q4_K_M, Q8, FP16)
- Reads VRAM and system RAM utilization during the run
- Publishes results in a shareable format so numbers are comparable across setups
Anything that only reports a single throughput number in one context is not useful for real hardware sizing.
Quick comparison
| App | Best for | Free plan | Starting price | Standout feature |
|---|---|---|---|---|
| llama-bench | Reproducible LLM throughput | Yes | Free | Ships with llama.cpp |
| LM Studio | GUI benchmarks + model shopping | Yes | Free | Model catalogue built in |
| Ollama benchmark | One-command LLM stress test | Yes | Free | Runs against any pulled model |
| UL Procyon AI | Industry-standard vendor benchmark | Trial | Paid license | ONNX + DirectML paths |
| Geekbench AI | Cross-platform AI score | Yes | Free tier | iOS, Android, Windows, macOS |
| MLPerf Client | Reference benchmark from MLCommons | Yes | Free | Deep hardware profiling |
| koboldcpp benchmark | Roleplay/chat workload | Yes | Free | Simulates chat sessions realistically |
The apps
1. llama-bench – Best reproducible LLM throughput
llama-bench ships as part of llama.cpp and is the closest thing the local-LLM world has to a standard yardstick. Point it at a GGUF file, tell it a context size and batch size, and it reports prompt tokens per second, generation tokens per second, and time to first token. It runs on any CUDA, ROCm, Metal, or CPU backend, so a single tool gives comparable numbers across your Nvidia rig, Radeon build, and MacBook.
Where it falls short: Command line only. No GUI. Some options (BLAS, batch size) require reading the docs.
Pricing:
- Free: Open-source under MIT.
- Paid: None.
Platforms: Windows, macOS, Linux.
Download: llama.cpp on GitHub
Bottom line: The default benchmark when comparing quantizations or models. Learn its flags once.
2. LM Studio – Best GUI benchmarks with a model shopping trip
LM Studio is a desktop app that wraps llama.cpp with a graphical interface for downloading, running, and now benchmarking models. The built-in benchmark tab tests any downloaded model across a preset workload and reports metrics in a way non-experts can read. It also shows VRAM headroom left over, which helps size the next model up.
Where it falls short: Closed source. Model catalogue overlaps with HuggingFace and is not exhaustive.
Pricing:
- Free: Full app.
- Paid: None.
Platforms: Windows, macOS (Apple Silicon), Linux.
Download: LM Studio
Bottom line: The friendliest way to benchmark a fresh model without touching a terminal.
3. Ollama benchmark – Best one-command LLM stress test
Ollama ships with a run --verbose flag that produces per-token timings. Combined with community-maintained scripts, it doubles as a benchmark tool. Because Ollama pulls models by name, running the same benchmark on a second machine takes two commands.
Where it falls short: Not a full benchmark suite. Focused on completion speed; less useful for prompt-heavy workloads.
Pricing:
- Free: Open-source under MIT.
- Paid: None.
Platforms: Windows, macOS, Linux.
Download: Ollama
Bottom line: Install Ollama for daily model use; use --verbose when you want a quick throughput read.
4. UL Procyon AI – Best industry-standard vendor benchmark
UL Procyon AI Computer Vision Benchmark is the same UL that publishes 3DMark and PCMark. It runs standard ONNX models (MobileNet, ResNet, Inception) across TensorRT, DirectML, OpenVINO, and CoreML backends and produces a score. Vendors quote Procyon numbers in reviews, so the score is directly comparable across the industry.
Where it falls short: Focused on vision workloads; less useful for LLM throughput. Paid license for commercial use.
Pricing:
- Free: Trial score reveals base numbers.
- Paid: License unlocks the full run.
Platforms: Windows.
Download: UL Procyon AI
Bottom line: Buy this if you make purchasing decisions using vendor benchmarks. Skip if you only care about LLM speed.
5. Geekbench AI – Best cross-platform AI score
Geekbench AI produces three numbers per run: FP32, FP16, and quantized INT8. It runs across Windows, macOS, Linux, iOS, and Android, so a laptop-vs-phone comparison for the same workload is one benchmark away. The public results database lets you compare your run against thousands of others.
Where it falls short: Not LLM-specific. The three scores collapse a lot of nuance.
Pricing:
- Free: Preview run.
- Paid: Full license unlocks batch runs and history export.
Platforms: Windows, macOS, Linux, iOS, Android.
Download: Geekbench AI
Bottom line: The go-to number for “how does my AI-capable hardware stack up.”
6. MLPerf Client – Best MLCommons reference benchmark
MLPerf Client is the MLCommons-backed reference benchmark for AI on client hardware. It runs a preset LLM workload (currently Llama 2 7B, moving to newer models) and reports latency, throughput, and quality regression against a reference implementation. Because MLCommons publishes results transparently, comparing setups is straightforward.
Where it falls short: Setup is more involved than the alternatives. Not aimed at end users.
Pricing:
- Free: Open-source.
- Paid: None.
Platforms: Windows, Linux (native), macOS (community builds).
Download: MLPerf Client on GitHub
Bottom line: The authoritative benchmark. Worth the setup cost if you plan to publish a hardware comparison.
7. koboldcpp benchmark – Best roleplay and chat workload simulation
koboldcpp is a llama.cpp fork focused on roleplay and long-context chat. Its benchmark mode simulates a real chat session with rolling context, which is a more honest test of what a home LLM server will feel like day to day. The numbers include context-swap cost and system RAM overflow behavior, which llama-bench glosses over.
Where it falls short: Focused on chat/RP workloads; misses code-focused metrics. Interface reads busy at first.
Pricing:
- Free: Open-source.
- Paid: None.
Platforms: Windows, macOS, Linux.
Download: koboldcpp on GitHub
Bottom line: The most representative benchmark for anyone whose real use case is chat, not code.
How to pick the right one
Start with llama-bench for reproducible numbers across quantizations. Add LM Studio’s built-in benchmark tab if you prefer a GUI. Use Ollama’s verbose flag for a fast sanity check on a machine you already run models on. Add Geekbench AI when you need a single portable score. Save UL Procyon or MLPerf Client for formal hardware comparisons where third-party legitimacy matters. Use koboldcpp benchmark when your day-to-day workload is chat rather than code completion.
Skip vendor “AI TOPS” spec sheets as a substitute; the numbers are peak-theoretical and rarely reflect real inference throughput on the quantizations you use.
FAQ
Which GPU gives the best local LLM speed today? For consumer cards, the RTX 4090, 4080 Super, and 5080 lead on Nvidia; the RX 7900 XTX leads on AMD. Apple’s M3 Max and M4 Max close the gap for models that fit in unified memory.
How many tokens per second do I actually need? Real-time chat feels good above 20 t/s on the model’s output. Anything above 40 t/s is indistinguishable from a fast SaaS API.
Does more VRAM help or does compute win? For LLMs, VRAM is the harder constraint. Fitting the full model in VRAM without spilling to system RAM matters more than raw compute above a threshold.
Is a benchmark on Windows comparable to one on Linux? For CUDA-based tests, mostly yes. Driver differences add a few percent variance. For DirectML vs ROCm the difference is much larger and Linux numbers are usually higher.
What’s the cheapest benchmark option? llama-bench and Ollama both cost nothing. Both run on any consumer GPU.
Do these help me pick between LM Studio and Ollama? Yes. Benchmark the same model in both and compare tokens per second and prompt-processing speed. Ollama tends to win on model swap speed; LM Studio’s GUI helps for experiments.