Getting Gemma 4 running on a 16GB laptop is a Sunday afternoon. Knowing how it performs is a longer conversation. If you have swapped between Ollama, LM Studio, and llama.cpp for the third time and still cannot answer “which quantisation is actually better for my hardware”, one of these seven local LLM benchmarking apps is the way out of the guessing game.
We picked apps that report reproducible numbers on desktop for a single-user local LLM setup. Speed benchmarks (tokens per second on your hardware) sit alongside quality benchmarks (accuracy on public suites) so you can pick a tool that matches what you are asking.
What to look for in a local LLM benchmarking app
- What you are actually measuring. Throughput, latency, VRAM footprint, quality on a known dataset, and per-task quality on your own dataset are five different questions.
- A local runner. The tool needs to hit the model where it lives. That means Ollama, llama.cpp, LM Studio’s OpenAI-compatible server, or a direct Python callable.
- A known dataset. MMLU, TruthfulQA, HellaSwag, and IFEval are the frequently cited names. If none of them are in the tool, it is not a benchmark.
- Repeatable output. A JSON or CSV report with a hash of the model and the config, not just a screenshot.
- Cost visibility. Even a “local” run has a wall-clock cost. Tools that surface time-to-first-token and steady-state throughput make the trade-offs visible.
Quick comparison
| App | Best for | Platforms | Free | Standout |
|---|---|---|---|---|
| lm-evaluation-harness | Reproducing published quality scores | Windows, macOS, Linux | Free, open-source | Reference implementation of MMLU, IFEval, etc. |
| llama-bench | Raw throughput and latency | Windows, macOS, Linux | Free | Ships with llama.cpp, GGUF-native |
| MLPerf Client | Vendor-neutral hardware benchmark | Windows | Free | AMD, Intel, Nvidia all use it |
| DeepEval | Your own dataset, your own metrics | Windows, macOS, Linux | Free tier | Pytest-style syntax, LLM-as-judge |
| Promptfoo | Comparative eval across models | Windows, macOS, Linux | Free, open-source | Side-by-side output diffs in the browser |
| HELM | Comprehensive academic benchmark | Windows, macOS, Linux | Free, open-source | Reference for cross-model comparison |
| OpenAI Evals | Prompt-level test cases | Windows, macOS, Linux | Free, open-source | Easy authorship, huge community catalog |
The apps
1. lm-evaluation-harness — Best for known-benchmark reproducibility
lm-evaluation-harness, from EleutherAI, is the reference implementation used by most model releases to publish MMLU, IFEval, HellaSwag, ARC, TruthfulQA, and dozens of others. Point it at a local model and reproduce the same numbers the paper cited.
Where it falls short: Dataset downloads are large. First-run setup on a laptop asks for tens of gigabytes of cache.
Pricing:
- Free: Full open-source library
- Paid: None
Platforms: Windows, macOS, Linux (Python)
Download: GitHub
Bottom line: The pick when the goal is “reproduce the score in the paper on my hardware”.
2. llama-bench — Best throughput and latency test
llama-bench ships inside the llama.cpp repository and measures raw tokens per second across GGUF quantisations, prompt lengths, and backend configurations. It answers “will Q5_K_M actually be faster than Q4_K_S on this CPU” without a chat wrapper in the way.
Where it falls short: Speed only. Says nothing about output quality.
Pricing:
- Free: Bundled with llama.cpp
- Paid: None
Platforms: Windows, macOS, Linux
Download: GitHub
Bottom line: The pick when you need throughput numbers to compare a model on your machine.
3. MLPerf Client — Best vendor-neutral hardware benchmark
MLPerf Client is the MLCommons benchmark that AMD, Intel, and Nvidia all quote for LLM inference performance on end-user hardware. Numbers are directly comparable across vendors because everyone runs the same suite.
Where it falls short: Windows first. The model list is fixed and curated, so it is not a general-purpose benchmarking tool.
Pricing:
- Free: Full suite
- Paid: None
Platforms: Windows
Download: mlcommons.org
Bottom line: The pick to compare a Ryzen AI laptop with a Core Ultra one without arguing over methodology.
4. DeepEval — Best for your own dataset
DeepEval is a Python framework that turns evaluation into pytest-style test cases. Metrics include LLM-as-judge scoring, hallucination detection, contextual precision, and answer relevancy, and every test can run against a local model.
Where it falls short: Some advanced metrics call out to hosted models by default. Reading the config to point them at a local judge is worth an evening.
Pricing:
- Free: Full open-source library
- Paid: Managed observability tier
Platforms: Windows, macOS, Linux
Download: GitHub
Bottom line: The pick when the benchmark you want to run is one you write yourself.
5. Promptfoo — Best for side-by-side comparison
Promptfoo runs the same prompts against several models and produces a browser-based diff view. It doubles as a red-teaming tool, but for a benchmarking workflow the side-by-side output view is the reason to install it.
Where it falls short: The comparison view is stronger than the raw metrics view. Pair it with lm-evaluation-harness or DeepEval for reference scores.
Pricing:
- Free: Full open-source CLI
- Paid: Managed team tier
Platforms: Windows, macOS, Linux (Node.js)
Download: promptfoo.dev
Bottom line: The pick when the question is “does model A or model B answer my ten prompts better”.
6. HELM — Best academic reference
HELM, from Stanford’s CRFM, is the “Holistic Evaluation of Language Models” project. It publishes an authoritative cross-model leaderboard using a wide-scenario benchmark, and the same code runs locally against your own model.
Where it falls short: Heavy. Full HELM runs take hours on a workstation. It is not a laptop tool for a quick sanity check.
Pricing:
- Free: Full open-source project
- Paid: None
Platforms: Windows, macOS, Linux
Download: GitHub
Bottom line: The pick when the numbers need to be defensible in a paper or a technical report.
7. OpenAI Evals — Best community-authored test cases
OpenAI Evals is a framework for writing prompt-level test cases (the community catalogue includes math, logic puzzles, code, and reasoning) and running them against any model behind an OpenAI-compatible endpoint.
Where it falls short: The catalogue quality varies. Some evals are stronger than others.
Pricing:
- Free: Open-source framework
- Paid: None
Platforms: Windows, macOS, Linux
Download: GitHub
Bottom line: The pick when the fastest way to test a model is to lift a community eval, point it at your local endpoint, and read the results.
How to pick the right one
If you want to reproduce published quality scores: lm-evaluation-harness. Set aside disk space and coffee.
If you want to compare throughput on your hardware: llama-bench. It is the fastest way to answer a quant-vs-quant question.
If you are comparing laptops or GPUs: MLPerf Client. Vendor-neutral, apples-to-apples.
If your benchmark is your own dataset: DeepEval. Pytest-style tests for LLMs.
If the goal is a side-by-side view of two models on your prompts: Promptfoo.
If the numbers need academic credibility: HELM. Accept the long run.
If community-authored tests are enough: OpenAI Evals. Lift a directory, point at your model, run.
FAQ
Which local LLM benchmark is the best overall? lm-evaluation-harness for quality reproducibility. llama-bench for speed on your hardware.
Can I benchmark a model running in Ollama or LM Studio? Yes. Every tool on this list supports an OpenAI-compatible endpoint, and Ollama, LM Studio, and Jan all expose one.
What is MMLU and why does everyone quote it? MMLU is a multiple-choice benchmark across many academic subjects. It is quoted because a lot of models publish their score and comparisons are easy to make.
Is a benchmarking app the same as a red-teaming app? No. Benchmarking measures capability and speed. Red teaming stress-tests security. Promptfoo can do both; lm-evaluation-harness only does the first.
Do I need a GPU to run these benchmarks? No. lm-evaluation-harness and llama-bench run on CPU. Speed benchmarks are the ones where a GPU makes the biggest visible difference.
Are these tools free? Every tool on this list has a free tier. Only DeepEval and Promptfoo sell managed observability on top.
