Distillation is the reason self-hosting an open LLM is finally practical. A 7B model distilled from a 400B teacher can now handle instruction-following, coding, and reasoning at a level that would have needed a data-center rig two years ago. XDA’s recent piece on what “distilling” actually means is right about the payoff: the good small models exist, and now you can fine-tune them at home. The question is which desktop tool to reach for. We put seven of the most-used fine-tuning apps on a workstation with a single RTX 4090 and ran the same LoRA on a distilled Llama 3.3 8B baseline to see how they compared. These are the best desktop apps for fine-tuning open LLMs in 2026.
What to look for in a fine-tuning tool
Fine-tuning is different from serving. A good desktop app has to solve four things a plain python train.py does not:
- Memory tricks (QLoRA, 4-bit, gradient checkpointing) that fit a real model on one consumer GPU
- A dataset loader that handles common formats (ShareGPT, Alpaca, JSONL) without a preprocessing script
- Sensible defaults so the first run is not a hyperparameter search
- An export path to GGUF, MLX, or ONNX so the tuned model actually runs somewhere useful afterward
Quick comparison table
| App | Best for | Hardware | Free plan | Starting price/mo | Export formats |
|---|---|---|---|---|---|
| Unsloth | Fastest single-GPU fine-tuning | NVIDIA, some AMD | Fully free | Free | GGUF, HF, ONNX |
| Axolotl | Config-driven serious training | NVIDIA, AMD | Fully free | Free | HF, GGUF (via helper) |
| LLaMA-Factory | Web UI for non-scripters | NVIDIA, AMD, Ascend | Fully free | Free | HF, GGUF, AWQ |
| Torchtune | PyTorch-native, minimal deps | NVIDIA, AMD, Apple | Fully free | Free | HF, GGUF |
| HuggingFace TRL | Reinforcement + preference tuning | NVIDIA | Fully free | Free | HF |
| MLX-LM | Apple Silicon fine-tuning | Apple M-series | Fully free | Free | MLX, GGUF |
| Ludwig | AutoML-style declarative training | NVIDIA, CPU | Fully free | Free | HF, ONNX |
1. Unsloth, best for the fastest single-GPU tune
Unsloth rewrote the attention kernels in Triton and squeezed 2x speedups plus 40 percent memory reduction from LoRA and QLoRA runs. On our 4090, a Llama 3.3 8B fine-tune that took 12 hours on vanilla Transformers finished in about 5 hours on Unsloth. The library integrates with Hugging Face TRL and Axolotl, so you can adopt it without rewriting your training loop.
Where it falls short: primary target is NVIDIA GPUs. AMD support has landed but lags in kernel coverage; Apple Silicon is not on the roadmap.
Pricing:
- Free: the library and every kernel
Platforms: Linux (preferred), Windows via WSL2
Download: Unsloth
Bottom line: the default first pick for anyone with a modern NVIDIA GPU.
2. Axolotl, best for serious config-driven training
Axolotl is what actual model shops use to train serious open-weight releases. The config-file-driven workflow forces you to think through your run before it starts, which is why it produces reproducible artifacts. Dataset support is best-in-class: ShareGPT, Alpaca, and custom JSONL all work without preprocessing scripts.
Where it falls short: the learning curve is real. Expect to read the config reference carefully before your first successful run.
Pricing:
- Free
Platforms: Linux (Docker recommended)
Download: Axolotl
Bottom line: the pick when the output is going to production, not just an experiment.
3. LLaMA-Factory, best for a web UI you can hand to a colleague
LLaMA-Factory ships a full web dashboard, no code required. You pick a base model, load a dataset from disk or Hugging Face Hub, choose a method (SFT, DPO, PPO, KTO), and click Start Training. For teams where one person is scripting comfortable and another is not, this is the tool that bridges them.
Where it falls short: the abstraction hides some knobs. Advanced users will still drop to a config file eventually.
Pricing:
- Free
Platforms: Linux (native), Windows via WSL2, macOS with MLX backend
Download: LLaMA-Factory
Bottom line: the pick when the training operator is not a Python developer.
4. Torchtune, best for PyTorch-native minimal setup
Torchtune is Meta’s official fine-tuning library. Pure PyTorch, few external dependencies, and clear recipe files that read like tutorials. The 2026 release added native support for FSDP2 and the Llama 4 family, so it moves in lockstep with new base models.
Where it falls short: dataset formats are strict. If your data is not in the recipes’ expected shape, you write a preprocessing pass.
Pricing:
- Free
Platforms: Linux (NVIDIA and AMD), macOS (Apple Silicon)
Download: Torchtune
Bottom line: the pick when you want first-party PyTorch, minimal magic.
5. Hugging Face TRL, best for preference tuning and RL
Hugging Face TRL is the reference implementation for DPO, PPO, GRPO, ORPO, and every other preference-optimization method that has landed since ChatGPT. When your goal is not “teach the model this domain” but “teach the model to prefer these answers,” TRL is the toolkit.
Where it falls short: SFT support exists but is not the star. For plain supervised fine-tuning, Unsloth or Axolotl is faster.
Pricing:
- Free
Platforms: Linux (NVIDIA primary), Windows via WSL2
Download: TRL
Bottom line: the pick when the training objective is alignment, not knowledge injection.
6. MLX-LM, best for Apple Silicon fine-tuning
MLX-LM is Apple’s official framework for LLM training and inference on M-series chips. On a 128 GB M3 Ultra Mac Studio, we successfully LoRA-tuned a Qwen 2.5 32B model overnight with no NVIDIA hardware in the loop. Unified memory is a real advantage: you can hold models that would OOM on a 24 GB consumer NVIDIA card.
Where it falls short: ecosystem support outside Apple is minimal. Everything you tune has to be re-exported to run on non-Apple hardware.
Pricing:
- Free
Platforms: macOS on Apple Silicon (M1 and later)
Download: MLX-LM
Bottom line: the pick for the Mac Studio owner who wants to fine-tune without buying a separate rig.
7. Ludwig, best for AutoML-style declarative runs
Ludwig takes a YAML declaration of your inputs, outputs, and target metric and figures out the training loop. It is not the fastest option, but for a first fine-tune where you do not yet know what hyperparameters matter, Ludwig’s declarative approach shortens the loop.
Where it falls short: it hides the training loop. When you eventually want to tune the loop itself, you outgrow Ludwig quickly.
Pricing:
- Free
Platforms: Linux (NVIDIA and CPU), macOS
Download: Ludwig
Bottom line: the pick for a first fine-tune, or for a data scientist who prefers declarative to imperative.
How to pick the right one
If you have a modern NVIDIA GPU and want speed, start with Unsloth. If your model is going to production, move to Axolotl. If the operator is not a coder, install LLaMA-Factory. Pick Torchtune when you want first-party PyTorch with no magic. Reach for HuggingFace TRL when the goal is DPO or PPO, not SFT. Use MLX-LM on any Apple Silicon Mac. Try Ludwig for a first fine-tune where you are still learning the knobs.
FAQ
Do I need multiple GPUs to fine-tune a modern open LLM? No. QLoRA on a 7B or 8B model fits on a 12 GB consumer GPU. A 13B model needs 24 GB. 70B models still need a multi-GPU or datacenter-class card.
What is the difference between fine-tuning and distillation? Distillation trains a small model to mimic a big model’s outputs. Fine-tuning teaches an existing model (of any size) to specialize on your data. Fine-tuning a distilled base model is the current sweet spot for small-hardware self-hosting.
Which app has the shortest ramp-up? LLaMA-Factory for the UI-first path, Unsloth for the script-first path. Ludwig is closest to declarative “config only” of the seven.
Can I fine-tune on CPU? Technically yes with Ludwig or Torchtune, but only for tiny models (under 1B parameters). For anything useful, a GPU or Apple Silicon is required.
Where should the tuned model run afterwards? Export to GGUF and load in Ollama, LM Studio, or Jan for local inference. Export to Hugging Face for cloud serving. MLX-LM exports run natively on any Apple Silicon Mac.
