XDA turned an old phone into a local LLM server and got Gemma 4 running well enough for real productivity work. That is a pretty good use for an old Android sitting in a drawer, and thanks to llama.cpp’s ARM optimisations, a Snapdragon 8 Gen 1 or newer can host a 3B to 7B quantised model that answers other devices on your Wi-Fi. Seven apps handle the ground, some are chat clients that also expose an OpenAI-compatible API, some are terminals that let you run llama.cpp or ollama directly.
What to look for in a local LLM Android app
- API server mode. For “old phone as LLM server,” the app must expose an HTTP endpoint other devices can call, ideally OpenAI-compatible.
- Model format support. GGUF via llama.cpp is the standard, MLC’s own format is another option.
- GPU offload. Vulkan or OpenCL backend that uses the phone’s Adreno or Mali GPU changes tokens per second by 3x to 5x.
- Battery and thermal control. Sustained inference cooks a phone. Look for foreground service controls and thermal throttling.
- RAM footprint. Older phones with 4 to 6 GB RAM cannot fit larger models even at aggressive quantisation.
- Background persistence. The app must survive Android’s aggressive memory manager on cheap OEMs.
Quick comparison
| App | Best for | Free | API server | Rating |
|---|---|---|---|---|
| Termux | Full llama.cpp or ollama in a terminal | Yes | Yes, via llama-server | 4.7 |
| PocketPal AI | GUI chat with server toggle | Yes | Yes, experimental | 4.5 |
| Layla | Polished chat plus role-play | Yes | Trial Pro API | 4.4 |
| MLC Chat | MLC-optimised models | Yes | Local only in stable | 4.3 |
| ChatterUI | SillyTavern-like interface | Yes | Yes, via built-in server | 4.4 |
| SmolChat | Minimal chat client | Yes | No | 4.3 |
| llama.cpp Android | Reference build | Yes | Yes, via llama-server | 4.5 |
The seven apps
1. Termux, best full llama.cpp or ollama in a terminal
Termux is the Linux-in-your-pocket toolkit that lets you build and run llama.cpp or ollama directly. pkg install cmake python git, clone llama.cpp, compile with LLAMA_VULKAN=1, and start llama-server bound to your LAN IP. The old phone now serves an OpenAI-compatible API to any device on Wi-Fi.
Where it falls short: Not for the terminal-averse. First setup takes an hour of package management.
Pricing:
- Free: Full toolkit
- Paid: None
Platforms: Android
Termux for local LLM hosting wins on flexibility. Anything llama.cpp supports, Termux supports.
Bottom line: The pick for readers comfortable in a terminal. Ceiling is much higher than any GUI on this list.
2. PocketPal AI, best GUI with server toggle
PocketPal AI is an open source Android chat client that loads GGUF models from Hugging Face and runs them on device. Recent builds add an experimental server toggle that exposes the same model on your LAN via an OpenAI-compatible endpoint.
Where it falls short: Server toggle is still marked experimental and lacks TLS.
Pricing:
- Free: Open source under MIT
- Paid: None
Platforms: Android, iOS
Download: Google Play · F-Droid
PocketPal AI for local LLM hosting wins on GUI simplicity. Pick a model from Hugging Face, tap load, start chatting.
Bottom line: The pick for readers who want a GUI first and API second.
3. Layla, best polished chat plus role-play
Layla is a paid-friendly Android chat client that runs quantised models on device with a slick UI. Character role-play, persistent memory, and voice mode included. Pro tier adds an API server mode that exposes the loaded model.
Where it falls short: Free tier caps context length. Character features add UI complexity.
Pricing:
- Free: Basic chat
- Paid: Pro subscription for extended context and API mode
Platforms: Android
Download: Google Play
Layla for local LLM hosting wins on chat polish and role-play tooling.
Bottom line: The pick for readers who want a chat-focused UI, not a dev tool.
4. MLC Chat, best MLC-optimised models
MLC Chat by the MLC team ships pre-compiled models tuned for Qualcomm Adreno and MediaTek GPUs. Tokens per second are the best on this list for phones the app officially supports, at the cost of a smaller model library than llama.cpp.
Where it falls short: API server mode is on the roadmap, not in the stable release yet.
Pricing:
- Free: Open source
- Paid: None
Platforms: Android, iOS
MLC Chat for local LLM hosting wins on raw inference speed when your phone is on the supported list.
Bottom line: The pick for readers whose device is on MLC’s supported list and want maximum tokens per second locally.
5. ChatterUI, best SillyTavern-like interface
ChatterUI is an Android chat client modelled on SillyTavern. Character cards, chat branches, and a built-in server that other devices on your LAN can hit as an OpenAI-compatible endpoint.
Where it falls short: The character card format is the point, but it makes ChatterUI heavier than PocketPal for straight Q&A.
Pricing:
- Free: Open source
- Paid: None
Platforms: Android
Download: GitHub
ChatterUI for local LLM hosting wins if your existing chat stack is SillyTavern-shaped.
Bottom line: The pick for readers who already use SillyTavern.
6. SmolChat, best minimal chat client
SmolChat is a minimalist Android client for llama.cpp GGUF models. It does not serve an API, but its footprint is small enough that older phones with 4 GB RAM can still fit a 3B quantised model.
Where it falls short: Local chat only, no server mode.
Pricing:
- Free: Open source
- Paid: None
Platforms: Android
Download: GitHub
SmolChat for local LLM hosting wins as the lightest option for 4 GB devices.
Bottom line: The pick when the target phone is genuinely old and RAM-constrained.
7. llama.cpp Android, best reference build
llama.cpp Android is the official reference build from ggerganov. Not on any store, you sideload the APK from GitHub releases. Runs llama-server directly with the same flags you would use on Linux.
Where it falls short: Zero UI polish, and you must know llama.cpp’s flags before you start.
Pricing:
- Free: Open source under MIT
- Paid: None
Platforms: Android
Download: GitHub
llama.cpp Android for local LLM hosting wins as the reference implementation.
Bottom line: The pick for readers who want the same tool they run on a laptop, on Android.
How to pick the right one
- Comfortable in a terminal, want maximum flexibility: Termux
- Want GUI plus an API toggle: PocketPal AI
- Want chat polish with role-play: Layla
- Phone is on MLC’s supported list and you want max speed: MLC Chat
- Already run SillyTavern: ChatterUI
- Phone has only 4 GB RAM: SmolChat
- Want the reference llama.cpp build: llama.cpp Android
FAQ
Can an old Android phone really run an LLM?
Yes, on Snapdragon 855 and later, with 6 GB or more RAM. Expect 3 to 5 tokens per second on a 3B quantised model on older hardware, and 10 to 20 tokens per second on more recent phones.
Which model runs best on Android?
Gemma 2B, Phi-3 Mini, Llama 3.2 3B, and Qwen 2.5 3B all fit comfortably at Q4_K_M quantisation. Gemma 4 runs on newer phones with 8 GB or more RAM.
Can I use my phone as an LLM API server for my laptop?
Yes. Termux running llama-server bound to your LAN IP works. Point your laptop’s chat client at http://<phone-ip>:8080/v1 for the OpenAI-compatible endpoint.
Will running an LLM kill my phone battery?
Sustained inference drains fast, plan for wall power. Also expect the phone to run warm, elevate it for airflow.
Is llama.cpp free?
Yes, MIT licensed. Every app on this list that uses it is either open source or paid on top of the free llama.cpp base.
