Running a Local LLM Chatbot on Raspberry Pi 5 with Ollama
Running a large language model entirely on a Raspberry Pi sounds like a stretch, but small quantized models have gotten genuinely usable on Pi 5 hardware — not fast, and not a replacement for cloud-hosted models on complex tasks, but capable of running a real offline chatbot, a local documentation assistant, or a privacy-preserving text tool with no internet connection and no API costs. Ollama makes this dramatically simpler than it was even a year ago by handling model download, quantization format, and a local API in one package. This guide covers getting Ollama running on a Pi 5, picking a model that actually fits the hardware, and realistic expectations for performance.
Why Pi 5 (and Not Earlier Models)
The Pi 5's jump to a faster CPU and, critically, LPDDR4X memory with meaningfully higher bandwidth than the Pi 4 is what makes local LLM inference viable at all — memory bandwidth, not raw CPU clock speed, is usually the bottleneck for token generation speed on CPU-only inference. An 8GB Pi 5 is the practical minimum for anything beyond the smallest models; the 4GB model will run 1-2B parameter models but leaves little headroom for the OS and any other services.
Installing Ollama
Ollama publishes an ARM64 build that installs with a single script, the same as on x86 Linux:
curl -fsSL https://ollama.com/install.sh | shThis installs Ollama as a systemd service, so it starts automatically and exposes a local API on localhost:11434. Confirm it's running with systemctl status ollama, and pull your first model once that's confirmed.
Choosing a Model That Fits
Model size is the single most important decision on this hardware — picking a model sized for a desktop GPU will simply be too slow or fail to load. Quantized ("Q4" or similar) small models are the only realistic choice:
ModelApprox. Size (Q4 quantized)Pi 5 8GB Experience Llama 3.2 1B~0.7GBFast, usable for simple Q&A and short conversations; noticeably limited reasoning Llama 3.2 3B~2GBGood balance — several tokens/sec, reasonable coherence for a local assistant Phi-3 Mini (3.8B)~2.3GBStrong for its size on reasoning/coding tasks, similar speed to Llama 3.2 3B Qwen 2.5 7B~4.5GBRuns, but slowly — usable for non-interactive/batch tasks more than live chatPull and run a model with:
ollama pull llama3.2:3b ollama run llama3.2:3bRealistic Performance Expectations
Expect roughly 3–8 tokens per second on a 3B-class model on an 8GB Pi 5, depending on active cooling and whether anything else is competing for memory bandwidth. This is slow compared to a cloud API or a desktop GPU, but entirely usable for asynchronous or low-interactivity use cases: a home-automation voice assistant that doesn't need instant replies, an offline note-summarizer, or a chatbot for a kiosk where a few seconds of "thinking" is acceptable. Active cooling (a fan, not just a passive heatsink) noticeably helps sustain performance during longer generations, since the Pi 5 will thermal-throttle under sustained CPU load.
Building a Simple Local Chat Interface
Ollama exposes a REST API on port 11434 that any script or web frontend can call. A minimal curl-based test:
curl http://localhost:11434/api/generate -d '{ "model": "llama3.2:3b", "prompt": "Explain what a heat-set insert is in one sentence.", "stream": false }'Open WebUI (a community project) is a popular self-hosted chat frontend that talks to Ollama's API and gives you a browser-based ChatGPT-style interface entirely on your local network — worth adding once basic Ollama usage is confirmed working, especially for a Pi you want other household devices to reach.
When This Isn't the Right Tool
Be honest about the limits: complex reasoning, long-context tasks, coding help beyond simple snippets, and anything latency-sensitive will frustrate on this hardware. This setup shines for privacy-sensitive offline use, kiosk/embedded assistants, learning how LLM tooling works without a cloud bill, and situations where "no internet required" is a genuine requirement rather than a nice-to-have.
A Pi 5 running Ollama isn't going to replace a cloud LLM subscription for serious work, but as a genuinely offline, private, always-on local assistant for simple tasks, it's a surprisingly solid and inexpensive way to put a real language model permanently on your home network.