Running a Local LLM Chatbot on Raspberry Pi 5 with Ollama

Running a large language model entirely on a Raspberry Pi sounds like a stretch, but small quantized models have gotten genuinely usable on Pi 5 hardware — not fast, and not a replacement for cloud-hosted models on complex tasks, but capable of running a real offline chatbot, a local documentation assistant, or a privacy-preserving text tool with no internet connection and no API costs. Ollama makes this dramatically simpler than it was even a year ago by handling model download, quantization format, and a local API in one package. This guide covers getting Ollama running on a Pi 5, picking a model that actually fits the hardware, and realistic expectations for performance.

Why Pi 5 (and Not Earlier Models)

The Pi 5's jump to a faster CPU and, critically, LPDDR4X memory with meaningfully higher bandwidth than the Pi 4 is what makes local LLM inference viable at all — memory bandwidth, not raw CPU clock speed, is usually the bottleneck for token generation speed on CPU-only inference. An 8GB Pi 5 is the practical minimum for anything beyond the smallest models; the 4GB model will run 1-2B parameter models but leaves little headroom for the OS and any other services.

Installing Ollama

Ollama publishes an ARM64 build that installs with a single script, the same as on x86 Linux:

curl -fsSL https://ollama.com/install.sh | sh

This installs Ollama as a systemd service, so it starts automatically and exposes a local API on localhost:11434. Confirm it's running with systemctl status ollama, and pull your first model once that's confirmed.

Choosing a Model That Fits

Model size is the single most important decision on this hardware — picking a model sized for a desktop GPU will simply be too slow or fail to load. Quantized ("Q4" or similar) small models are the only realistic choice:

ModelApprox. Size (Q4 quantized)Pi 5 8GB Experience Llama 3.2 1B~0.7GBFast, usable for simple Q&A and short conversations; noticeably limited reasoning Llama 3.2 3B~2GBGood balance — several tokens/sec, reasonable coherence for a local assistant Phi-3 Mini (3.8B)~2.3GBStrong for its size on reasoning/coding tasks, similar speed to Llama 3.2 3B Qwen 2.5 7B~4.5GBRuns, but slowly — usable for non-interactive/batch tasks more than live chat

Pull and run a model with:

ollama pull llama3.2:3b ollama run llama3.2:3b

Realistic Performance Expectations

Expect roughly 3–8 tokens per second on a 3B-class model on an 8GB Pi 5, depending on active cooling and whether anything else is competing for memory bandwidth. This is slow compared to a cloud API or a desktop GPU, but entirely usable for asynchronous or low-interactivity use cases: a home-automation voice assistant that doesn't need instant replies, an offline note-summarizer, or a chatbot for a kiosk where a few seconds of "thinking" is acceptable. Active cooling (a fan, not just a passive heatsink) noticeably helps sustain performance during longer generations, since the Pi 5 will thermal-throttle under sustained CPU load.

Building a Simple Local Chat Interface

Ollama exposes a REST API on port 11434 that any script or web frontend can call. A minimal curl-based test:

curl http://localhost:11434/api/generate -d '{ "model": "llama3.2:3b", "prompt": "Explain what a heat-set insert is in one sentence.", "stream": false }'

Open WebUI (a community project) is a popular self-hosted chat frontend that talks to Ollama's API and gives you a browser-based ChatGPT-style interface entirely on your local network — worth adding once basic Ollama usage is confirmed working, especially for a Pi you want other household devices to reach.

When This Isn't the Right Tool

Be honest about the limits: complex reasoning, long-context tasks, coding help beyond simple snippets, and anything latency-sensitive will frustrate on this hardware. This setup shines for privacy-sensitive offline use, kiosk/embedded assistants, learning how LLM tooling works without a cloud bill, and situations where "no internet required" is a genuine requirement rather than a nice-to-have.

A Pi 5 running Ollama isn't going to replace a cloud LLM subscription for serious work, but as a genuinely offline, private, always-on local assistant for simple tasks, it's a surprisingly solid and inexpensive way to put a real language model permanently on your home network.