Self-Hosting Whisper and Piper on Raspberry Pi: Local Speech-to-Text and Text-to-Speech for Home Assistant

This site already covers building a Home Assistant voice satellite — the microphone-and-speaker hardware that sits in a room and listens for a wake word. That project assumes something else on the network is doing the actual speech recognition and speech synthesis. This guide covers that other half: running Whisper (speech-to-text) and Piper (text-to-speech) as self-hosted services on a Raspberry Pi, so your entire voice pipeline — from "hey assistant" to a spoken response — stays on your local network with nothing sent to a cloud provider.

Where This Fits in the Pipeline

A local voice assistant chain has distinct stages: wake-word detection (typically openWakeWord, running on the satellite device itself or here on the server), speech-to-text (Whisper, converting your spoken command into text), intent processing (Home Assistant's own conversation engine, matching the text to an action), and text-to-speech (Piper, converting the response back into spoken audio). The Wyoming protocol, developed for this exact purpose, is the lightweight network protocol that lets these pieces run as separate services — potentially on separate machines — and talk to each other and to Home Assistant over the network. This guide is about running the Whisper and Piper services; if you haven't built the satellite hardware yet, that's the companion piece.

Whisper: Choosing an Implementation and Model Size

OpenAI's original Whisper is too heavy to run in real time on a Pi's CPU at any but the smallest model sizes. Two community reimplementations make this practical:

• whisper.cpp: A C/C++ reimplementation with no Python dependency overhead, using quantized models (4-bit and 5-bit quantization options) that trade a small amount of accuracy for a large reduction in memory and CPU use — usually the better choice on a Pi specifically.
• faster-whisper: Built on CTranslate2, generally faster than the reference Python implementation but with more dependency weight than whisper.cpp — a reasonable choice if you're already comfortable in a Python/Docker environment and want easier integration with other Python tooling.

Model SizeApproximate RAMRealistic Latency on Pi 5 (short command)Accuracy tiny~75MBUnder 1 secondAdequate for short, clear commands; struggles with names and less common words base~150MB1-2 secondsNoticeably better accuracy, still fast enough to feel responsive small~500MB3-6 secondsGood accuracy, but the latency starts to feel sluggish for a voice assistant use case

For a Home Assistant voice satellite specifically, where commands are short and vocabulary is somewhat predictable (device names, room names, action verbs), the base model quantized via whisper.cpp is the practical sweet spot on a Pi 5 — small and tiny both exist as fallbacks if you need faster response on a Pi 4 or are running several concurrent satellite streams.

Piper: Fast, Local Text-to-Speech

Piper is built specifically for fast local inference on modest hardware using ONNX Runtime, and it's genuinely fast enough on a Pi CPU that TTS is rarely the bottleneck in this pipeline — the perceived latency is almost always dominated by Whisper's transcription step, not Piper's synthesis. Piper voices are distributed as small model files (tens of megabytes each) covering many languages and several voice qualities per language; low-quality models sound noticeably more robotic but respond faster, medium-quality is the usual balance point for a home assistant that needs to sound reasonably natural without taxing the Pi.

Wiring It Together with Wyoming

Home Assistant OS makes this close to plug-and-play: the Wyoming Whisper and Wyoming Piper add-ons are available directly in the Home Assistant add-on store if you're running HA OS on the same Pi (or a separate one on the network), each exposing a Wyoming protocol endpoint that you point Home Assistant's Assist pipeline configuration at. If you're running Home Assistant Container or Core instead of HA OS, the equivalent is running the wyoming-whisper and wyoming-piper Docker images directly and pointing Home Assistant's integration at their exposed ports — the add-on store versions are simply a packaged wrapper around the same underlying Docker images.

Running the Server Elsewhere on the Network vs. Locally

Nothing requires Whisper and Piper to run on the same Pi as Home Assistant itself, or even on a Pi at all. Because Wyoming is a network protocol, a common and often better pattern is: Home Assistant runs on one Pi (or a NAS, or a mini PC), while Whisper and Piper run on a separate, more powerful machine on the network — even a spare desktop with a GPU, which dramatically improves Whisper transcription speed since CUDA-accelerated Whisper is far faster than CPU inference. If your only available hardware is a single Pi 5 handling Home Assistant, Whisper, Piper, and everything else simultaneously, expect noticeably more latency under concurrent load (someone giving a voice command while an automation is also running) than a dedicated setup, simply from CPU contention.

Realistic Expectations

A fully local voice pipeline on Pi-class hardware will not match the near-instant response of a cloud assistant backed by a data center GPU — a one-to-three second delay between finishing a spoken command and hearing a response is normal and expected with the base Whisper model on a Pi 5. What you gain in exchange is a voice assistant that keeps working with your internet down, sends nothing about what's said in your home to a third party, and costs nothing per query regardless of how much you use it. For many Home Assistant users building a privacy-focused smart home, that trade is the entire point.

Pair this with the Wyoming voice satellite project already on this site and openWakeWord for wake-word detection, and you have a complete local voice assistant chain running entirely on hardware you own, with no subscription and no cloud dependency for the core interaction loop.