Offline Voice Control on ESP32: Wake-Word Detection and Speech Recognition with ESP-SR

Most "voice control" tutorials for the ESP32 are really cloud speech-to-text with extra wiring: the chip streams raw audio to a server, waits for a response, and does something with the result. That works, but it needs WiFi, it needs an internet connection, it leaks audio off the device, and it adds real latency to something that should feel instant. Espressif's own ESP-SR framework does the opposite: wake-word detection and a small fixed vocabulary of command recognition run entirely on the chip, offline, with no cloud round-trip. This guide covers what ESP-SR actually does, which hardware it needs, and how to get a working "Hi ESP" wake word plus a handful of voice commands running without writing a neural network from scratch.

What ESP-SR Actually Is

ESP-SR (Espressif Speech Recognition) is a component library, not a single library call — it bundles a wake-word detection model (WakeNet), a small-footprint command recognition model (MultiNet), acoustic front-end processing (AEC for echo cancellation, beamforming on multi-mic boards, noise suppression), and the glue to run all of it in real time on an ESP32-S3 or ESP32 with enough RAM and a decent microphone. It's built for exactly the "always listening for a trigger word, then a short fixed command" pattern used in smart speakers and appliances, not open-ended dictation — don't expect it to transcribe a sentence the way a phone assistant does.

Hardware You Actually Need

• MCU: ESP32-S3 is the realistic choice — it has the vector instructions and PSRAM headroom the models want. A plain ESP32 can run a single wake word in constrained configurations, but the S3 is what Espressif actively targets and documents.
• PSRAM: 8MB octal PSRAM is effectively required for the fuller command models; the wake-word-only path is lighter but still benefits from it.
• Microphone: An I2S digital MEMS mic like the INMP441 (the same part covered in our ESP32 I2S audio guide) works for single-mic wake-word detection. Espressif's own reference boards (ESP32-S3-Korvo, ESP32-S3-BOX) use dual- or multi-mic arrays with hardware-assisted beamforming for far-field pickup in a noisy room — a single mic on a breadboard will work at desk distance but won't match a smart-speaker's range across a kitchen.

Getting a Wake Word Running

• Set up ESP-IDF (ESP-SR is a managed IDF component, not an Arduino library, though Arduino-as-IDF-component wrappers exist) and add the esp-sr component to your project via idf.py add-dependency.
• Pick a pre-trained wake word from Espressif's shipped model set — "Hi ESP," "Alexa," and a handful of others ship as ready-to-use WakeNet models. Custom wake words require Espressif's paid model-training service or a community-trained alternative; there isn't a simple free path to "train your own word from ten samples" the way there is with some TinyML platforms.
• Wire the I2S mic, configure the AFE (audio front-end) pipeline in your sdkconfig, and run the wake-word example project first before adding command recognition — it isolates microphone and gain problems early, since a mic that's clipping or too quiet will fail wake-word detection in ways that are hard to debug once command recognition is layered on top.
• Once the wake word reliably triggers, add MultiNet for command recognition: a small fixed set of phrases ("turn on the light," "turn off the light") gets compiled into a command list the model matches against after the wake word fires. This is closed-vocabulary recognition, not open dictation — each command needs to exist in the model's command set.

How This Differs from the Rest of the Site's Voice Coverage

If you've read our guide to building a Home Assistant Voice Satellite with a Raspberry Pi, this covers different ground on purpose. The Pi-based voice satellite uses openWakeWord and a full speech pipeline running on a Linux SBC talking to Home Assistant's cloud-free Assist pipeline over the Wyoming protocol — it's a room-scale smart speaker replacement with real dictation. ESP-SR on an ESP32-S3 is the embedded-firmware version of the same idea: a single low-power chip, a fixed small vocabulary, no OS, and a response time measured in tens of milliseconds because there's no network hop at all. Use the Pi/Home Assistant path when you want a general-purpose voice assistant; use ESP-SR when you're building a single-purpose device — a voice-controlled shop light, a "status" query for a 3D printer enclosure, a wake-word-gated BLE trigger — that needs to work with no network dependency at all.

Practical Limits Worth Knowing Up Front

CapabilityESP-SR reality Wake word vocabularyLimited to Espressif's shipped models unless you pay for custom training Command vocabularySmall fixed phrase list compiled at build time, not open dictation Noise robustnessGood with the AFE pipeline and a decent mic; single cheap mic in a noisy room will miss triggers Far-field rangeMulti-mic reference boards do noticeably better than a lone breadboard mic Language supportEnglish and Mandarin wake words are best supported; check current model availability for other languages before committing

ESP-SR is a genuinely different tool from cloud speech APIs, not a downgraded version of one — it trades vocabulary size for privacy, latency, and independence from a network connection, which is exactly the trade a lot of maker projects actually want. A shop light that turns on the instant you say its wake word, with zero cloud dependency and zero audio ever leaving the device, is a better fit for ESP-SR than for anything that needs a WiFi round trip to work at all.