Private Voice Assistant
Table of Contents
What if your AI assistant could hear you across the room and respond with a human voice—without ever touching the cloud?
A Private Voice Assistant
Every major voice assistant—Alexa, Siri, Google Home—works the same way: your voice travels to distant servers, gets processed, and a response comes back. Your conversations, your questions, your habits—all stored somewhere you don’t control.
I wanted a privacy-preserving option, a voice assistant that:
- Never sends data to the cloud
- Runs entirely on hardware I own
So I built one. On a Raspberry Pi 5. And here’s how the voice pipeline works. But before that:
GÖD’S GATE
👉 I named my AI assistant Prometheus, after one of the characters of my sci-fi epic GÖD’S GATE. Check it out! 👾📚
Run it yourself?
If you want to run it yourself, clone this GitHub repo and follow the setup.
Donations?
You can also toss me a coin: Paypal
The Architecture
The voice system consists of four Docker containers working in harmony:
┌────────────────┐ ┌──────────────────┐ ┌─────────┐
│ 🎤 "Hey Jarvis"│ ──▶ │ openWakeWord │────▶│ Whisper │
│ (Wake Word) │ │ (Detection) │ │ (STT) │
└────────────────┘ └──────────────────┘ └─────────┘
│
▼
┌─────────┐ ┌────────┐ ┌────────────┐
│ Ollama │───▶│ Gemma │───▶│ Piper TTS │───▶ 🔊
│ (API) │ │ (LLM) │ │ (Voice) │
└─────────┘ └────────┘ └────────────┘
The flow is simple:
- You say “Hey Jarvis”
- openWakeWord detects the trigger phrase
- Whisper transcribes your question
- The AI generates a response
- Piper speaks the answer aloud
All of this happens locally. Your voice never leaves your network.
Component 1: Wake Word Detection (openWakeWord)
The Problem: How do you know when someone is talking to you versus having a conversation nearby?
The Solution: A lightweight neural network that continuously listens for a specific phrase—the “wake word.”
┌─────────────────────────────────────────────────────────────────┐
│ Wake Word Detection │
├─────────────────────────────────────────────────────────────────┤
│ │
│ 🎤 Microphone ──▶ openWakeWord ──▶ "Hey Jarvis" detected! │
│ │ │
│ ▼ │
│ Start Recording │
│ │
└─────────────────────────────────────────────────────────────────┘
Why openWakeWord?
- Runs on CPU (no GPU needed)
- Pre-trained models available (“Hey Jarvis”, “Alexa”, etc.)
- Low latency detection (~100ms)
- Open source and privacy-preserving
The wake word acts as a gatekeeper. Until it hears the magic phrase, any picked up sound is discarded. This means your Pi isn’t transcribing every conversation in the room—just the ones directed at it.
Component 2: Speech-to-Text (Whisper)
The Problem: Once activated, how do you convert spoken words into text the AI can understand?
The Solution: OpenAI’s Whisper model, running locally via the faster-whisper implementation.
┌──────────────────────────────────────────────────────────────────────────┐
│ WHISPER STT FLOW │
├──────────────────────────────────────────────────────────────────────────┤
│ │
│ Audio Input ──▶ Whisper (tiny) ──▶ "What's the weather like today?" │
│ (5 seconds) (2 seconds) (Text output) │
│ │
└──────────────────────────────────────────────────────────────────────────┘
Key decisions:
- Model size: tiny — At 39MB, it’s fast enough for real-time use on Pi
- Compute type: int8 — Quantized for efficient CPU inference
- Sample rate: 16kHz — Optimal for speech recognition
Whisper supports 99 languages and handles accents, background noise, and natural speech patterns remarkably well—even the tiny model.
Component 3: The AI Brain (Ollama + Gemma)
The Problem: You have your request in text format, but how do you get an intelligent answer from an AI?
The Solution: Google’s Gemma-2-2b model running through Ollama, wrapped in a FastAPI service.
This is the same AI that powers the text chat. The voice pipeline simply feeds it transcribed text and receives a response. The AI doesn’t know (or care) whether the input came from WhatsApp, a web portal, or a microphone.
Why this matters: One AI, multiple interfaces. The voice system is just another way to interact with the same intelligence.
Component 4: Text-to-Speech (Piper)
The Problem: Once you have the AI response in text, how do you make a computer sound human as it speaks it aloud?
The Solution: Piper TTS—a neural text-to-speech engine that generates natural-sounding speech entirely offline.
┌──────────────────────────────────────────────────────────────────────────┐
│ PIPER TTS INTERNAL FLOW │
├──────────────────────────────────────────────────────────────────────────┤
│ │
│ 1. TEXT INPUT │
│ "Hello, I am your AI assistant" │
│ │ │
│ ▼ │
│ 2. PHONEMIZATION │
│ Convert text to phonemes (speech sounds) │
│ "HH AH L OW , AY AE M Y AO R EY AY AH S IH S T AH N T" │
│ │ │
│ ▼ │
│ 3. NEURAL NETWORK (VITS Model) │
│ Generate audio waveform from phonemes │
│ │ │
│ ▼ │
│ 4. AUDIO OUTPUT │
│ Raw PCM audio ──▶ USB Speaker ──▶ 🔊 │
│ │
└──────────────────────────────────────────────────────────────────────────┘
Voice options:
- American male (Ryan) — neutral and informative
- British female (Alba) — clear and professional
- American female (Amy) — warm and conversational
- …
The voice model is about 60MB and generates speech in real-time on the Pi 5.
The Complete Pipeline
When you say “Hey Jarvis, what time is it?”, here’s what happens:
┌─────────────────────────────────────────────────────────────────────────┐
│ VOICE PIPELINE │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ You: "Hey Jarvis" │
│ │ │
│ ▼ │
│ ┌─────────────┐ │
│ │ openWakeWord │ ──▶ Wake word detected! │
│ └─────────────┘ │
│ │ │
│ ▼ │
│ You: "What time is it?" │
│ │ │
│ ▼ │
│ ┌─────────────┐ │
│ │ Whisper STT │ ──▶ "What time is it?" │
│ └─────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────┐ │
│ │ Ollama │ ──▶ "I don't have access to the current time..." (yet) │
│ └─────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────┐ │
│ │ Piper TTS │ ──▶ 🔊 Speaks the response │
│ └─────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────┘
Total latency: About 5-8 seconds from wake word to spoken response. Not instant, but entirely private.
Hardware Requirements
| Component | Purpose | Notes |
|---|---|---|
| Raspberry Pi 5 (8GB) | Runs everything | 4GB works but 8GB recommended |
| USB Microphone | Voice input | Any USB mic works |
| USB Speaker | Voice output | Any USB speaker works |
The Pi’s built-in audio is notoriously problematic. USB audio devices are plug-and-play and far more reliable.
Two Ways to Trigger Speech
The voice system offers flexibility:
1. Wake Word (Hands-Free)
You: "Hey Jarvis, tell me a joke"
Pi: *listens* *thinks* *speaks the response*
2. Text Trigger (Via Chat)
You (WhatsApp): "speak tell me a joke"
Pi: *sends text reply* *speaks the same response aloud*
The “speak” keyword works in both WhatsApp or any other portal you have created (see my previous post about using WhatsApp to speak with the AI).
Commanding the AI to speak from your text-based interface is more of a gimmick, but it’s useful to play pranks on your guests.
What I Learned
1. USB audio is essential. The Pi’s built-in audio is limited and finicky. USB devices just work.
2. Sample rates matter. Mismatched audio formats cause silent failures. The plughw: ALSA plugin handles conversion automatically.
3. Wake word training is hard. I wanted “Hey Prometheus” but training a custom wake word requires Google Colab, and I simply do not want to use Google Colab—I want to do things on my machine. For some reason, the code to train locally was not available in the openWakeWord repo, and my attempts to recreate it failed too often. So I settled for the pre-trained “Hey Jarvis” which works out of the box.
4. Latency is acceptable. 5-8 seconds isn’t instant, but for a completely local, private system running on a $120 computer, it’s remarkable.
5. Docker simplifies everything. Each component runs in its own container with its own dependencies. Updates are clean, rollbacks are easy.
The Privacy Payoff
When you use a commercial voice assistant:
- Your voice is recorded and sent to servers
- It may be reviewed by humans for “quality improvement”
- Your request history is stored indefinitely (potentially)
- Your data trains models you don’t control
When you use this system:
- Your voice never leaves your home network
- No recordings are stored
- No data is collected
- You own the hardware, the software, and the conversation
That’s the trade-off: a few seconds of latency for complete privacy.
Glossary
| Term | Definition |
|---|---|
| ALSA | Advanced Linux Sound Architecture—the standard Linux audio system that manages sound cards and audio routing |
| Docker | Platform for running applications in isolated containers with their own dependencies |
| faster-whisper | Optimized implementation of Whisper that runs 4x faster using CTranslate2 |
| Gemma | Google’s open-source large language model, available in 2B and 7B parameter versions |
| int8 quantization | Reducing model precision from 32-bit to 8-bit, dramatically reducing memory usage with minimal accuracy loss |
| LLM | Large Language Model—AI systems trained on text that can understand and generate human language |
| Ollama | Tool for running LLMs locally with a simple API |
| ONNX | Open Neural Network Exchange—a standard format for machine learning models |
| openWakeWord | Open-source wake word detection library for triggering voice assistants |
| PCM | Pulse Code Modulation—the standard format for uncompressed digital audio |
| Phoneme | The smallest unit of sound in speech (e.g., the “p” sound in “pot”) |
| Piper TTS | Fast, local text-to-speech engine using neural networks |
| plughw: | ALSA plugin device that automatically converts audio formats between applications and hardware |
| Sample Rate | How many audio samples are recorded per second (16kHz = 16,000 samples/second) |
| STT | Speech-to-Text—converting spoken audio into written text |
| TTS | Text-to-Speech—converting written text into spoken audio |
| VITS | Variational Inference with adversarial learning for end-to-end Text-to-Speech—the neural architecture Piper uses |
| Wake Word | A specific phrase that activates a voice assistant (e.g., “Hey Jarvis”, “Alexa”) |
| Whisper | OpenAI’s open-source speech recognition model, trained on 680,000 hours of audio |
This is part of my ongoing project to build a completely private, self-hosted AI assistant on a Raspberry Pi. The full code is available on GitHub.
Disclaimer
Any views expressed in this post are solely those of the author and do not represent the opinions or policies of any affiliated organizations.
