Intermediate

Voice Interface

Voice is the most natural way to interact with a personal assistant. Learn to add speech recognition, voice synthesis, wake word detection, and natural conversation flow to make your assistant truly hands-free.

The Voice Pipeline

A voice-enabled assistant processes audio through several stages:

1. Wake Word Detection

   A lightweight, always-on model listens for a trigger phrase ("Hey Assistant"). Runs locally with minimal power.

2. Speech-to-Text (STT)

   Converts the user's spoken words into text. Can run on-device or in the cloud.

3. Natural Language Understanding

   The LLM processes the transcribed text, understands intent, and generates a response.

4. Text-to-Speech (TTS)

   Converts the assistant's text response back into natural-sounding speech.

5. Audio Output

   Plays the synthesized speech through speakers or earbuds.

Speech-to-Text Options

Service	Accuracy	Latency	Best For
OpenAI Whisper (API)	Very high	1-3 seconds	General purpose, multilingual
Whisper (local)	High	Varies by hardware	Privacy-focused, offline use
Google Speech-to-Text	Very high	<1 second streaming	Real-time applications, long audio
Deepgram	High	<300ms streaming	Low-latency applications
Web Speech API	Good	Real-time	Browser-based assistants

Text-to-Speech Options

Modern TTS produces remarkably natural-sounding speech:

• OpenAI TTS: High-quality voices with multiple options. Good for general-purpose assistants.
• ElevenLabs: Industry-leading voice quality with custom voice cloning capabilities.
• Google Cloud TTS: Wide language support with WaveNet and Neural2 voices.
• Coqui TTS (open source): Self-hosted option with good quality and full data privacy.
• Edge TTS: Free, fast, and good quality. Available through Microsoft Edge's speech service.

Reducing Latency

Voice conversation feels unnatural when there is a long pause between the user speaking and the assistant responding. Target under 2 seconds total latency:

• Streaming STT: Start processing speech while the user is still talking, not after they finish
• Streaming LLM responses: Start TTS as soon as the first tokens arrive from the LLM, not after the full response
• Streaming TTS: Begin playing audio as it is generated rather than waiting for the full audio file
• Sentence-level chunking: Generate TTS for each sentence independently and play them in sequence
• Pre-warming connections: Keep API connections alive to eliminate connection overhead

Conversation Design

Wake Word Implementation

Wake word detection runs continuously and must be extremely lightweight:

• Porcupine (Picovoice): Cross-platform wake word engine. Supports custom wake words. Very low power consumption.
• Snowboy (open source): Customizable wake word detection that runs entirely on-device.
• Custom approach: Train a small CNN or RNN on spectrograms of your chosen wake word. Models under 1MB work well.