Voice Integration Advanced

Voice transforms AI NPCs from text-based interactions to immersive spoken conversations. This lesson covers the full voice pipeline: capturing player speech with STT, processing it through the dialogue system, and delivering NPC responses with expressive TTS - all with latency low enough for real-time conversation.

The Voice Pipeline

1. Speech-to-Text (STT)

   Capture microphone input and transcribe to text using Whisper, Deepgram, or Azure Speech. Target under 500ms transcription time.

2. LLM Processing

   Pass transcribed text through the dialogue system. Use streaming responses to start TTS before the full response is generated.

3. Text-to-Speech (TTS)

   Convert the NPC's response to audio using ElevenLabs, Azure Neural Voices, or Cartesia. Stream audio chunks to the game engine.

4. Lip Sync

   Drive avatar lip sync from the TTS audio in real-time, using the techniques from the AI Avatars in Unity course.

TTS Provider Comparison

Latency Optimization

Voice Activity Detection

Implement Voice Activity Detection (VAD) to determine when the player starts and stops speaking. This prevents the NPC from interrupting and handles background noise. Use WebRTC VAD or Silero VAD for reliable detection.