Speech Models

What Are Speech Models?

Speech models are AI systems that process and generate spoken language. They fall into two primary categories: Speech-to-Text (STT), also called Automatic Speech Recognition (ASR), which converts spoken audio into written text, and Text-to-Speech (TTS), which generates natural-sounding spoken audio from written text.

Modern speech models have reached a level of quality that makes them nearly indistinguishable from human speech. They power virtual assistants (Siri, Alexa, Google Assistant), real-time transcription services, audiobook narration, accessibility tools, and the rapidly growing voice AI industry.

Speech-to-Text (ASR)

ASR models listen to audio and produce text transcripts. The best modern models handle accents, background noise, multiple speakers, domain-specific jargon, and dozens of languages with remarkable accuracy.

How ASR Works

Modern ASR systems use an encoder-decoder architecture. The encoder converts raw audio waveforms into a sequence of feature representations (typically mel spectrograms), and the decoder generates text tokens from those features. End-to-end models like Whisper handle both steps in a single neural network, replacing the older pipeline approach of separate acoustic model, language model, and pronunciation dictionary.

Key ASR Models and Services

Text-to-Speech (TTS)

TTS models convert written text into natural-sounding speech. The latest generation produces audio so realistic that listeners often cannot distinguish it from a real human voice. Key quality dimensions include naturalness, expressiveness (emotion, emphasis), prosody (rhythm and intonation), and speaker consistency.

Key TTS Models and Services

Voice Cloning and Voice Conversion

Voice cloning creates a synthetic replica of a specific person's voice from a short audio sample. Modern systems need as little as 3-15 seconds of reference audio to produce a convincing clone. This technology powers personalized audiobooks, custom brand voices, and accessibility tools for people who have lost their ability to speak.

Voice conversion transforms one speaker's voice to sound like another in real time - changing pitch, timbre, and speaking style while preserving the original words and timing. This is used in entertainment, dubbing, and privacy-preserving applications.

Speaker Diarization and Identification

Speaker diarization answers the question "who spoke when?" - it segments an audio recording by speaker, labeling each segment as Speaker 1, Speaker 2, etc. This is essential for transcribing meetings, interviews, podcasts, and phone calls where multiple people are speaking.

Speaker identification goes further by matching voice segments to known identities: "This segment was spoken by Alice, this by Bob." It requires pre-enrolled voice profiles but enables personalized experiences and security applications.

How Diarization Works

1. Voice Activity Detection (VAD): Identify which parts of the audio contain speech vs silence.
2. Speaker Embedding: Extract a voice fingerprint (embedding vector) for each speech segment.
3. Clustering: Group segments with similar embeddings into speaker clusters.
4. Labeling: Assign speaker labels to each segment based on cluster membership.

Services like AssemblyAI, Deepgram, and pyannote.audio offer built-in diarization that can handle overlapping speech and distinguish 10+ speakers in a single recording.

Real-Time vs Batch Processing

Real-Time (Streaming)

Audio is processed as it arrives, with results returned within 100-500ms. Essential for live captioning, voice assistants, phone systems, and any interactive application. Streaming ASR sends partial (interim) results that update as more context arrives.

Batch Processing

Entire audio files are uploaded and processed at once. Offers higher accuracy because the model has access to the complete recording context. Best for post-meeting transcription, media archival, podcast processing, and legal/medical documentation where quality matters more than speed.

Use Cases

Transcription Services

Converting meetings, lectures, interviews, and legal proceedings into searchable text. AI transcription is 10-50x cheaper than human transcription and delivers results in minutes instead of days.

Accessibility

Real-time captioning for deaf and hard-of-hearing users. Screen readers powered by TTS for visually impaired users. Voice-controlled interfaces for users with motor disabilities. Speech AI is one of the most impactful accessibility technologies.

Virtual Assistants

Siri, Alexa, Google Assistant, and custom voice bots all depend on ASR to understand user commands and TTS to respond naturally. The quality of the speech pipeline directly determines user satisfaction.

Call Centers

Real-time transcription enables live agent assistance, compliance monitoring, and sentiment analysis during calls. Post-call analytics identify trends, training opportunities, and customer pain points across thousands of conversations.

Content Creation and Dubbing

TTS generates audiobook narration, podcast intros, video voiceovers, and e-learning content at a fraction of the cost of human voice actors. AI dubbing translates and re-voices video content into dozens of languages while preserving the speaker's voice characteristics.

Multilingual Communication

Cascading ASR + translation + TTS enables real-time speech translation - speak English and hear the translation in Japanese. Products like Google Translate, Microsoft Translator, and Meta's SeamlessM4T are making this increasingly seamless.

Code Example: Transcribing Audio with Whisper

OpenAI's Whisper is the most popular open-source ASR model. Here is how to use it locally in Python:

import whisper

# Load the model (tiny, base, small, medium, large-v3)
model = whisper.load_model("base")

# Transcribe an audio file
result = model.transcribe("meeting_recording.mp3")

# Full transcript
print(result["text"])

# Access segments with timestamps
for segment in result["segments"]:
    start = segment["start"]
    end = segment["end"]
    text = segment["text"]
    print(f"[{start:.1f}s - {end:.1f}s] {text}")

# Output:
# [0.0s - 4.2s]  Welcome everyone to the quarterly review.
# [4.2s - 8.7s]  Let's start with the sales numbers.
# [8.7s - 13.1s]  Revenue grew 23% compared to last quarter.

For the OpenAI API (cloud-hosted, no GPU needed):

from openai import OpenAI

client = OpenAI()

# Transcribe using the API
with open("meeting_recording.mp3", "rb") as audio_file:
    transcript = client.audio.transcriptions.create(
        model="whisper-1",
        file=audio_file,
        response_format="verbose_json",
        timestamp_granularities=["segment"]
    )

print(transcript.text)

# Access word-level timestamps
for segment in transcript.segments:
    print(f"[{segment.start:.1f}s] {segment.text}")

Multilingual Speech Models

Speech AI is increasingly multilingual. Key developments include:

• Whisper: Trained on 680,000 hours of multilingual audio, supporting 99+ languages for transcription and translation to English.
• SeamlessM4T (Meta): A single model that handles speech-to-speech, speech-to-text, text-to-speech, and text-to-text translation across nearly 100 languages.
• MMS (Meta): Massively Multilingual Speech - ASR for 1,100+ languages and TTS for 1,100+ languages, covering many low-resource languages previously unsupported.
• Google USM: Universal Speech Model trained on 12 million hours of audio across 300+ languages.

Multilingual models are critical for building inclusive products that serve global audiences. They also enable code-switching - seamlessly handling conversations that mix multiple languages, which is common in multilingual communities.

Summary

• Speech models handle two core tasks: Speech-to-Text (ASR) and Text-to-Speech (TTS).
• Leading ASR models include Whisper (open-source), Deepgram (speed), AssemblyAI (accuracy), and Google/Azure (enterprise).
• Leading TTS models include ElevenLabs (quality), OpenAI TTS (simplicity), Azure Neural TTS (variety), and XTTS (open-source).
• Voice cloning can replicate a voice from seconds of audio - powerful but requires ethical use.
• Speaker diarization identifies who spoke when, essential for meetings and multi-speaker transcription.
• Real-time processing enables live applications; batch processing delivers higher accuracy for offline use.
• Key metrics are WER for ASR and MOS for TTS quality.
• Multilingual models like Whisper, SeamlessM4T, and MMS are making speech AI accessible across 100+ languages.