Speaker Diarization Advanced

Speaker diarization answers the question "who spoke when?" in multi-speaker audio. Combined with speech-to-text, it produces transcripts that attribute each utterance to a specific speaker - essential for meeting transcription, interviews, and call analytics.

How Speaker Diarization Works

  1. Voice Activity Detection (VAD)

    Detect which segments of audio contain speech and which are silence or noise.

  2. Speaker Embedding Extraction

    For each speech segment, extract a vector (embedding) that represents the speaker's voice characteristics.

  3. Clustering

    Group segments with similar embeddings together. Each cluster represents one speaker.

  4. Label Assignment

    Assign speaker labels (Speaker 1, Speaker 2, etc.) to each segment based on clustering results.

Using pyannote.audio

pyannote.audio is the leading open-source speaker diarization toolkit, built on PyTorch:

Bash 
# Install pyannote.audio
pip install pyannote.audio
Python 
from pyannote.audio import Pipeline

# Load pretrained diarization pipeline
# Requires a Hugging Face token (free)
pipeline = Pipeline.from_pretrained(
    "pyannote/speaker-diarization-3.1",
    use_auth_token="YOUR_HF_TOKEN"
)

# Run diarization on audio file
diarization = pipeline("meeting.wav")

# Print speaker segments
for turn, _, speaker in diarization.itertracks(yield_label=True):
    print(f"[{turn.start:.1f}s - {turn.end:.1f}s] {speaker}")

Combining Whisper + pyannote.audio

The most powerful approach is to combine Whisper's transcription with pyannote's diarization to produce speaker-attributed transcripts:

Python 
from pyannote.audio import Pipeline
from faster_whisper import WhisperModel

# Step 1: Run diarization
diarization_pipeline = Pipeline.from_pretrained(
    "pyannote/speaker-diarization-3.1",
    use_auth_token="YOUR_HF_TOKEN"
)
diarization = diarization_pipeline("meeting.wav")

# Step 2: Run transcription
whisper_model = WhisperModel("medium", compute_type="int8")
segments, _ = whisper_model.transcribe("meeting.wav")
segments = list(segments)

# Step 3: Align transcription with speaker labels
def get_speaker(midpoint, diarization):
    for turn, _, speaker in diarization.itertracks(yield_label=True):
        if turn.start <= midpoint <= turn.end:
            return speaker
    return "Unknown"

for seg in segments:
    midpoint = (seg.start + seg.end) / 2
    speaker = get_speaker(midpoint, diarization)
    print(f"[{speaker}] {seg.text}")

Cloud-Based Diarization

All major cloud STT services include built-in diarization:

When to use cloud diarization: Cloud diarization is convenient when you are already using a cloud STT service. For maximum control and offline use, pyannote.audio with Whisper gives you the best results and runs entirely on your hardware.
Service Max Speakers How to Enable
Google Cloud 6 (default) enable_speaker_diarization=True in config
Azure Speech 36 Use Conversation Transcription API
AWS Transcribe 10 ShowSpeakerLabels=True in settings

Try It Yourself

Record a conversation with 2-3 speakers (or find a podcast clip) and run the Whisper + pyannote.audio pipeline to produce a speaker-attributed transcript.

Next: Best Practices →
← Real-Time Transcription Best Practices →

Ready to Go Deeper?

Live instructor-led courses from our partners. Affiliate disclosure.

🧠

AI & ML Courses - 30% Off

Live instructor-led AI, machine learning, data science, and cloud courses for working professionals. Use code Limited30 at checkout.

Edureka
📊

DataCamp - AI & Data Science

Hands-on Python, machine learning, and AI courses with interactive exercises and real projects.

DataCamp
🎓

edX - Top AI Courses

University-level AI courses from MIT, Harvard, Stanford. Earn certificates that employers recognize.

edX