Voice agents

A voice agent is a text agent with three new constraints: the person can interrupt, they cannot see anything, and a pause over a second feels broken. Most voice failures are latency and turn-taking failures, not language failures.

How it works

Transcribe incrementally, and act on the pause rather than waiting for a full stop.
Keep responses short. Nobody can hold a five sentence spoken answer.
Support interruption at any moment, and stop speaking instantly when it happens.
Confirm anything consequential by repeating it back before doing it.
Have an obvious path to a human, and take it early rather than after three failed attempts.

Optional: load a real model

See it work

When it pays, and when it does not

Use it when	Skip it when
Callers are already phoning you	Everyone would rather use an app, where voice adds nothing
The task is short and transactional	The task needs a form, a document, or a comparison
You can respond in under a second	Your pipeline takes three seconds, which feels broken on a call
Handoff to a human is instant	Transfers are slow, which turns a small failure into an angry caller

How it fails

Latency

The single biggest factor in whether a voice agent feels usable. Optimise the pipeline before the prompt.

No interruption handling

An agent that keeps talking over a caller is worse than a menu system.

Overconfident transcription

Names, addresses, and numbers get transcribed wrong. Repeat them back before acting.

What it costs

Latency budget	Speech to text, model, text to speech, all under a second. That budget shapes every other choice.
Cost per minute	Higher than text, and calls are long.
Failure visibility	A bad call is invisible unless you record and sample. Do both.

Design the handoff first. On a phone call there is no undo and no back button, and a fast transfer to a person is the feature that makes the rest survivable.

Related: Routing · Human in the loop · Ticket triage · all patterns · agent jobs

Free from AI School - no signup, everything runs in your browser.