Short answer: voice agents feel slow because most of them run speech-to-text, a language model, and text-to-speech one after another, and every step adds wait time. People leave about 200ms between turns in conversation. Past roughly a second of silence, callers start talking over the agent or hang up.
Where the delay comes from
A typical stacked pipeline looks like this:
| Stage | What happens | Typical cost |
|---|---|---|
| End-of-turn detection | Decide the caller has finished speaking | 200–800ms of waiting for silence |
| Speech-to-text | Finalize the transcript | 100–300ms |
| Language model | Generate the first words of the reply | 200–1,000ms+ |
| Text-to-speech | Produce the first audio | 100–300ms |
| Network and telephony | Carry audio both ways | 50–200ms |
Add those up sequentially and you get the 1.5–2 second pause callers complain about.
How to measure latency honestly
A single "latency" number means little unless you know what it measures. Ask any vendor, including us, three questions:
- Which interval? The one that matters is from the moment the caller stops speaking to the first audio of the reply.
- Which percentile? The median (p50) hides the slow calls. Ask for p90 or p95 as well.
- Where was it measured? On a real phone line, or in a browser next to the data center?
How to make an agent fast
- Stream everything. Start transcribing while the caller speaks, start generating while the transcript finalizes, and start speaking while the reply is still being generated.
- Detect turns semantically, not by silence. Waiting for a fixed silence window adds hundreds of milliseconds to every turn.
- Right-size the model. The largest model is rarely needed for scheduling or FAQs.
- Keep audio close to the caller. Telephony and inference in the same region cut round trips.
- Use fillers sparingly. A short acknowledgement covers a slow tool call, but it does not fix a slow pipeline.
VoilitAI streams every stage end to end so replies land in under 600ms. Ask us for the percentile breakdown on a demo call.
Why agents talk over callers
Talk-over usually has one of three causes:
- End-of-turn detection fires too early when the caller pauses mid-sentence.
- No barge-in handling, so the agent keeps speaking when the caller interrupts.
- Echo from the caller's speakerphone gets heard as new speech.
The fix is turn detection that understands whether a sentence is finished, audio that stops the moment the caller starts speaking, and echo cancellation on the line.
Why agents hallucinate, and how to stop it
Any language model can make things up. Production agents need guardrails:
- Ground answers in your knowledge base instead of the model's memory.
- Use tools for facts that change: prices, availability, order status.
- Write hard rules for what the agent must never promise: refunds, diagnoses, legal advice.
- Test before launch by simulating hundreds of calls, including difficult callers.
- Review transcripts after launch and fix what you find.
Choosing a platform on performance
If you are comparing Vapi, Retell, Bland, Synthflow, or VoilitAI, test them on the same script over a real phone line and compare:
- Reply latency at p50 and p95
- How they handle interruptions
- How often they go off script
- Transfer reliability
- All-in cost per resolved call
Book a demo and run the test on your own script.