Voice agent latency on real phone calls

The finding

On the phone, the gap narrows and the order shifts

Connected straight to each vendor’s realtime endpoint, medians ranged from 0.41 s to 1.15 s, nearly a 2.8× spread. Put the same agents behind a real phone call, timed from the call recording the way a caller hears it, and every fully tested vendor landed between 1.04 s and 1.56 s, a 1.5× spread. The phone network, SIP hand-off and turn detection added between about 0.3 and 0.7 s to each vendor’s median. That narrowed the differences without erasing them, and it changed the order: Retell AI went from third over the browser connection to tied fastest on the phone.

What this means if you’re choosing: a vendor’s browser demo speed doesn’t tell you how it feels on a phone line. On these defaults, ElevenLabs Agents and Retell AI felt quickest on the phone, and Vapi was about half a second slower.

ElevenLabs Agents

0.41 s

1.04 s

Ultravox

0.49 s

1.18 s

Retell AI

0.76 s

1.04 s

Vapi

1.15 s

1.56 s

Browser connection (vendor’s realtime endpoint)Real phone call (Twilio → SIP)

Tier B · real phone calls

Real phone calls (Twilio to SIP)

Twilio placed each call to the vendor’s SIP address and recorded both sides. Includes both network legs and turn detection, which is what a caller actually waits through.

Real phone call results per vendor

Measure	Vapi	Retell AI	ElevenLabs Agents	Ultravox
Median response timep50, end of caller speech to first agent audio	1.56 s	1.04 s	1.04 s	1.18 s
90th percentilep90: 9 in 10 replies were faster	1.83 s	1.22 s	1.34 s	1.55 s
Fastest to slowest	-0.26 s – 2.58 s	0.76 s – 3.62 s	0.84 s – 1.86 s	1.06 s – 4.66 s
Calls / turns timed	11 calls · 34 turns	11 calls · 33 turns	11 calls · 33 turns	11 calls · 33 turns
Talked over the caller	0 turns	0 turns	0 turns	0 turns
Transcription error rateWER of the vendor’s own transcript	2.2%321 words	0.0%319 words	0.6%319 words	6.9%319 words

Tier A · browser connection

Direct realtime connection (no phone network)

A scripted caller connected to each vendor’s own realtime endpoint (the kind of connection a web demo uses) and streamed the same three caller lines at real-time pace. Caller audio was clean synthetic speech, so the transcription numbers here are a best case.

Browser connection results per vendor

Measure	ElevenLabs Agents	Ultravox	Retell AI	Vapi
Median response timep50, end of caller speech to first agent audio	0.41 s	0.49 s	0.76 s	1.15 s
90th percentilep90: 9 in 10 replies were faster	0.49 s	0.64 s	1.09 s	1.55 s
Fastest to slowest	0.32 s – 0.89 s	0.38 s – 0.84 s	0.46 s – 1.51 s	0.79 s – 2.08 s
Calls / turns timed	10 calls · 30 turns	11 calls · 32 turns	10 calls · 30 turns	10 calls · 30 turns
Talked over the caller	0 turns	0 turns	0 turns	1 turn
Transcription error rateWER of the vendor’s own transcript	0.0%270 words	0.0%288 words	0.0%270 words	0.0%270 words
Missed repliesno reply within 8 s	0	0	0	0

Cost

What our test calls actually cost

Measured from each vendor’s own billing on these calls (about 32 seconds each), on default settings. Your rate depends on the model and voice you choose.

Measured cost per minute per vendor

Measure	Ultravox	Vapi	ElevenLabs Agents	Retell AI
Per minute, as measured	$0.05/min list price	about $0.08–0.09/min	about $0.08/min equivalent	about $0.15/min
What one call cost	about $0.24 in total, after the free 30 minutes	$0.042–0.051 per ~32 s call	about $0.045 per ~32 s call (plan credits)	$0.080–0.084 per ~32 s call
Note	Most of our Ultravox time used its free minutes; the rest is billed at $0.05/min. The total also covers Ultravox acting as our scripted caller, and is our calculation from billed seconds (Ultravox shows no dollar usage).	Vapi’s $0.05/min hosting plus pass-through model, transcription and voice charges.	Charged as Starter-plan credits; no extra dollars were billed.	With Retell’s default voice and LLM; Retell lists $0.07–0.31/min depending on the voice and model you pick.

Twilio added about $0.01/min for the phone leg (SIP dial-out plus recording). See each vendor’s current list prices on its cost calculator page.

Per vendor

Notes on each platform

• 

  ElevenLabs Agents

  ElevenLabs default LLM, eleven_v4_turbo voice

  Fastest over the browser connection (0.41 s median) and tied fastest on the phone at 1.04 s median (p90 1.34 s), with all 33 phone turns timed.

• 

  Ultravox

  Ultravox default speech-native model and default voice

  Second-fastest on the browser connection (0.49 s median) and 1.18 s median on the phone across 11 calls. Its 6.9% phone transcription score comes entirely from missing the caller’s closing “Thanks, bye” before the call ended; the three test questions were transcribed with no errors.

• 

  Retell AI

  gpt-4.1-mini, ElevenLabs “Adrian” voice (Retell defaults)

  Tied fastest on the phone at 1.04 s median (p90 1.22 s), and the only vendor with zero transcription errors on the phone path (319 words). Tested through Retell’s current SIP endpoint with its default gpt-4.1-mini and ElevenLabs voice.

• 

  Vapi

  gpt-4o-mini, Deepgram nova-3, Vapi “Elliot” voice (Vapi defaults)

  Slowest on both paths: 1.56 s median on the phone (p90 1.83 s) and 1.15 s over the browser connection. Its −0.26 s fastest time is real, not a typo: in one call a reply started 0.26 s before our caller had finished speaking. Our analyzer counts a reply as talking over the caller only when it starts more than 0.3 s before the caller stops, so this one is timed as a reply (a negative one) and not counted as talk-over. Ran on Vapi’s default gpt-4o-mini, Deepgram nova-3 and Vapi voice.

Methodology

How we tested

Same job for everyone

Every vendor got the same system prompt: a dental-office receptionist that answers in one or two sentences. The caller said the same three lines in every call (booking a cleaning, a follow-up, a question about hours), with 1.5 s of silence after each.

Vendor defaults

Each vendor ran its own default model and default voice, recorded in the notes above. Nothing was tuned, so the stacks differ: this compares the platforms as you’d get them out of the box.

Real phone path

Twilio called each vendor’s SIP address from a US phone number, with dual-channel recording. The caller was a scripted Ultravox voice agent reading the fixed lines, so wording varied slightly from call to call. Using Ultravox as the caller doesn’t affect the timings, which are measured on each vendor’s side of the recording.

Timing from recordings

Response time is the gap from the end of the caller’s speech to the first agent audio louder than −40 dBFS. On phone calls it’s measured on Twilio’s recording; each caller turn is matched to the agent’s first reply that starts within 8 s. Browser-connection timings were measured where our caller ran.

Transcription error rate

Word error rate (WER) compares the vendor’s own transcript of the caller with what the caller actually said: substituted, missing and extra words, divided by the words spoken, after the same text normalization for every vendor. On phone calls the reference is the caller agent’s own log, matched to each vendor call by start time.

Counts

Vapi: 11 phone calls (34 turns timed); Retell AI: 11 phone calls (33 turns timed); ElevenLabs Agents: 11 phone calls (33 turns timed); Ultravox: 11 phone calls (33 turns timed). Browser connection: ElevenLabs Agents 10 sessions, Ultravox 11 sessions, Retell AI 10 sessions, Vapi 10 sessions.

When and where

October 20, 2026, about 1:10–3:35 AM ET, plus 8 Ultravox phone calls about 1:35–2:15 PM ET, from a server in Ashburn, Virginia (US East).

Keep going

Compare these platforms

Building a voice agent? Ask to be included in the next round.