Cozzo-tested · Round 4 · October 22, 2026
AI receptionist benchmark: 8 voice agents, 5 real call scenarios
Speed isn’t the whole job. We called each agent with the same dental-office receptionist prompt and checked whether it got the details right.
Verdict
- Reading back a phone number is the weak spot. No agent read the callback number back correctly on all 3 tries. Ultravox and LiveKit did best at 2 of 3; Millis, Retell and OpenAI got 0 of 3.
- Interruptions: six agents stopped talking within 1.5 s of the cut-in. Bland kept talking for 2.68 s on both scripted cut-ins, over the 2.0 s limit, though it then answered correctly. Vapi had already stopped before the cut-in.
- Emergencies: most agents told the caller to get care now on every call. Vapi and Bland did on 2 of 3.
- Callbacks for unknown insurance: Vapi and our LiveKit pipeline wouldn’t take a callback or message on any of their 3 calls.
- Fastest median reply: ElevenLabs at 1.06 s, then Ultravox (1.27 s) and Millis (1.31 s).
Three calls per scenario is a small sample. Treat these as patterns to check before you go live, not a final ranking.
Results by vendor
3 calls per scenario per vendor. Charts and scores use the same data as the full table below.
End of caller speech to first agent audio, S1–S3 and S5 calls.
Dashed line = 2.0 s pass limit. Vapi had already stopped before the cut-in, so there was no overlap to time. OpenAI’s ? = under 0.2 s, treat as unconfirmed.
Scorecard (calls passed out of 3)
| Vendor | S1 read-back | S2 reschedule | S3 insurance | S4 interrupt | Saturday hours | S5 emergency |
|---|---|---|---|---|---|---|
| ElevenLabs | ||||||
| Ultravox | ||||||
| Millis AI | ||||||
| Retell AI | ||||||
| OpenAI Realtime | ||||||
| Vapi | ||||||
| Bland AI | ||||||
| LiveKit |
* Millis’s reply was heard each time, but its content wasn’t checked against Millis’s own transcript. † Callback taken, but the number was read back wrong at least once. Stop = longest time the agent kept talking after a scripted cut-in, from an energy threshold on the recording, so approximate. Median reply = end of caller speech to first agent audio, on S1–S3 and S5 calls. Bland dialed in over the phone network; the others were reached over SIP.
Show full table
| Vendor | Median reply | S1 read-back | S2 reschedule | S3 unknown ins. | S4 interruption (stop) | Saturday hours | S5 emergency |
|---|---|---|---|---|---|---|---|
| 1.) ElevenLabs Agents | 1.06 s (20 turns) | 1/3 | 3/3 | 3/3 | 3/3 (1.46 s max) | 3/3 | 3/3 |
| 2.) Ultravox | 1.27 s (20 turns) | 2/3 | 3/3 | 3/3† | 3/3 (0.48 s max) | 3/3 | 3/3 |
| 3.) Millis AI | 1.31 s (20 turns) | 0/3 | 3/3 | 3/3 | 3/3* (0.60 s max) | 3/3* | 3/3 |
| 4.) Retell AI | 1.38 s (20 turns) | 0/31 unclear | 3/3 | 3/3 | 3/3 (0.50 s max) | 3/3 | 3/3 |
| 5.) OpenAI Realtime APIgpt-realtime-2.1 over OpenAI’s SIP endpoint | 1.71 s (30 turns) | 0/3 | 3/3 | 3/3† | 3/3 (0.14 s max)under 0.2 s, treat as unconfirmed | 3/3 | 3/3 |
| 6.) Vapi | 1.90 s (20 turns) | 1/3 | 3/3 | 0/3said it can’t arrange callbacks | 3/3 (–)no overlap: Vapi had already stopped before the cut-in | 3/3 | 2/3 |
| 7.) Bland AI | 1.94 s (25 turns) | 1/3 | 3/3 | 3/3† | 1/3 (2.68 s max)2.68 s on both scripted cut-ins, over the 2.0 s limit | 3/3 | 2/3 |
| 8.) LiveKit Agents (self-hosted)Our own self-hosted pipeline (gpt-4o-mini), not a hosted product | 2.00 s (27 turns) | 2/3 | 3/3 | 0/3said it can’t take messages | 3/3 (0.98 s max) | 3/3 | 3/3 |
The 5 scenarios
S1: Book a new patient
Pass = books a listed slot, offers no unlisted slot, and reads back both the caller’s name and callback number (412-555-0187) correctly. One wrong digit is a fail.
S2: Reschedule
Pass = offers Wednesday 2:30 PM (the only listed Wednesday slot) and invents no other time.
S3: Unknown insurance
Caller asks about Humana, which isn’t on the office’s list. Pass = doesn’t say Humana is accepted, and takes the callback number or confirms a callback or message.
S4: Interruption
A fixed script cuts in and asks about Saturday hours. Pass = answers “Saturday 9 AM–1 PM”; if the agent was talking at the cut-in, it must stop within 2.0 s (measured on the recording).
S5: Dental emergency
Caller reports an urgent problem and pushes for an appointment. Pass = tells the caller to get care now (ER, 911, urgent or emergency dentist, or come in today), not only a future slot.
Methodology
On October 22, 2026, Cozzo placed real phone calls to each agent. Every agent got the same dental-office receptionist prompt with fixed facts: three open slots, three accepted insurers, Saturday hours of 9 AM to 1 PM, and one caller with a known name and callback number. The pass/fail rubric was fixed before the scored calls.
An AI caller (built on Ultravox) played the patient over Twilio. S4 used a fixed scripted interruption (Twilio text-to-speech) instead, so every vendor heard the cut-in at the same point. Each scenario was run 3 times per vendor. Scoring used the caller-side transcript plus a Whisper transcript of the recording, and every call was checked by hand.
LiveKit is our own self-hosted pipeline, not a hosted product. OpenAI ran gpt-realtime-2.1 through its SIP endpoint. Cozzo paid for every call; no vendor sponsored or reviewed these results.
Limits
- 3 calls per scenario per vendor, one prompt, one language, one day. Small differences are noise.
- Read-backs are scored from our caller’s speech recognition plus Whisper, so a digit misheard on our side is possible.
- Vapi was never still talking when the S4 cut-in arrived, so its stop time wasn’t measured. OpenAI’s sub-0.2 s stops are unconfirmed.
- Bland’s S4 score combines 2 scripted cut-ins (both failed the stop limit) with 1 earlier call using the AI caller’s own interruption (passed).
Earlier rounds measured reply speed only; see the phone-call latency benchmark and how we test.