Connecting Vapi or Retell to a Real Carrier Network

You can have the best model in the world, the most natural voice, and a prompt you have tuned for a month. If the trunk underneath it adds 300 milliseconds of latency, the person on the other end will still think they are talking to a bad robot.

This is the part of AI voice that gets least attention, because it is not the interesting part. It is also the part that decides whether your agent sounds like a colleague or like a call center in a wind tunnel.

Why AI agents are harder on a network than people are

Humans are remarkably tolerant of bad phone lines. We repeat ourselves, we talk over each other and recover, we infer words we did not quite hear. Two people can hold a conversation over a link that is objectively terrible.

An AI agent has none of that tolerance, and it compounds the problem in three ways:

Latency stacks. A human conversation has one network hop between two mouths. An agent conversation has the network hop, plus speech-to-text, plus model inference, plus text-to-speech, plus the hop back. Your model pipeline might account for 500ms of that. If the carrier leg adds another 300ms, you are near a second of silence after every sentence, the point at which people start saying “hello? are you there?” and talking over the agent.

Turn-taking breaks first. Barge-in detection, knowing when the caller has started speaking so the agent stops talking, depends on hearing the caller promptly and cleanly. Jitter and packet loss make an agent either talk over people or pause awkwardly, and both read as “broken” long before the words are wrong.

Transcription degrades before audio sounds bad. A line a human would call perfectly fine can push word error rates up enough that the agent mishears a postcode or an order number. The failure does not present as bad audio. It presents as an agent that seems stupid.

What actually causes those problems

Almost always, one of four things:

• Too many hops. Traffic handed from wholesaler to wholesaler picks up latency at every handoff. Each one adds a queue.
• Routes that are geographically silly. A call from Dallas to Houston should not transit three states and two networks. On cheap routes, it sometimes does.
• Transcoding. Every codec conversion along the path costs quality and milliseconds. A path that converts twice is measurably worse than one that does not.
• Congested peering. A route that is fine at 3am and unusable at 5pm is a capacity problem on someone else’s network, and you will not be told about it.

None of these show up in a rate sheet. All of them show up in your agent’s transcripts.

What to ask a provider before you build on them

Vendors will all say “low latency” and “carrier grade”. These questions get past that:

1. Whose network does the call actually run on? If the answer involves a third party you have not heard of, latency and quality are that party’s decision, not your provider’s, and not something they can fix when it degrades.
2. How many hops to the carriers I actually terminate to? Ask for the specific path to AT&T, Verizon and T-Mobile. A provider that operates its own network can answer this. A reseller usually cannot.
3. What codec will be used, and will anything transcode? You want to know whether the path is clean end to end.
4. Can I get a static IP allow-list instead of registration? For an always-on agent, IP authentication is more stable than a registration that can drop.
5. What happens at 3am when a route degrades? “Open a ticket” is a different answer from “an engineer reroutes it”.

Connecting the agent

Mechanically, this part is simple, and it is the same for ElevenLabs, Vapi, LiveKit, Retell, or something you built yourself, they all speak SIP.

• Inbound: point a DID at the SIP URI your agent platform gives you. Calls to that number land on the agent.
• Outbound: the platform registers to the trunk, or you allow-list its IPs, and sets the caller ID it should present.
• Both directions on the same number, so callbacks work and the number your customer sees is the number that reaches your agent.

Because the integration is plain SIP, the agent platform is not a lock-in decision. If you want to move from one vendor to another in six months, and in a market moving this fast, you might, you repoint the trunk. The numbers, routing, and carrier relationships stay where they are. That is worth designing for deliberately: keep the voice infrastructure independent of the model vendor, so you can change your mind about the model.

A note on disclosure

Several states now require you to tell people they are speaking to an automated system, and the FCC treats AI-generated voices as artificial under the TCPA. The rules are still moving. This is worth getting right early, because retrofitting disclosure into a deployed agent estate is far more annoying than building it in. Our Acceptable Use Policy sets out what we expect from AI traffic on our network.

Testing it properly

Before you put an agent in front of customers:

• Place real calls into the area codes you actually dial, at your busy hour, not at 10am on a Tuesday.
• Measure round-trip latency on the carrier leg specifically, isolated from your model pipeline, so you know which half to fix.
• Read the transcripts, not just the audio. Word error rate degrades before audio sounds obviously bad.
• Test barge-in deliberately: interrupt the agent mid-sentence and see how fast it yields.

If you want to test against our network before committing anything, we will set up a trunk and a number so you can run your own agent over it and measure it yourself. Get in touch, the numbers are more convincing than our description of them.