Final Part of the series: What It Takes to Build Production-Grade Voice AI Agents
TL;DR: The AI components of a voice agent (LLM, STT, TTS) are the easy part. The hard part is everything around them: telephony errors that lie about what happened and hide data quality problems, audio that degrades across real phone networks in ways lab testing never reveals, silence detection that must distinguish "thinking" from "gone," and interruption handling that turns out to be a cultural problem, not a technical one. This post covers the production failures we discovered only after shipping to real users making real calls over real carrier networks.
Voice AI demos are designed to impress you. The AI components – the LLM reasoning, the natural-sounding speech, the real-time transcription – are genuinely impressive and increasingly commoditized.
What will determine whether your deployment succeeds or fails is everything around the AI: how you handle telephony failures, cope with degraded audio on real phone networks, detect silence, and handle interruptions across languages. These are the problems that don't exist in demos and only appear when real users make real phone calls over real carrier networks.
We've shipped production voice agents across multiple deployments using a real-time communication framework for media orchestration, SIP trunks for telephony, and a modular AI pipeline (STT, LLM, TTS). Here's what broke, and how we fixed each one.
SIP Errors Are a Dark Art
When an outbound call fails, you get a SIP status code. In theory, these are well-defined by the RFC. In practice, carriers reinterpret them liberally.
We started with a handful of SIP code mappings. After running production campaigns, we needed to roughly triple that to nearly 20. The gaps showed up as "unknown error" entries in our analytics – calls that failed for reasons we couldn't categorize, making it impossible to distinguish a busy signal from a disconnected number from a carrier-side timeout.
The specific discrepancies matter. The RFC says code 480 means "Temporarily Unavailable." Our telephony provider uses it for "call rejected." The RFC says 603 means "Decline." In practice, we see it when a call rings out with no answer.
If you map these codes according to the standard, your analytics will confidently tell you the wrong thing about why calls fail.
Our mapping now covers the full range of 400-level client errors, timeout and availability codes (408 no answer, 410 number disconnected, 480 call rejected, 486 busy), cancellation codes (487 request terminated), server errors (500, 502, 503, 504), and 600-level global failures (600 busy everywhere, 603 no answer, 604 does not exist). Each maps to a standardized internal status (failed, error, or cancelled) and a human-readable reason.
We also added a fallback parser that extracts a status code from the SIP error text when the primary code field doesn't match any known mapping. Carrier implementations are inconsistent enough that this catches real cases.
But the real cost of incorrect SIP mappings isn't the error codes themselves – it's the downstream blindness. When your system can't distinguish "customer was busy" from "number doesn't exist" from "number switched off" from "carrier rejected the call," you lose the ability to diagnose systemic problems.
Is your contact list stale? Is your trunk provider having routing issues in a specific region? Are customers genuinely unavailable, or is the carrier returning misleading codes? That data quality gap feeds back into everything: campaign strategy, retry logic, call scheduling, and the reporting you hand to stakeholders.
Telephony also adds its own latency budget that no amount of LLM optimization can claw back. SIP call setup time and first-audio delay vary by carrier and region. On some carriers, the user waits two to three seconds after answering before they hear the agent's first word – and that delay eats into their patience before the conversation even begins.
The lesson: don't trust that SIP codes mean what the RFC says. Build your mappings empirically from production data, plan to expand them, and treat telephony data quality as a first-class concern – it feeds every decision downstream.
Audio Quality on Real Phone Networks
Speech recognition that works perfectly in testing breaks on production telephony audio. The gap is significant and systemic.
Carrier codec compression strips out audio frequencies that STT models rely on. Rural connectivity on 2G/3G networks introduces packet loss and audio dropouts. Background noise from real environments – truck depots, highway wind, loudspeaker announcements, other conversations – creates interference patterns that clean test audio never has.
The effects cascade through the pipeline. A brand name that transcribes perfectly over a clean connection comes through as three or four different phonetic variants on telephony audio, depending on the carrier, the caller's regional accent, and ambient noise. We documented six distinct STT spellings for a single major brand name from production calls – variants we'd never have invented in testing.
When the STT gets it wrong, the LLM processes a garbled input. It either misunderstands and responds to the wrong thing (the user hears the agent confidently discussing something they never said), or it fails to recognize a domain term and asks the user to repeat themselves. Either way, the conversation derails.
The mitigation is multi-layered. Domain vocabulary in the prompt covers known phonetic variants. STT provider selection matters enormously – a provider purpose-built for Indian languages caught variants that a general-purpose English-first provider consistently missed. And the prompt needs to handle ambiguity gracefully – when the transcription is unclear, the agent should ask for clarification rather than guess.
The lesson: test on real telephony audio from your target carriers and regions. Clean test audio tells you nothing about production accuracy.
But mitigation only goes so far. The agent will still mishear things. The production-grade response to this isn't better STT – it's an agent that knows it can be wrong.
We designed the conversational flow around the assumption that any piece of information might have arrived garbled. When the agent hears something ambiguous, it confirms rather than assumes. When a caller gives a partial answer, the agent probes for specifics rather than accepting the gap and moving on.
This isn't politeness – it's data quality. A call where the agent confidently recorded the wrong brand name is worse than a call where the agent asked one extra question. The cost of confirmation is a few seconds; the cost of a wrong assumption is an entire wasted call.
This changes how you think about agent design.
The agent isn't a transcription pipeline that processes clean input – it's a conversational partner operating on unreliable information, and its job is to tease out accurate data through dialogue.
The callers who give you the least information upfront are often the ones with the most to say. A well-designed agent earns that information through follow-up, not by hoping the first answer was complete.
Silence Detection Is Harder Than It Sounds
Users pause to think. Background noise flares up. Someone sets their phone down to check something. The agent needs to distinguish all of these from "the user has left" – without feeling aggressive or robotic.
We built a custom silence detection system through several iterations. The final design uses escalating nudges in Hindi – gentle prompts like "Kya aap abhi bhi call par hain?" (Are you still on the call?) -- with enough patience between them that users don't feel rushed. If all nudges go unanswered, the agent ends the call gracefully.
The key lesson: a silence detector can't be a standalone timer. It needs full awareness of the agent's state – what the STT is doing, whether the LLM is mid-generation, whether TTS is synthesizing. All agent activity becomes input to the detector. Without that awareness, the detector nudges the user while the agent is still preparing its response, which is both jarring and nonsensical.
Getting silence detection right meant making it observe the entire agent lifecycle, not just the absence of audio.
We started with thresholds that felt responsive in testing – a few seconds of silence before nudging. In production, with real users on noisy phone lines, they were far too aggressive. Users were being nudged while still thinking about their answer.
The lesson: The silence threshold that feels uncomfortably long in a demo turns out to be exactly right for real conversation rhythm, especially when someone is speaking in a second language.
Interruption Handling Is a Cultural Problem
Most voice agent frameworks give you two interruption knobs: duration-based (seconds of overlapping speech before the agent yields) and word-count (number of recognized words before yielding).
Duration-based was too aggressive for our Hindi-speaking users. A few hundred milliseconds of audio is roughly one short Hindi word or a brief vocalization. It cut off the agent every time a user said "haan" or "achha" – the Hindi equivalent of "uh-huh" and "okay." These are backchannel signals indicating engagement, not an intent to take the floor. But the duration detector can't tell the difference.
We disabled duration-based detection entirely and kept only word count, requiring at least two recognized words before treating overlapping speech as an interruption. Two recognizable words means someone is genuinely trying to say something. A brief vocalization is just conversational rhythm.
The asymmetry matters. False-positive interruptions – the agent stops speaking when it shouldn't – are far more disruptive than false-negatives – the agent briefly talks over the user. We accept occasional half-second overlaps in exchange for never cutting off the agent mid-sentence because of a stray "hmm."
This isn't a parameter you set once. Hindi conversation includes constant backchannel signals that English doesn't. Other languages have different patterns entirely.
The right interruption configuration is language-specific, culture-specific, and conversation-style-specific.
What works for an English customer support bot fails completely for a Hindi feedback collection call.
The lesson: interruption handling is a linguistic and cultural design problem, not a technical parameter to tune once and forget.
The Gap Between Demo and Production
Every one of these problems was invisible in development. SIP errors don't happen when you call your own phone. Audio quality doesn't degrade when you test on clean connections. Silence thresholds feel fine when you're the one being silent. Interruption handling works when your test users speak English.
The AI components – the LLM, the speech recognition, the text-to-speech – are increasingly commoditized. The hard, differentiating work is in the production engineering around them: telephony error handling, audio quality resilience, silence and interruption design, operational safety nets. This is where months of engineering time goes, and where the gap between demo and production is widest.
If you're evaluating whether to build voice agents in-house, this is the part of the iceberg that's underwater.
The AI is the visible tip. The production hardening is everything below the waterline.
This post is part of our series on building production-grade voice AI agents. Read the other deep dives: Latency Tuning, Cost Optimization, Prompt Engineering for Voice, Testing Strategies and Analytics & Observability
Shipping a voice agent to production? Let's harden your system - We can audit your deployment architecture for the failure modes covered here – SIP error handling, audio quality resilience, silence and interruption design, and operational safety – before they become production incidents. We've been through these problems across multiple deployments.