Designing voice agents people don’t hang up on.

Nobody abandons a chatbot because it took two seconds to answer. People abandon voice agents in under ten. The telephone has trained all of us for a century: silence means the line is dead, a late reply means the other party is confused, and being talked over means nobody is listening. A voice agent that violates any of those rules gets hung up on before it has said anything useful.

We build voice agents for customer support, and we keep a reference build running in public: a multilingual financial-services support agent, fluent in Hindi, Tamil, and English, live at /voice-agent-demo. Every claim in this piece comes from building and instrumenting that bench. The numbers are our targets, not industry folklore.

The latency budget

In human conversation, the gap between turns is roughly 200 milliseconds. People will forgive a machine more than that, but not much more: past about one second of voice-to-voice latency, callers start saying “hello?” and the turn structure of the call collapses. So we treat one second as a hard ceiling and budget every stage against it.

Two things make this budget survivable. First, everything streams. The ASR emits partial transcripts while the caller is still speaking, the model starts generating the moment the final transcript lands, and TTS synthesizes the first clause without waiting for the full response. A pipeline that waits for complete outputs at any stage blows the budget on its own.

Second, end-of-speech detection deserves more respect than it gets. It looks like the cheap stage, but it is a genuine trade-off: trigger too early and you cut callers off mid-thought, which is worse than being slow; trigger too late and you donate hundreds of milliseconds to dead air on every single turn. We tune it per language, because pause patterns differ, and we let typed activity extend the window — in our demo, a keystroke in the chat panel signals user activity so the agent holds its turn.

Turn-taking: the agent must stop talking first

Barge-in is the feature that separates a conversation from an IVR menu. When the caller starts speaking, the agent must go silent immediately — not at the end of the sentence, not after the current audio chunk drains. The moment speech onset is detected, we stop playback and flush every queued TTS buffer.

The subtle part is keeping the conversation state honest afterwards. The model generated a full reply, but the caller only heard the first half of it. If the transcript records the whole generated turn, the model will later refer to things the caller never heard, and the call gets strange fast. So we truncate the agent’s turn in the transcript at the last word that actually played, and the next generation is conditioned on what was heard rather than what was written.

We also make turn state visible. The waveform in our demo is driven by the agent’s real output audio — per-frame frequency data, orange while the agent speaks, blue while it listens. Callers on the web page can see whose turn it is; callers on a phone hear it, because a properly interrupted agent yields within a syllable.

Code-switching is the normal case

In India, asking a caller to pick a language at the start of a call is asking them to lie. Real callers open in English, drop into Hindi for the emotional part, and quote an SMS in Tamil, sometimes inside one sentence. A per-call language setting is wrong within three turns. Detection has to happen per utterance.

For the transcript layer, the honest answer is also the cheap one. Hindi and Tamil are written in different Unicode blocks, so a script check is a single regex pass with no model call and no network round-trip. This is the exact function that tags each line of our demo transcript HI, TA, or EN:

type Lang = 'Hindi' | 'Tamil' | 'English';

// Per utterance, not per call. One regex pass,
// no model call, no network round-trip.
function detectLang(text: string): Lang {
  if (/[\u0900-\u097F]/.test(text)) return 'Hindi'; // Devanagari block
  if (/[\u0B80-\u0BFF]/.test(text)) return 'Tamil'; // Tamil block
  return 'English';
}

Know the limits of this check. Romanized Hindi typed in Latin script reads as English here, and that is fine, because script detection is doing transcript labeling, not comprehension. Understanding the spoken switch is the model’s job: the agent is prompted to answer in whatever language the caller last used, and the ASR is multilingual end to end. The division of labor matters — a cheap deterministic check where a deterministic answer exists, the model where it doesn’t.

The payoff is behavioral. When a caller switches to Tamil mid-sentence and the agent follows without comment, trust jumps in a way no scripted greeting achieves. Language menus tell callers they are talking to a machine; silent following tells them the machine is listening.

Sound design is trust engineering

Between tap and connection, our demo takes a second or two to negotiate a session. Early versions spent that time in silence, and testers reliably tapped again, assuming the button was broken. Silence on a call does not read as loading. It reads as dead.

So we built a small sound layer on Howler with three sounds, each mapped to a call state. A looped ringback at half volume plays the entire time the session is connecting, because ringing is the one sound every human already knows means “wait, it’s working.” A distinct hangup tone fires once when the call ends, so the caller never wonders whether the agent went quiet or went away. A soft click confirms button presses. The sounds are driven off the derived call phase rather than the raw connection status, and the ringback stop lives in the state transition itself, so a fast connect cannot race the loop into playing over a live agent.

None of this is decoration. Every sound answers a question the caller would otherwise answer pessimistically: is it working, is it over, did my tap register. Trust in a voice agent is mostly the absence of those doubts.

The handoff should feel like a promotion

Every voice agent has a boundary, and pretending otherwise is how agents trap people. The design question is what crossing the boundary feels like. Done badly, a handoff is an admission of failure: a long hold, a new voice, and the caller repeating everything from the start. Done well, it feels like being moved up a tier.

Our rules are strict. An explicit request for a human is honored on the first ask, with zero persuasion attempts — arguing with someone who wants out is the fastest trust destroyer we have measured. A second failed attempt at the same task triggers a handoff offer before the caller has to ask. And the transfer carries everything: a structured summary of the call, the caller’s current language, and what has already been tried, so the human opens with “I can see you were asking about the refund” rather than “how can I help you today?” The agent frames it as escalation to a specialist, because from the caller’s side, that is exactly what it is.

What we measure

Aggregate satisfaction scores hide everything interesting about voice. We watch three signals on the test bench, each chosen because it is hard to game:

The third one is the important correction. Raw containment is easy to inflate by making the human hard to reach, and plenty of deployments do. We count a contained call as a success only if the exit was never requested or was granted immediately when it was. Containment achieved by trapping people is churn with a delay.

The best evaluation is still adversarial listening. Open the demo and be rude to it: interrupt it mid-sentence, switch to Tamil halfway through a Hindi question, ask for a human and watch what it does. Then hold your own agent to the same standard — and if you want help getting it there, talk to us.