02ZeroTwo/ LABS
Case studies  ·  03 · Voice Agent
CASE STUDY · 03 / 03Voice · 2026

The case study
you can
interrupt.

Most case studies ask you to trust the numbers. This one doesn't: the system it describes is live on this site. A customer-support voice agent for financial services that speaks Hindi, Tamil, and English — and switches when the caller does, mid-sentence. Talk to it first, then read how it works.

Engagement
In-house reference build
Languages
Hindi · Tamil · English
Core stack
LiveKit · fine-tuned open STT/TTS · WebRTC
Status
Live — try it now

// FULL DISCLOSURE — this is our own reference implementation, not client work. We built it to prove the pattern before selling it. Every number below comes from our test bench, and you can generate your own by calling the agent.

01 · The problem

Support in India is trilingual. IVRs aren't.

A support call in Indian financial services rarely stays in one language. Callers open in Hindi, quote an SMS in English, and switch to Tamil when the topic gets stressful. Traditional voice systems force a language choice at menu level — press 1 for English — and lose the caller the moment they drift.

What breaks in conventional systems

  • Language is chosen once, up front, and enforced for the whole call.
  • Code-switching mid-sentence confuses single-language ASR into garbage transcripts.
  • Latency over ~1.5 seconds reads as "it hung up on me" and callers start talking over the system.
  • No barge-in: the system finishes its paragraph while the caller repeats "agent, agent, agent".
  • Silence during connection and transfer reads as a dropped call.

What the reference build does instead

  • Detects language per utterance, not per call — the agent follows the caller.
  • Streams ASR → LLM → TTS so first audio lands in under a second on our bench.
  • Stops speaking the instant the caller starts (barge-in), then recovers the turn cleanly.
  • Plays deliberate sound cues — ringback, connect, hangup — so the line never feels dead.
  • Escalates to a human with full transcript context when confidence drops.
02 · Architecture

A streaming loop with a latency budget.

Everything is designed backwards from one number: the caller should hear a voice within a second of finishing their sentence. The stack we deploy for clients runs on LiveKit Agents over WebRTC, with our own fine-tuned open-source STT and TTS models per language and use case — so the models, the weights, and the latency are ours to control. Each stage gets a slice of the budget and is streamed, never batched.

Browser / phonemic + speakersound cues (howler)WebRTCfull duplexVOICE RUNTIME · LIVEKIT AGENTSASRstreaming, hi/ta/enLLMpolicy + toolsTTSvoice-matchedbarge-in: caller speech cancels TTSguardrails · PII rules · scope limits · confidence floorHuman handoffwarm transfer + transcriptCRM / ticketingsummary + labels, post-calllatency budget (bench targets):end-of-speech ~150ms → ASR final ~150ms → LLM first token ~300ms → TTS first audio ~250ms ≈ under 1s
Fig 01 — the streaming loop. The dashed feedback path is the part most builds skip: caller speech cancels text-to-speech immediately, which is what makes the agent feel like a listener instead of a kiosk.
03 · Language handling

Per-utterance detection, not per-call settings.

The transcript panel in the demo labels every message with the language it arrived in. The detection is deliberately boring: Unicode ranges first, model-based detection only when script is ambiguous. Boring is fast, and fast is what a live transcript needs.

The check that runs on every message

// Script detection — from the shipped demo
const DEVANAGARI = /[\u0900-\u097F]/; // हिन्दी
const TAMIL      = /[\u0B80-\u0BFF]/; // தமிழ்

function detectLang(text: string) {
  if (DEVANAGARI.test(text)) return 'hi';
  if (TAMIL.test(text))      return 'ta';
  return 'en';
}

Why it matters commercially

  • Callers stay in their strongest language, so resolution doesn't depend on their English.
  • Romanized Hindi ("mera card block ho gaya") is handled by the agent's multilingual model, not the script check — the two layers cover each other.
  • The transcript stores language per turn, which makes QA and analytics honest about what the caller actually said.
  • Adding a language is configuration plus evals, not a rebuild.
04 · Sound design

Silence is a bug.

On a phone call, dead air means something broke. The demo treats audio cues as part of the product: a ringback loop while connecting, a soft click on actions, a proper hangup tone at the end. People trust calls that sound like calls.

  • 01

    Ringback while connecting

    WebRTC session setup takes a moment. A familiar ring fills it, so nobody wonders whether the button worked.

  • 02

    State-driven, race-safe

    Sounds are driven by the call state machine, with an explicit fix for the connect/hangup race — the ring can never keep looping over a live conversation.

  • 03

    Survives development reality

    The Howler registry is cached on globalThis so hot reloads and strict mode don't double-register audio — small engineering that keeps the demo dependable.

  • 04

    A real ending

    Calls end with a hangup tone, not an abrupt cut. Endings are where support interactions are judged.

05 · Test bench

Bench numbers, not client claims.

These are measurements from our own test bench and browser sessions — labeled as such on purpose. Client production numbers belong to clients; yours would come from .

01 · First audio
<1s
From end of caller speech to agent voice, bench p50.
02 · Languages
3
Hindi, Tamil, English — switched per utterance, mid-call.
03 · Barge-in
<200ms
Time for the agent to stop talking once the caller starts.
04 · Build
6 wks
From first commit to the demo you can call right now.
HEAR IT YOURSELF

Reading about voice is silly. Call it.

The demo runs in your browser — no sign-up, no phone number. If it holds up to your interruptions, imagine it trained on your products, your policies, and your customers' languages.

Talk to the agent

30 minutes with the engineer who built this demo. A founder replies within 2 days — no cold calls, no newsletter, no handoff to sales.