How to Build a Voice AI Agent

Key facts

What is a voice AI agent?

A voice AI agent is an AI agent you talk to instead of type to. It turns your speech into text, decides what to do with an LLM — answer, ask a follow-up, or call a tool such as a booking system — and speaks the reply back. The best ones handle interruptions, pauses and accents well enough that a phone call feels natural.

Voice agents are one of the fastest-growing uses of agentic AI: clinics use them to book appointments, shops to answer order-status calls, banks and telcos to triage support, and schools to answer admission questions — often in Urdu, Hindi or a mix with English.

• Voice bot vs voice agent — a voice bot follows a fixed menu ("press 1 for…"). A voice agent understands free speech and can act: look up an order, book a slot, send an SMS.
• Voice agent vs chatbot — same brain, different interface. A voice agent adds speech recognition, speech synthesis and strict latency limits, because silence on a call feels broken after about a second.

How a voice agent works: the pipeline

Most production voice agents are a cascaded pipeline of four parts:

1. Voice activity detection (VAD) — notices when the caller starts and stops speaking, so the agent knows when it is its turn.
2. Speech-to-text (STT / ASR) — transcribes the audio. Examples: OpenAI gpt-4o-mini-transcribe, Whisper, Deepgram, AssemblyAI.
3. The agent (LLM + tools) — reads the transcript and conversation history, calls tools if needed, and writes a short spoken-style reply.
4. Text-to-speech (TTS) — turns the reply into audio. Examples: OpenAI gpt-4o-mini-tts, ElevenLabs, Cartesia, Azure Speech.

The alternative is a speech-to-speech (realtime) model that takes audio in and gives audio out in one step, such as OpenAI's Realtime API or Gemini Live. It has lower latency and keeps tone of voice, but you get less control over each step and it usually costs more per minute.

Voice agent tools and frameworks in 2026

• LiveKit Agents — open-source Python framework for real-time voice agents over WebRTC and phone (SIP). Handles VAD, turn detection and interruptions.
• Pipecat — open-source Python framework for building voice and multimodal pipelines from swappable STT, LLM and TTS services.
• Vapi, Retell AI, Bland — hosted voice-agent platforms: configure a prompt and tools, get a phone number. Fast to start, billed per minute.
• Twilio — phone numbers and call streaming, often combined with any of the above.
• OpenAI Realtime API / Gemini Live — speech-to-speech models you can call directly.

Learn the pipeline by hand first (below). Once you understand where latency and errors come from, a framework like LiveKit or Pipecat makes sense instead of feeling like magic.

Step 1: Install and set up

This tutorial uses OpenAI for all three model steps so you need only one key. sounddevice records from your microphone.

mkdir voice-agent && cd voice-agent
python -m venv .venv
# Windows: .venv\Scripts\activate   ·   macOS/Linux: source .venv/bin/activate
pip install -U openai sounddevice soundfile numpy python-dotenv

OPENAI_API_KEY=your-key-here

Step 2: Listen — record and transcribe

Record a few seconds of audio, save it as a WAV file, and send it to the transcription endpoint. A fixed recording window keeps the first version simple; real agents use VAD to stop when you stop talking.

import sounddevice as sd
import soundfile as sf
from openai import OpenAI
from dotenv import load_dotenv

load_dotenv()
client = OpenAI()
RATE = 16_000

def listen(seconds: int = 5) -> str:
    print("🎙️  Speak now…")
    audio = sd.rec(int(seconds * RATE), samplerate=RATE, channels=1)
    sd.wait()
    sf.write("input.wav", audio, RATE)
    with open("input.wav", "rb") as f:
        result = client.audio.transcriptions.create(model="gpt-4o-mini-transcribe", file=f)
    return result.text

if __name__ == "__main__":
    print("You said:", listen())

Transcription models handle Urdu, Hindi and code-switched speech, but accuracy varies by accent and microphone. Test with your real users' voices before you promise anything.

Step 3: Think — an agent with a tool

The brain is an ordinary tool-calling agent loop. The one voice-specific rule: tell the model it is speaking, so it answers in one or two short sentences without markdown, lists or links.

import json

SYSTEM = (
    "You are a friendly voice assistant for a clinic in Karachi. "
    "You are speaking on a call: reply in one or two short sentences, "
    "no lists, no markdown. Reply in the language the caller uses."
)

SLOTS = {"monday": ["10:00", "14:30"], "tuesday": ["11:00"], "wednesday": []}

def free_slots(day: str) -> str:
    slots = SLOTS.get(day.lower())
    if slots is None:
        return f"No schedule for {day}."
    return ", ".join(slots) if slots else "Fully booked."

TOOLS = [{
    "type": "function",
    "function": {
        "name": "free_slots",
        "description": "List free appointment times for a weekday.",
        "parameters": {
            "type": "object",
            "properties": {"day": {"type": "string", "description": "Weekday, e.g. monday"}},
            "required": ["day"],
        },
    },
}]

def think(history: list) -> str:
    while True:
        reply = client.chat.completions.create(
            model="gpt-4o-mini", messages=history, tools=TOOLS
        ).choices[0].message
        history.append(reply)
        if not reply.tool_calls:
            return reply.content
        for call in reply.tool_calls:
            args = json.loads(call.function.arguments)
            history.append({"role": "tool", "tool_call_id": call.id, "content": free_slots(**args)})

Step 4: Speak — text-to-speech and the full loop

Turn the reply into audio, play it, and loop. Say "goodbye" to end the call.

import soundfile as sf
import sounddevice as sd

def speak(text: str) -> None:
    with client.audio.speech.with_streaming_response.create(
        model="gpt-4o-mini-tts", voice="alloy", input=text, response_format="wav"
    ) as response:
        response.stream_to_file("reply.wav")
    data, rate = sf.read("reply.wav")
    sd.play(data, rate)
    sd.wait()

history = [{"role": "system", "content": SYSTEM}]
speak("Assalam o alaikum! How can I help you today?")
while True:
    heard = listen()
    print("Caller:", heard)
    if "goodbye" in heard.lower() or "khuda hafiz" in heard.lower():
        speak("Allah hafiz, take care!")
        break
    history.append({"role": "user", "content": heard})
    answer = think(history)
    print("Agent:", answer)
    speak(answer)

That is a complete voice agent: it hears you, checks the schedule with a tool, and answers aloud. It is turn-based and a little slow — which is exactly what the next section fixes.

From demo to production: latency, interruptions and phones

• Stream every stage. Start TTS on the first sentence of the reply instead of waiting for the whole answer; use streaming STT. Target under ~800 ms from end of speech to first audio.
• Turn detection and barge-in. Use VAD (for example Silero VAD) to detect end of speech, and stop playback the moment the caller interrupts.
• Fillers for slow tools. Say "One moment, let me check" before a tool call that takes more than a second.
• Phone calls. Connect a Twilio (or local SIP) number to your agent through LiveKit, Pipecat or a hosted platform.
• Guardrails. Confirm before booking or paying, never read out private data without verification, and log every transcript. See agent security.
• Language. Test STT and TTS on real Urdu/Hindi callers; pick the voice that sounds natural to them, not to you.

Learn to build voice agents live

AI Season's 6 weeks AI agents bootcamp ends with a full session on conversational voice calling agents: wiring speech-to-text and text-to-speech into an agent, managing turn-taking, interruptions and latency, and shipping a voice agent that handles a real call flow. Before that you build the foundations it needs — tool calling, RAG, LangGraph, MCP, guardrails and deployment.

It is taught live in Urdu (easy for Hindi speakers) with English code, 12 sessions in total. Cohort 02 starts 1st January 2027. The fee is PKR 3,000 early-bird (about ₹1,040) — a fraction of what most voice-AI and agentic AI programmes charge — and the top three scorers get a full refund.

Frequently asked questions

What is a voice AI agent?

A voice AI agent is an AI agent you speak to. It converts speech to text, uses an LLM to decide what to say or which tool to call, and converts its reply back to speech. Unlike an IVR menu, it understands free speech and can take actions such as booking an appointment.

How do I build a voice AI agent in Python?

Chain three steps: transcribe the caller's audio with a speech-to-text model, pass the text to a tool-calling LLM agent, and turn the reply into audio with a text-to-speech model. Loop that per turn. For phone calls and low latency, move to a framework such as LiveKit Agents or Pipecat.

Which is better for voice agents: a pipeline or a realtime speech-to-speech model?

A cascaded STT → LLM → TTS pipeline gives more control, easier debugging and lower cost, and lets you pick the best Urdu or Hindi speech models. A realtime speech-to-speech model gives the lowest latency and most natural flow. Many teams learn and prototype with the pipeline first.

Can a voice AI agent speak Urdu or Hindi?

Yes. Modern speech-to-text and text-to-speech models support Urdu and Hindi, including speech that mixes them with English. Quality varies by provider and accent, so test with real callers before launch.

Is there a voice AI agent course in Urdu or Hindi?

AI Season's live AI agents bootcamp includes a full session on building a conversational voice calling agent, taught in Urdu (easy for Hindi speakers) with English code. Cohort 02 starts 1st January 2027 and costs PKR 3,000 early-bird.