Audio and Speech Agents

Overview

In the previous lesson, we explored how agents process visual information -- reading documents, understanding screenshots, and answering questions about images. Vision gives agents eyes, but many real-world interactions happen through sound. A customer support agent that can only read text misses the urgency in a caller's voice. A field assistant that requires typing is useless when your hands are full. Audio is the most natural human interface, and agents that can hear and speak unlock entirely new categories of applications.

This lesson covers the complete audio pipeline for LLM agents: converting speech to text (STT), routing the transcript through agent reasoning, and converting the response back to speech (TTS). We will examine the key technologies -- Whisper, Deepgram, ElevenLabs, OpenAI TTS -- and build a working voice agent pipeline. Along the way, we will tackle the central engineering challenge of voice agents: latency. A 3-second pause after every sentence makes a voice agent feel broken, so we will explore streaming architectures, chunked processing, and pipeline parallelism that bring response times under 500 milliseconds.

This connects directly to Module 3's tool-use patterns. Speech-to-text and text-to-speech are specialized tools in the agent's toolkit -- the same function-calling and tool-interface-design principles apply, but the data flowing through the pipeline is audio rather than JSON.

The Voice Conversation Flow

A voice agent conversation follows a precise sequence. The user speaks, the audio is captured and transcribed, the transcript enters the LLM reasoning loop, and the response is synthesized back into speech. Understanding this flow is essential before diving into implementation.

sequenceDiagram
  participant User as User (Microphone)
  participant VAD as Voice Activity Detection
  participant STT as Speech-to-Text
  participant Agent as LLM Agent
  participant Tools as Agent Tools
  participant TTS as Text-to-Speech
  participant Speaker as User (Speaker)

  User->>VAD: Raw audio stream
  Note over VAD: Detect speech start
  VAD->>STT: Audio segments
  STT->>Agent: Transcript text
  Note over Agent: Reasoning + planning

  alt Tool use required
      Agent->>Tools: Function call
      Tools-->>Agent: Tool result
      Note over Agent: Incorporate result
  end

  Agent->>TTS: Response text (streamed)
  TTS->>Speaker: Audio chunks (streamed)
  Note over Speaker: User hears response<br/>while TTS continues

  Note over VAD: Detect speech end/<br/>interruption
  User->>VAD: New utterance
  Note over VAD: Barge-in detection
  VAD-->>TTS: Cancel current playback
  VAD->>STT: New audio segments

Several things happen in this flow that distinguish voice agents from text-based chat. Voice Activity Detection (VAD) sits at the front of the pipeline, determining when the user starts and stops speaking. Without VAD, the agent would either clip words or wait indefinitely. Barge-in detection allows the user to interrupt the agent mid-response -- a behavior humans expect in natural conversation but that requires canceling the current TTS output and restarting the pipeline. And the entire response path from LLM to TTS to speaker is streamed, meaning the user starts hearing the response before the LLM has finished generating it.

Speech-to-Text: Giving Agents Ears

Speech-to-text (STT) is the entry point of any voice agent. The quality and speed of transcription directly determine the agent's effectiveness. Two approaches dominate the landscape.

Whisper: The Open-Source Foundation

OpenAI's Whisper is an open-source transformer model trained on 680,000 hours of multilingual audio. It supports 99 languages, handles accents and background noise robustly, and can be self-hosted for complete data privacy. Whisper operates in batch mode -- you send a complete audio segment, and it returns the full transcript. This makes it straightforward to integrate but introduces latency proportional to the audio length.

Whisper comes in several sizes. The tiny model (39M parameters) transcribes in near real-time but with lower accuracy. The large-v3 model (1.5B parameters) achieves near-human accuracy but requires a GPU and takes longer. For agent pipelines, the medium model often hits the sweet spot between speed and accuracy.

Deepgram Nova: Streaming-First

Deepgram Nova is a cloud-native STT API built for real-time applications. Unlike Whisper's batch processing, Deepgram accepts a continuous audio stream over WebSocket and returns interim results as the user speaks, with final results when it detects a pause. This streaming architecture dramatically reduces perceived latency because the agent can begin processing a partial transcript before the user finishes speaking.

Deepgram also provides endpointing -- automatic detection of when the user has finished a thought -- which simplifies VAD logic. Its API includes features like smart formatting, punctuation, and speaker diarization out of the box.

Choosing Between Them

| Feature | Whisper (self-hosted) | Deepgram Nova | |---|---|---| | Latency | Batch (higher) | Streaming (lower) | | Privacy | Full control (on-premise) | Cloud-dependent | | Cost | Compute cost only | Per-minute API pricing | | Languages | 99 languages | 30+ languages | | Streaming | Requires wrapper (e.g., faster-whisper) | Native WebSocket | | Accuracy | Excellent (large-v3) | Excellent (Nova-2) |

For prototyping and applications where latency is secondary, Whisper is the practical choice. For production voice agents where sub-second response time matters, Deepgram or a similar streaming API is typically necessary.

Text-to-Speech: Giving Agents a Voice

The other end of the pipeline is text-to-speech (TTS) -- converting the agent's text response into natural-sounding audio. Modern TTS has moved far beyond robotic monotone; today's models produce speech that is difficult to distinguish from human recordings.

OpenAI TTS

OpenAI offers two TTS models. The standard tts-1 model is optimized for speed and real-time use, while tts-1-hd produces higher-quality audio at the cost of added latency. Both support six built-in voices and accept a streaming response, meaning you can pipe LLM output tokens directly into TTS and start playback immediately.

ElevenLabs

ElevenLabs specializes in expressive, high-fidelity voice synthesis. Its key differentiator is voice cloning -- you can upload a sample of any voice and generate speech that sounds like that person. For agent applications, ElevenLabs offers a streaming API with WebSocket support, enabling the same chunked-output pattern as Deepgram. It also provides fine-grained control over stability (how consistent the voice sounds) and similarity boost (how closely it matches the target voice).

Streaming TTS for Low Latency

The critical insight for voice agents is that TTS should never wait for the complete LLM response. Instead, as the LLM streams tokens, you accumulate text until you hit a sentence boundary (a period, question mark, or exclamation point), then send that sentence to TTS while the LLM continues generating. The user hears the first sentence while the second is being synthesized and the third is still being generated by the LLM. This sentence-level pipelining is what makes voice agents feel responsive.

Real-Time Voice Agent Architecture

Putting these components together requires a streaming architecture that moves data through the pipeline with minimal buffering. The following diagram shows the production-grade design used in real-time voice agents.

graph LR
  subgraph Client["Client (Browser / Mobile)"]
      MIC["Microphone<br/>16kHz PCM"] --> WS_OUT["WebSocket<br/>Send Audio"]
      WS_IN["WebSocket<br/>Receive Audio"] --> SPK["Speaker<br/>Playback"]
  end

  subgraph Server["Voice Agent Server"]
      WS_OUT --> VAD_S["VAD<br/>Silero / WebRTC"]
      VAD_S --> BUF["Audio Buffer<br/>Accumulate segments"]
      BUF --> STT_S["STT Engine<br/>Whisper / Deepgram"]
      STT_S --> QUEUE["Transcript Queue"]
      QUEUE --> LLM["LLM Agent<br/>Claude / GPT-4o"]
      LLM --> SPLIT["Sentence Splitter<br/>Buffer until boundary"]
      SPLIT --> TTS_S["TTS Engine<br/>ElevenLabs / OpenAI"]
      TTS_S --> WS_IN
  end

  subgraph State["Agent State"]
      HIST["Conversation<br/>History"] --> LLM
      LLM --> HIST
      TOOL["Tool Registry<br/>From Module 3"] --> LLM
  end

  style Client fill:#ebf8ff,stroke:#3182ce,stroke-width:2px
  style Server fill:#f0fff4,stroke:#276749,stroke-width:2px
  style State fill:#fefcbf,stroke:#d69e2e,stroke-width:2px
  style MIC fill:#bee3f8,stroke:#3182ce
  style SPK fill:#bee3f8,stroke:#3182ce
  style VAD_S fill:#c6f6d5,stroke:#276749
  style STT_S fill:#c6f6d5,stroke:#276749
  style LLM fill:#c6f6d5,stroke:#276749
  style TTS_S fill:#c6f6d5,stroke:#276749
  style SPLIT fill:#c6f6d5,stroke:#276749
  style HIST fill:#fefcbf,stroke:#d69e2e
  style TOOL fill:#fefcbf,stroke:#d69e2e

The architecture uses WebSocket connections for bidirectional audio streaming. The client sends raw PCM audio at 16kHz, and the server sends synthesized audio chunks back. This avoids the overhead of HTTP request-response cycles for every utterance. The Agent State box on the right connects this to everything we learned in Module 6 -- conversation history persists across turns, and tools from Module 3 are available for the LLM to call when voice requests require actions like database lookups, API calls, or calculations.

Voice Agent Latency Optimization

Latency is the defining engineering challenge for voice agents. Users expect conversational response times under one second. Every millisecond spent in the pipeline is felt directly. Here are the primary optimization strategies.

Pipeline parallelism is the most impactful technique. Instead of waiting for each stage to complete before starting the next, stages overlap. The STT begins processing the next audio chunk while the LLM reasons about the previous transcript. The TTS synthesizes the first sentence while the LLM generates the second. This turns a serial 3-second pipeline into a parallel one that delivers first audio in under 500ms.

Model selection matters enormously. Using Whisper tiny instead of large-v3 for initial transcription, then re-transcribing with the larger model if confidence is low, gives you speed for common cases and accuracy when needed. Similarly, using tts-1 (faster) instead of tts-1-hd (higher quality) reduces TTS latency at a barely perceptible quality cost.

Prefetching and caching help with common responses. If your agent frequently says "Let me look that up for you," pre-synthesize that audio and play it instantly while the actual lookup happens. This filler phrase pattern mimics human conversation behavior and masks processing time.

Connection pooling eliminates the overhead of establishing new connections to STT and TTS services on every turn. Keep WebSocket connections to Deepgram and ElevenLabs alive across the entire conversation session.

Key takeaway: Perceived latency matters more than actual latency. Streaming the first sentence of audio to the user while generating the rest creates the impression of instant response, even if the full response takes several seconds to complete.

Building a Voice Agent Pipeline

Let us build a complete voice agent that accepts audio input, transcribes it, runs it through an LLM agent with tool access, and synthesizes the response back to speech. This example uses OpenAI's Whisper API for STT, Anthropic's Claude for reasoning, and OpenAI's TTS API for speech synthesis.

This code follows the same agentic loop pattern from Module 8's Claude agent lesson. The run_agent function cycles between the LLM and tool execution until the model produces a final text response -- the only difference is that the input comes from audio transcription and the output goes to speech synthesis. The conversation history list persists across turns, giving the voice agent multi-turn memory exactly as we implemented in text-based agents.

Streaming Voice Pipeline

The pipeline above works correctly but processes each stage sequentially -- the user waits for the full STT, then the full LLM response, then the full TTS before hearing anything. For production voice agents, we need streaming. The following example shows how to stream the LLM response through sentence-level TTS.

The key pattern here is sentence-level chunking. The buffer accumulates streamed tokens until a sentence boundary regex matches, then dispatches that sentence to TTS as an async task. The LLM continues streaming the next sentence while TTS synthesizes the previous one. This overlap is what brings perceived latency down from seconds to hundreds of milliseconds.

Audio Analysis Beyond Speech

Voice agents are not limited to transcription and synthesis. Audio contains rich information beyond words. Tone and sentiment analysis can detect frustration, urgency, or confusion in a caller's voice, allowing the agent to adapt its behavior. Speaker diarization identifies who is speaking in a multi-person conversation, enabling agents to track participants in a meeting. Sound classification lets agents respond to non-speech audio events -- a doorbell, a baby crying, or an alarm.

These capabilities are typically implemented as specialized tools that the agent can invoke. Following the tool-use patterns from Module 3, you would define a analyze_audio_sentiment tool or a identify_speakers tool, expose them to the LLM through the standard tool-calling interface, and let the agent decide when to use them based on context.

Connection to Module 3: Audio processing functions are tools in the same way that web search or database queries are tools. The agent does not need to understand the audio processing internals -- it calls the tool with audio data and receives structured results. The tool interface design principles from Module 3 (clear descriptions, well-typed schemas, informative error messages) apply directly.

Latency Budget Breakdown

Understanding where time is spent helps you target optimizations. Here is a typical latency budget for a voice agent turn.

| Stage | Sequential | With Streaming | |---|---|---| | VAD + capture | 200-500ms | 200-500ms | | STT (Whisper medium) | 800-1500ms | 300-600ms (streaming STT) | | LLM first token | 300-800ms | 300-800ms | | LLM full response | 1000-3000ms | Overlapped with TTS | | TTS first audio | 200-500ms | 200-500ms | | TTS full audio | 500-2000ms | Overlapped with playback | | Total (user perceives) | 3-8 seconds | 0.7-1.5 seconds |

The streaming column shows why pipeline parallelism is so important. The user's perceived wait is only from the end of their speech to the first audio playback -- about one second in optimal conditions. Everything after that is streamed concurrently.

Summary

Voice agents extend the LLM agent paradigm from text to the most natural human interface -- spoken conversation. The core pipeline follows a clear pattern: VAD detects speech boundaries, STT converts audio to text (using Whisper for flexibility or Deepgram for streaming speed), the LLM agent reasons and calls tools exactly as it would in a text-based system, and TTS converts the response back to speech (using OpenAI TTS for simplicity or ElevenLabs for expressiveness and voice cloning).

The defining engineering challenge is latency. Sequential processing creates multi-second delays that break the illusion of conversation. The solution is pipeline parallelism: streaming STT results to the LLM as the user speaks, chunking LLM output at sentence boundaries for immediate TTS synthesis, and overlapping audio playback with continued generation. These techniques bring perceived latency under one second.

Audio processing functions -- sentiment analysis, speaker diarization, sound classification -- integrate as tools through the same patterns covered in Module 3. The agent does not need to understand audio internals; it calls tools and acts on structured results.

In the next lesson, we will explore agents that generate visual content -- images and video -- adding creative output capabilities to the multi-modal toolkit.