Sub-second voice AI stacks: Deepgram, ElevenLabs, and LLM trade-offs

The Sub-Second Latency Budget

Human conversation relies on turn-taking pauses between 200 and 500 milliseconds. When an automated voice agent breaches 1,000 milliseconds of total round-trip delay, callers instinctually re-speak, causing double-talk and context corruption. Building a responsive voice agent requires optimizing four sequential stages: audio transport ingestion, speech-to-text (STT) transcription, large language model (LLM) time-to-first-token (TTFT), and text-to-speech (TTS) time-to-first-audio (TTFA).

To achieve a sub-second response loop, engineers must stream data across every layer simultaneously. Buffering complete sentences before passing them to the next component adds fatal overhead. Here is how to evaluate and combine Deepgram, ElevenLabs, and low-latency LLM endpoints to hit sub-800ms performance in production.

Audio Ingestion and Transport Overhead

Latency optimization starts before audio hits your transcription engine. Standard Public Switched Telephone Network (PSTN) calls routed via SIP trunks incur inherent jitter buffer delays and transcoding steps. Switching from G.711 to Opus codecs over WebRTC cuts packet overhead, though as detailed in SIP vs WebRTC: Voice agent latency and audio trade-offs, developers must weigh these protocol choices against the specific constraints of legacy telephony infrastructure.

Protocol selection directly dictates baseline latency. Choosing WebRTC for web applications or optimized SIP endpoints for PSTN inbound traffic provides the clean baseline needed for downstream audio processing.

Speech-to-Text: Deepgram Nova-2 Streaming Configuration

For live speech recognition, batch API calls are non-viable. Deepgram's Nova-2 model operating over streaming WebSockets remains the benchmark for real-time applications. To squeeze maximum speed from Deepgram, configure the WebSocket payload with specific flags:

• Interim Results: Enable partial transcripts so the orchestrator can analyze intent before the speaker finishes their sentence.
• Voice Activity Detection (VAD) Silencing: Set utterance_end_ms between 400ms and 600ms. Setting this below 300ms triggers false turns when callers pause to breathe. Setting it above 800ms guarantees a sluggish response.
• Encoding: Match your transport layer directly—such as raw PCM 16-bit or mulaw at 8kHz for PSTN—to avoid server-side transcoding delay.

Deepgram typically returns final transcript chunks within 100 to 150 milliseconds of VAD silence detection.

LLM Orchestration and Token Streaming

The language model represents the largest potential bottleneck in a conversational pipeline. Waiting for full generation produces multi-second delays. The architecture must stream output tokens directly into the TTS engine as soon as a complete phrase or clause is generated.

To maintain sub-500ms TTFT, choose lightweight models or specialized inference providers. OpenAI's GPT-4o mini, Anthropic's Claude 3 Haiku, or Gemini 1.5 Flash yield fast TTFT performance. Keep system prompts short and structured. Elaborate prompt guardrails increase pre-fill compute time. Store caller history in external state databases rather than inflating prompt context windows on every turn.

Text-to-Speech: ElevenLabs Streaming WebSockets

Converting text back to audio requires low-latency synthesis. ElevenLabs Turbo v2.5 and Flash models are designed specifically for interactive voice applications. By using WebSocket streaming endpoints instead of standard HTTP POST requests, you eliminate TCP handshake friction on every sentence turn.

• Chunking Strategy: Send text to ElevenLabs in small sentence fragments using clause punctuation as stream triggers.
• Audio Output Format: Request pcm_24000 or ulaw_8000 depending on your caller channel. Transcoding high-fidelity MP3 streams into telephony mulaw mid-flight adds preventable CPU milliseconds.
• Optimize Latency Settings: Utilize ElevenLabs' streaming latency optimization query parameters to trade minor synthesis stability for immediate first-frame audio delivery (~75-120ms).

Stack Math and Orchestration Alternatives

Chaining these components independently requires balancing software keys, API limits, and connection timeouts. A complete sub-second stack breaks down as follows:

• Transport/VAD: ~150ms
• Deepgram STT: ~120ms
• LLM TTFT: ~180ms
• ElevenLabs TTFA: ~100ms
• Total Estimated Round-Trip: ~550ms

When evaluating infrastructure cost models, self-managed pipelines utilizing Bring Your Own Key (BYOK) agreements offer complete provider control. As examined in Voicetta's BYOK cost analysis, running individual provider keys drops per-minute telemetry expense compared to wrapped all-in-one services, starting around 1.2 cents per minute excluding telephony costs.

For engineering teams that prefer not to build custom WebSocket routing layers from scratch, platforms like Voicetta serve as a unified Communication Brain. Voicetta orchestrates voice, phone, SMS, and WhatsApp conversations on a shared timeline while supporting direct BYOK connections for OpenAI, Anthropic, Gemini, Deepgram, and ElevenLabs. This provides full provider flexibility and enterprise-ready voice workflows without vendor lock-in or fragile glue code.