Reduce AI Voice Assistant Latency: The 2026 Guide

Your voice agent replies in two seconds and the conversation dies. The reader waits, gets bored, starts talking over the assistant, or gives up and types. Conversational-linguistics research puts the average human response gap at around 200 ms, and gaps beyond 500 ms already read as unnatural. Anything past 1.5 seconds feels like a delay you have to apologise for.

The good news is that the budget for a well-tuned voice agent in 2026 is well under one second end-to-end, and every millisecond above that is fixable. The bad news is that most self-hosted voice pipelines waste 800 to 1500 ms on batch synthesis, blocking STT, and a re-encode step nobody talks about.

This post walks through where the seconds actually go, which optimisations move the needle, and what a sub-500 ms stack looks like in 2026. If you're wiring a voice agent up from scratch, read the voice setup guide first and come back for the latency pass.

Where the seconds actually go

A voice turn has four consecutive steps, and each one has its own budget. Miss any one of them and the whole turn slips.

The two step-changes are streaming and provider choice. A pipeline that runs STT to completion before starting the LLM, then waits for the full LLM response before starting TTS, then re-encodes the audio before uploading, easily lands at 3-4 seconds per turn. The same pipeline with streaming edges between stages, a low-latency TTS, and the audio codec the channel already accepts lands closer to 500-800 ms.

Streaming is the single biggest win

Batch synthesis is the most common reason a self-hosted voice agent feels slow. In a batch pipeline the LLM finishes its entire reply, hands the full text to the TTS, waits for the audio to render, and only then plays. If the reply is 30 tokens the user is waiting for all 30 before hearing a single syllable.

Streaming reverses this. The LLM emits tokens as they are generated, the TTS starts synthesising as soon as it has the first phrase, and the first audio chunk hits the channel while the tail of the reply is still being written. Independent write-ups in 2026 measure this switch as a 300-600 ms reduction in perceived latency, and up to 400-800 ms off P95 turn latency when you also stream STT into the LLM.

Two practical rules:

• Stream in and out of the LLM. Don't buffer the full transcription before you start the LLM call, and don't buffer the full LLM response before you start TTS. Every SDK worth using supports token streaming.
• Pick a TTS that streams first-chunk under 150 ms. Batch-mode ElevenLabs Multilingual v2 is a beautiful voice, but a 500-800 ms time-to-first-audio is a conversation killer for real-time turns.

TTS provider latency, honest numbers

Vendor benchmarks lie because they compare their fast tier against everyone else's realistic tier. Coval, Gradium, and Future AGI all published independent numbers in 2026, and the useful summary is short:

• Cartesia Sonic Turbo - roughly 40 ms time-to-first-audio, the fastest production TTS available. Cartesia Sonic-3 sits closer to 90 ms with slightly richer prosody.
• ElevenLabs Flash v2.5 - around 75 ms on the real-time path, with the tradeoff that voice cloning fidelity is lower than Multilingual v2 or v3.
• Deepgram Aura-2 - approximately 313 ms P50 in the Coval benchmark. Competitive, not first-tier for pure latency.
• OpenAI tts-1 - around 200 ms first-chunk, trailing the dedicated real-time providers but fine if you already live inside OpenAI.
• Piper (self-hosted) - dominated by CPU. On a small VPS it often loses to hosted realtime providers even without the network hop.

Raw latency is no longer the differentiator at the top of the market. Cartesia, ElevenLabs Flash, Rime, and Deepgram all publish sub-150 ms first-chunk, so the useful axis is prosody, cloning, and cost. For a broader comparison of the paid TTS landscape read the TTS providers guide.

LLM time to first token

The LLM budget in a real-time voice loop is 150-400 ms to the first token. Three techniques take this from "sometimes fits" to "always fits":

• Prompt caching. For any pipeline that resends the same system prompt (which is almost all of them), enabling prompt caching drops time-to-first-token by 200-400 ms with no code changes beyond a flag.
• Small fast models. GPT-4o-mini, Claude Haiku 4.5, and Groq-hosted Llama models routinely deliver first tokens in 100-180 ms. Reach for the bigger model where the extra reasoning depth is worth the extra 500 ms, not by default.
• Speculative decoding on your own inference. If you self-host the model, speculative decoding can cut TTFT by 30-50% for the same output quality. It's a bigger lift than picking a smaller model, but the ceiling is higher.

The audio codec trap

Almost every voice-latency post ignores the re-encode step, which is why self-hosted Telegram bots consistently feel slower than the numbers on paper suggest. Telegram accepts Opus voice notes natively. If your TTS returns MP3 or WAV, something in your pipeline has to re-encode to Opus before the upload succeeds. That step alone costs 200-500 ms on a small VPS, entirely wasted.

The fix is provider-side: request Opus output directly from the TTS. ElevenLabs supports it, Cartesia supports it, most modern providers do. On self-hosted Whisper the same trap runs in reverse when the incoming voice note is Opus and you decode it through a slow codec path.

If your voice agent is dropping audio entirely rather than just running slow, that's a different problem - the voice troubleshooting checklist covers the common failure modes.

A sub-500 ms 2026 stack

Assembled from the numbers above, the fast lane looks like this:

• STT: Deepgram Nova-3 streaming - 60-100 ms
• LLM: GPT-4o-mini or Claude Haiku 4.5 with prompt caching - 100-180 ms first token
• TTS: Cartesia Sonic Turbo or ElevenLabs Flash v2.5 in native Opus - 40-80 ms first chunk
• Transport: WebRTC or direct Telegram upload - 20-40 ms

Add 50 ms for VAD and audio capture, keep streaming edges between all stages, and total end-to-end perceived latency lands under 400 ms. That's inside the natural human turn-taking window.

The stack most people actually run - hosted Whisper batch, GPT-4o, batch ElevenLabs Multilingual v2, re-encode to Opus, upload - lands closer to 2.5-3 seconds. Two of the four steps are wrong and the audio has an unnecessary round trip.

What a managed voice agent actually removes

Everything above is tunable if you like tuning it. It's also the part of a self-hosted voice stack that shifts every quarter as providers ship new models. If you'd rather not chase Cartesia Sonic 3 versus Sonic 4 in a config file, the honest recommendation is a managed agent that keeps the latency stack current for you.

Hermify runs a managed Hermes Agent on Telegram with streaming STT, prompt caching on by default, a low-latency TTS chain, and native Opus output already wired up. Typical measured perceived latency on a Telegram voice note is 500-800 ms end-to-end. You connect your OpenRouter key, pick a tier, and start talking, with no docker-compose file, no ffmpeg, and no codec debugging.

Get started with Hermify and have your voice agent live in about a minute.

Sources

• Designing Voice Assistants: STT, LLM, TTS, Tools, and Latency Budget - smallest.ai
• How to Optimize Voice Agent Latency: 12 Techniques for 2026 - Future AGI
• Time to First Audio: Measuring and Reducing TTS Latency in Voice Agents - Gradium
• TTS Latency Benchmark 2026: Gradium, ElevenLabs, Cartesia, Deepgram - Gradium
• Best TTS Providers 2026: Why Vendor Benchmarks Lie - Coval
• Latency Budgets for Real-Time Voice - The Prompt Bench