Voice agents under 800ms: our latency budget, line by line

In ordinary conversation, people start replying about 200 milliseconds after the other person stops talking. On the phone we're a bit more forgiving. Studies of turn-taking and our own CSAT data both show that people stop noticing latency below about 700ms and start talking over the agent above about 1,100ms. Talking over the agent is how a call goes wrong: the caller repeats themselves, the agent answers the wrong thing, and the caller asks for a human.

A year ago, Pazau Voice had a median voice-to-voice latency of 1,340ms. Today the median is 620ms and the p95 is 790ms. This post covers where the time goes and what we changed.

"Voice-to-voice" means the time from the moment the caller stops speaking to the moment they hear the first audio of the reply. It's the only latency number that matters to the caller, so it's the one we measure.

The budget

Here's a typical turn in the US region at p50 and p95, measured across 30 days of production calls in July.

Stage	p50	p95	Notes
Caller network + carrier to our edge	45 ms	90 ms	PSTN / SIP, varies by carrier
Endpointing (deciding the caller is done)	180 ms	240 ms	The largest single line item
Streaming ASR finalization	40 ms	70 ms	Most of the transcript is already done
Procedure routing + context assembly	25 ms	45 ms	Cached per call
LLM time to first token	190 ms	320 ms	Speculative generation hides part of this
TTS time to first audio byte	85 ms	130 ms	Streaming synthesis
Edge to caller + jitter buffer	55 ms	95 ms
Total (sequential)	620 ms	990 ms
Measured, with overlap	620 ms	790 ms	Speculation overlaps the tail stages

The per-stage p95s add up to more than the measured p95 for two reasons: tail latencies rarely line up on the same turn, and speculative generation overlaps several stages. Here is each line and how we cut it down.

Endpointing: the hardest 180 milliseconds

Before you can reply, you have to know the caller has finished. A simple voice activity detector waits for a fixed silence, usually 500–800ms. That alone uses up the whole budget.

We replaced the fixed silence threshold with a small model that predicts end-of-turn. It combines three signals:

• Acoustic. Falling pitch and energy at the end of a clause.
• Lexical. Is the partial transcript a complete thought? "My policy number is" isn't. "My policy number is 4417 2209" probably is.
• Dialog state. If the agent just asked for a date of birth, a complete date is a strong end-of-turn signal.

The model runs every 20ms on the streaming audio and outputs a probability. We commit when it passes 0.82, or after 450ms of silence as a fallback. Median endpointing time dropped from 610ms to 180ms, and premature cut-offs, where the caller was still talking, happen on 1.4% of turns.

Speculative generation

Our biggest structural change was to stop waiting for endpointing to finish before we start thinking.

When the end-of-turn probability passes 0.5, we start LLM generation on the current partial transcript, while the caller may still be finishing. If the final transcript matches the partial one, or differs only in ways that don't change the meaning, we already have tokens ready when the commit happens. If it differs meaningfully, we discard the speculation and start again.

onPartialTranscript(async (partial, eotProb) => {
  if (eotProb > 0.5 && !spec) {
    spec = agent.generate({ transcript: partial, speculative: true });
  }
});

onEndOfTurn(async (final) => {
  if (spec && semanticallyEquivalent(spec.input, final)) {
    return stream(spec);          // hit: ~180ms saved
  }
  spec?.cancel();
  return stream(agent.generate({ transcript: final }));
});

Speculation hits on 74% of turns. semanticallyEquivalent is deliberately strict. A changed digit in an account number, a negation, or a new entity always counts as a miss. Speculation never performs Actions. Tool calls wait until the transcript is final, so a speculative branch can't issue a refund based on something the caller hadn't finished saying.

LLM time to first token

We cut time to first token in three ways:

1. Prompt caching per call. The Procedure, customer context and Action schemas don't change from turn to turn, so they're cached at the provider. Only the new turn is uncached input. This saved about 110ms at p50.
2. Voice-shaped Procedures. Voice replies are short. The model is instructed to lead with the answer in a first sentence of under 15 words and continue only if needed. The first sentence reaches TTS sooner.
3. Model routing. Simple turns like confirmations, clarifications and "anything else?" go to a smaller, faster model. Turns that need reasoning or Actions go to the full model. The router adds 8ms and saves 140ms on the 58% of turns it routes to the small model.

TTS and streaming

We stream tokens into TTS at clause boundaries rather than waiting for the full sentence. TTS starts speaking the first clause while the model is still generating the rest. We synthesize in 40ms audio frames and send the first frame as soon as it's ready.

We pre-render some audio: greetings, standard disclosures ("this call may be recorded"), and hold prompts. These are cached per voice and per language and play with zero synthesis latency.

Barge-in: letting the caller interrupt

Low latency cuts both ways. If the agent replies fast but can't be interrupted, it feels like arguing with a recording.

Barge-in detection runs continuously while the agent speaks. It has to tell apart a real interruption ("no, wait, the other flight"), a backchannel ("mm-hm", "okay"), and background noise, such as a dog, a TV, or a car horn. It's one of the areas we've improved most this year, and you can follow each release in the changelog:

• Barge-in detection p95: 180ms, down from 410ms a year ago.
• False barge-in rate: 0.9% of agent utterances, down from 3.8%. Backchannels are now classified separately and don't stop the agent.
• On a real barge-in, TTS stops within one 40ms frame, and the agent's context records exactly how much of its reply the caller actually heard. If the caller interrupts halfway through a list of three flight options, the agent knows the caller only heard one.

When the budget runs out: fillers

Some turns need real work, like looking up a claim, checking inventory or running a multi-step Action. Those can take 1–3 seconds regardless of how fast the model is.

We handle this deliberately. If the Procedure's next step includes an Action with expected latency over 600ms, the agent says a short, natural acknowledgment first: "Let me pull up that claim." That's not a stalling phrase. It's what a person would say, and it tells the caller that work is happening. The filler is chosen by the model, varies from turn to turn, and plays within the normal budget. The Action runs in parallel.

We don't use typing sounds, hold music for gaps under 8 seconds, or repeated "one moment" phrases. In A/B tests, each of these lowered CSAT compared with a single honest acknowledgment.

Regional edge

Physics is the one line we can't optimize with software. A call from Munich routed to Virginia and back adds about 180ms before any processing happens.

Pazau Voice terminates calls at the edge nearest the caller and runs the full pipeline (endpointing, ASR, LLM, TTS) in the same region:

• US: Virginia and Oregon
• EU: Dublin and Frankfurt, fully resident, as described in our EU data residency launch

EU workspaces never route voice media or transcripts through the US, even for failover. EU median voice-to-voice latency is 640ms, within 20ms of the US.

PCI pause: when the agent must not listen

Some customers, including Tandem Credit Union and Meridian Mutual, take card payments on voice calls. PCI DSS requires that card numbers never reach systems that aren't in scope. For us, that includes the ASR provider and the LLM.

When a Procedure reaches a payment step, Pazau Voice switches the call to a PCI-scoped path:

1. ASR and LLM streams are paused. The agent can't hear during this step.
2. The caller enters digits by keypad (DTMF). Tones are captured by our PCI-scoped payment component and masked with flat tones on the recording.
3. The payment component tokenizes the card and returns only a token and a success or failure to the agent.
4. Streams resume and the agent confirms: "Thanks, that payment went through."

The handoff in and out of PCI mode takes 60ms. Recordings and Traces for that segment contain only "[payment captured — token tok_…]".

What's next

Our current target is a p95 under 700ms by the end of the year. The main levers are a smaller end-of-turn model we can run on the edge node itself, and speculation that commits earlier on high-confidence turns.

If you're evaluating voice AI, ask vendors for their p95 voice-to-voice latency measured on production calls, not a p50 from a demo on office Wi-Fi. Also ask what happens when the caller interrupts. Those two answers tell you more than any recorded demo.

Read more about Pazau Voice, or book a demo and call a live agent built on your own Procedures.