Voice AI System Design Interview: Turn Detection, Interruptions, and Latency

A voice assistant can produce its first token quickly and still feel impossible to interrupt. The caller says “Actually, Tuesday,” while Monday’s appointment details keep playing. The model has stopped generating, but several seconds of audio remain queued downstream. A fast model did not prevent a broken conversation.

A strong voice AI system design interview answer follows one turn through recognition, generation, playback, and any external actions. The central question is who owns the right to speak—and how every component learns that ownership has changed.

Evidence boundary: Official documentation below supports specific LiveKit and Twilio behaviors. The architecture, numbers, and interview exercises are original design examples, not vendor benchmarks or candidate reports. No employer’s interview sequence or question frequency is asserted.

Start with Design a Real-Time AI Voice System, then explain what happens when its happy path is interrupted.

What should you clarify before drawing the architecture?

Ask whether the product is a browser assistant, a telephone agent, or a push-to-talk application. Establish supported languages, typical background noise, concurrent sessions, and whether the assistant only answers questions or changes external records. An appointment booking agent needs a different action policy from a voice search demo.

Define response latency from an observable user event: the end of the user’s utterance to the first audible assistant response. Separately define interruption latency from the beginning of a genuine user interruption to the end of audible assistant output. These boundaries require playback instrumentation; a server log showing “cancel sent” cannot establish either outcome.

For this article’s proposed design, assume a full-duplex appointment assistant with streaming speech recognition, an LLM, streaming text-to-speech, and a transport that can clear queued audio. Treat these as exercise requirements, not universal infrastructure choices. Set numerical targets only after agreeing on the user population and measurement points.

Why is silence not enough to detect a completed turn?

Official fact: LiveKit distinguishes voice activity detection, speech-to-text endpointing, semantic turn detection, and manual turn control. Its documentation describes using acoustic activity alongside signals about whether a thought is complete. These mechanisms provide different evidence about turn boundaries. LiveKit turns overview.

Consider “Book an appointment for … next Tuesday.” A pause after “for” does not necessarily invite a response. Conversely, “No” can be a complete turn despite its short duration. A fixed silence timer must trade premature responses against delays after genuinely finished utterances.

Design inference: Keep the detector’s tentative decision separate from the application’s committed turn. An interim transcript may justify preparing a lookup, while the assistant still waits for sufficient evidence before speaking or changing a booking. If new speech arrives, invalidate speculative work tied to the earlier transcript revision.

Evaluate premature endpoints and delayed endpoints separately. Averages hide who gets interrupted: examine long pauses, accents, code-switching, noisy calls, and assistive speech patterns. Label a representative audio set with explicit annotation guidelines and adjudicate ambiguous boundaries instead of treating one annotator’s timing as perfect ground truth.

The follow-up question is often whether to lower the silence threshold. A useful answer proposes a controlled comparison of conversational success, cutoff rate, and response delay. Optimizing only delay rewards an agent that speaks before the user finishes.

How should the streaming components share state?

Use a session coordinator with a monotonic turn epoch. Recognition events carry an input revision; generated text and synthesized audio carry the epoch that authorized them. The output boundary checks that epoch before accepting another chunk. A cancelled request can still finish asynchronously, so cancellation alone is an insufficient stale-result guard.

Separate three records. The transcript record tracks recognized user speech and revisions. The generation record tracks proposed assistant text and tool requests. The playback record tracks audio submitted, cleared, and acknowledged by the transport. They can reference the same turn identifier without pretending they describe the same event.

A worker may calculate a result after its turn has expired. Discard its conversational output, but account for its compute and reconcile any external side effect. Keep the session owner explicit during reconnects or failover so two workers cannot both authorize speech for the same session.

This design uses serial coordination within a session and parallel capacity across sessions. A global lock across every call would create unnecessary interference. Persist the minimum recovery state required by the product, and define whether a disconnected session resumes or starts a new conversation.

What exactly happens when the user interrupts?

Official fact: In Twilio bidirectional Media Streams, outgoing media is buffered. Sending clear empties buffered audio and causes associated mark messages to return. A returned mark can therefore follow a clear, not just completed playback. Twilio WebSocket messages.

Design inference: Treat interruption as coordinated revocation. First fence the old epoch so no new output from it can enter the playback queue. Stop or clear the available playback path promptly, while cancelling LLM and TTS work in parallel. Continue accepting user audio. Do not wait for a slow model cancellation acknowledgment before asking the transport to stop speaking.

The following synthetic trace is a useful interview exercise. Response A belongs to epoch 7; the caller is correcting a date.

Time	Event	Required outcome
0 ms	User begins correction	Capture input
90 ms	Interruption accepted	Revoke epoch 7
100 ms	Clear requested	Stop queued A
120 ms	Late A chunk arrives	Reject epoch 7
160 ms	Playback stops	Record stop time

The observed interruption latency is 160 ms. Detection consumed 90 ms; the remaining 70 ms includes coordination and downstream stopping. Claiming a 100 ms result because the clear request was sent then would measure an internal command, not the caller’s experience.

Test the ordering with an adversarial scheduler: deliver the old audio chunk after clear, deliver duplicate completion events, and delay the cancellation acknowledgment. The invariant is that revoked output never becomes newly eligible for playback. The physical stop still depends on transport and device behavior; bytes already played cannot be recalled.

What if the interruption was just “uh-huh”?

Official fact: LiveKit documents adaptive interruption handling that distinguishes interruptions from conversational acknowledgments. Its tuning guide also describes false-interruption recovery and controls for preemptive generation. Starting work before turn confirmation can reduce delay while wasting compute when that work is cancelled. LiveKit turn-taking tuning.

Design inference: Distinguish a tentative pause from a committed cancellation. If the playback stack supports reversible pausing, briefly pausing can preserve a resume option. If audio has already been cleared, “resume” may require resynthesis from an appropriate boundary. Do not promise that a generic transport can recover discarded audio automatically.

Avoid restarting an entire paragraph after a cough. Maintain a conservative resume point and decide whether the situation merits continuing, asking for clarification, or yielding completely. Measure unnecessary stops and repetitions alongside missed interruptions. An agent that never yields can score well on false-interruption counts while being unpleasant to use.

How do you keep conversation history honest?

Suppose the generated answer says, “Your appointment is Monday at nine. Bring your insurance card.” The user interrupts after hearing “Your appointment is Monday.” Adding the entire generated message to conversational history can make the next response assume information was communicated when it was not.

Design inference: Track generated text separately from the portion believed to have been played. Prefer time-aligned text and audio where available, and retain uncertainty around coarse playback boundaries. Transport acknowledgment is evidence about delivery or playback behavior, not proof that the human understood the sentence.

In a synthetic example, segments A1 and A2 were submitted. A1 has a playback completion before interruption; A2’s mark returns after clear. Preserve A1 as played according to the transport evidence and mark A2 as cleared or uncertain, rather than silently treating both as spoken. If no reliable alignment exists, ask a short clarification instead of inventing an exact heard transcript.

Tool state needs a separate ledger. If the booking succeeded before the interruption, cancelling speech does not undo it. Associate a stable operation identifier with the booking attempt, query ambiguous outcomes, and use a defined compensation or correction operation when necessary. A new conversational epoch must not automatically authorize a second identical booking.

How would you diagnose a slow first audible response?

Create a trace with explicit event names: user speech ends, turn commits, required transcript becomes available, model request begins, usable text arrives, first audio becomes available, and playback starts. Place queueing and network spans where they actually occur. Some stages overlap, so summing standalone stage durations can double-count time.

Here is an original serial-path budget, deliberately simplified for calculation:

Stage	Delay
Turn decision	350 ms
Remaining recognition work	50 ms
Model to usable text	180 ms
Speech synthesis startup	90 ms
Transport and playback	80 ms

The total is 750 ms. Halving model time saves 90 ms, yielding 660 ms; it does not halve the conversation’s response latency. Reducing endpoint delay may save more, but must be evaluated against premature responses. In a streaming implementation, redraw the critical path before reusing this arithmetic.

Measure distributions per language, device, transport, and load condition. Include failed turns and timeouts in the outcome reporting, rather than dropping them from successful-turn latency charts. Also measure time to useful content: a fast “Let me check” can conceal a slow answer.

What fails under load or network trouble?

Bound audio queues by duration as well as bytes. A growing output queue increases the amount of stale speech that must be stopped. Monitor queue age, active sessions, model saturation, cancelled compute, and reconnects. Reserve capacity for interruption and teardown signals so ordinary media traffic cannot indefinitely delay control work.

Choose an overload policy explicitly. Admission control for new calls, shorter responses, or a clear handoff may be preferable to letting every existing session accumulate seconds of lag. Avoid silently dropping arbitrary speech chunks: missing words can change an answer’s meaning.

For a stalled model, use a bounded timeout and a truthful fallback. For a tool with an unknown result, reconcile before retrying. For reconnects, establish a fresh playback authorization boundary and reject chunks from the old connection. A transport reconnect should not replay an expired appointment confirmation.

Keep telemetry useful without making complete call recordings the default debugging dependency. Minimize retained content, restrict access, and specify deletion and consent requirements for the actual deployment. These are design decisions to clarify, not claims that one retention policy fits every product.

Sources and Further Reading

• LiveKit: Turns overview — detection approaches and interruption concepts.
• LiveKit: Turn-taking tuning — recovery and speculative-work trade-offs.
• Twilio: Media Streams WebSocket messages — media buffering, marks, and clearing.