The Business Challenge
The business situation the organisation faced before the project began.
Voice agents feel unnatural when they reply slowly, cut people off mid-thought, or keep talking when interrupted, and LLM feedback can quote things the speaker never said. The goal was a spoken interview-practice coach that converses at human pace, handles pauses and interruptions gracefully, and gives feedback grounded in what the candidate actually said.
Why AI Was the Right Solution
Why AI was the right approach — and what alternatives were considered.
Interview practice depends on dynamic conversation: follow-up questions that react to what was said and feedback tailored to the answer. Fixed scripts and pre-recorded questions cannot follow up, and a human mock interviewer is not available on demand. A voice agent combining streaming speech recognition, an LLM and speech synthesis can hold an adaptive spoken conversation at any time, which is exactly what practice requires.
How the Solution Was Delivered
Discovery, design, development, testing and rollout — the journey, not the tooling.
- 1Defined what makes spoken practice feel natural: fast replies, no talking over the user, no cutting off mid-thought, and honest feedback
- 2Designed the session: 3 questions from a bank of 14, follow-ups when answers are thin, scored feedback on 5 competencies
- 3Built a streaming pipeline: browser audio to streaming speech recognition, a fast LLM with reasoning off, and streaming text-to-speech
- 4Measured end-to-end voice-to-voice latency (~3s) and cut it to ~1.5s with sentence-by-sentence synthesis and model prewarming
- 5Tuned adaptive end-of-turn detection (1.3-2.6s) so thinking pauses are not treated as the end of an answer
- 6Added ~300ms barge-in so the agent stops speaking as soon as the user interrupts
- 7Added a grounding check that verifies every quote the model cites against the transcript, and covered the system with 66 automated tests
- 8Deployed a public live demo
Key Technical & Architecture Decisions
Architecture, model selection, workflow and trade-offs.
Latency over raw model size: a fast model on Groq with reasoning turned off, so replies can be spoken immediately, with the model picked from a preference list at startup for resilience.
Streaming everywhere: NVIDIA Nemotron streaming speech recognition and Magpie streaming text-to-speech over WebSockets, so no stage waits for a full utterance.
Sentence-by-sentence synthesis: speech starts as soon as the first sentence is generated rather than after the whole reply, plus model prewarming to remove cold-start delays.
Adaptive endpointing: an end-of-turn window between 1.3 and 2.6 seconds instead of a fixed timeout, balancing responsiveness against cutting people off.
Verification over trust: a separate grounding check validates every quote against the transcript before feedback is shown.
Challenges & How They Were Solved
Obstacles hit along the way and how they were overcome.
The first full pipeline took about 3 seconds from the end of the user's speech to the start of the reply, which felt unnatural. Streaming each stage, synthesising sentence by sentence and prewarming models brought it to about 1.5 seconds.
A fixed silence timeout either cut candidates off mid-thought or felt sluggish. An adaptive 1.3-2.6 second window tolerates thinking pauses while staying responsive.
When users interrupted, the agent kept talking. Barge-in detection now cancels the reply in about 300ms.
The model frequently attributed quotes to the candidate that they never said. The transcript grounding check catches these, rejecting 42 of 70 in testing.
User Adoption & Change Management
How users responded — training, change management and feedback loops.
The coach is available as a public live demo. Trust was designed into the conversation itself: it waits for candidates to finish thinking, stops immediately when interrupted, and only gives feedback that quotes what they actually said. These are the behaviours that make users willing to keep talking to a voice agent.
Business Outcomes & Impact
The measurable outcomes from the underlying AI Deployment — with the story behind them.
Voice-to-voice latency reduced ~50% (~3s to ~1.5s); ~300ms barge-in; grounding check blocked 42 of 70 (60%) ungrounded model quotes; 66 automated tests.
Business Outcome Categories
The story behind the numbers
In voice, latency is the experience. Cutting voice-to-voice response time roughly in half, from about 3 seconds to about 1.5, moves the conversation from walkie-talkie turn-taking toward natural dialogue, which is what makes practice feel like a real interview.
Barge-in of about 300ms and adaptive end-of-turn detection mean the coach neither talks over the candidate nor interrupts them while they are thinking, which keeps nervous users comfortable and engaged.
The grounding check is the trust feature. In two scripted test sessions it rejected 42 of 70 quotes (60%) that the model attributed to the candidate but that did not match the transcript. Without it, the feedback would have confidently misquoted users. With it, every piece of feedback points to something the candidate actually said.
Lessons Learned
What surprised the team, what worked well, and what would be done differently.
- Latency is the product in voice AI: ~3s to ~1.5s changed how the conversation felt
- Streaming plus sentence-by-sentence synthesis is the biggest latency lever
- Adaptive end-of-turn detection beats a fixed silence timeout
- Barge-in is essential for natural conversation
- LLM quotes need verification: 60% of cited quotes failed the transcript check in testing
Future Opportunities
Where this solution could go next.
Role-specific question banks and company-style interview modes.
Progress tracking across sessions so candidates can see their competency scores improve over time.
Richer delivery feedback such as pacing and filler words, alongside content feedback.
Applying the same low-latency, grounded pipeline to customer-facing voice agents such as receptionists and support lines.
Professional Reflections
Guidance for another builder tackling a similar problem.
Measure end-to-end voice-to-voice latency from the start, because users feel the total, not any single component, and optimise the slowest stage first.
Stream every stage and start speaking as early as possible; sentence-by-sentence synthesis is one of the highest-leverage changes.
Turn-taking needs as much engineering as the model: endpointing and barge-in decide whether a voice agent feels natural.
Never trust a model's quotes about a conversation without checking them against the transcript.