Code › tail-villain
Stabilizing Live Interview Turns
Stabilizing live interviews by defining reliable boundaries for user and assistant turns
Live interviews became harder after the audio started working.
A WebSocket could open, the model could speak, and the user’s voice could appear as transcript text. From a distance, that looks like the feature is alive. In real browser sessions, the unstable parts were much smaller and much more damaging: a pause could end the user turn too early, an empty push-to-talk hold could become a fake answer, and the first edge of the interviewer’s next sentence could get clipped.
This was not just an STT problem. The product had to know when the candidate started answering, when the answer ended, whether the transcript was meaningful enough to store, whether the assistant audio was actually finished, and whether the UI should allow the next push-to-talk press.
Stabilizing Live interviews meant defining a turn more carefully.
The tempting boundary was activity_end. If the user releases push-to-talk, or automatic detection decides the user stopped speaking, the assistant can answer. That sounds reasonable until silence and noise start creating activities too.
In Gemini Live, activity_end closes an activity window. It does not prove that the user gave a meaningful answer. A short noise, a blank hold, or a tiny accidental frame can still pass through the same path. If the relay treats that event as permission for the assistant to respond, the interviewer may continue as if the candidate answered.
So the backend gate moved from activity_end to finished, meaningful input transcription. The relay only treated a user turn as real when the finished transcript contained actual letters or numbers. That did not make speech recognition perfect, but it stopped blank activity from becoming product state.
The frontend had to follow the same principle. Live mode could not reuse every text-mode control. Speaker buttons and playback indicators made sense when a saved text message could be sent through a separate TTS path. In Live mode, the model already owned streamed audio. Showing those controls made the UI imply control it did not actually have.
Transcript rendering also needed a boundary. While a Live session is active, it is useful to combine persisted messages with the current in-memory Live transcript. After completion, the page should render persisted messages only. Otherwise, the same turn can appear twice until refresh.
Interruption exposed a different failure.
Pressing push-to-talk while the interviewer is speaking feels like a natural barge-in control. In practice, it mixes several systems at the worst possible moment: browser playback, microphone capture, echo cancellation, manual activity markers, streaming assistant output, and STT.
Real testing produced malformed transcripts that looked like unrelated characters or numbers. A separate STT path seemed like the obvious fix, but it performed worse in those sessions. It introduced language confusion and timing drift, so the canonical transcript moved back to Gemini Live input transcription.
That was a useful correction. A separate model can look cleaner on an architecture diagram and still be worse inside a real-time conversation. Live interviews care about when text arrives, what the user sees, and which transcript the evaluator will later trust. Accuracy alone is not the only variable.
Interruption moved back out of the default path. Browser echo cancellation, noise suppression, and auto gain control are necessary, but they are not a complete turn-boundary strategy. If the app forwards suspect frames while the assistant is still speaking, the relay can only react after bad audio has already reached the model.
Then the handoff problem showed up from the other side.
The assistant could be marked complete, but Gemini Live might still send valid late audio fragments. If the relay trusted turnComplete too aggressively, those fragments looked unsolicited and got dropped. To the user, that was not a clean defensive guard. It sounded like the interviewer was clipped.
Lowering the log level would only hide the evidence. The behavior had to change. The relay needed to keep accepting assistant tail audio until there was real user audio to protect, while still preserving no-interruption semantics once the candidate was actually speaking.
The client also needed to stop reopening the listening state too early. Push-to-talk stayed disabled until queued assistant audio finished, and a short release tail was added after the user let go so the last syllables had a chance to reach the relay before activityEnd. The first push-to-talk frames were no longer dropped just because their RMS was low. Push-to-talk is explicit user intent; quiet speech should not be discarded as empty input before the model even hears it.
These changes were small in code but large in perception. In text chat, a few hundred milliseconds rarely matter. In voice, that interval can be the final syllable of the candidate’s answer or the first word of the next follow-up question.
Once turns became more stable, other trust boundaries became visible.
Language was one of them. Study mode already had a stronger rule that the session language should not change just because the user asks mid-conversation. Regular interview prompts were weaker. They said to respond in the selected language, but they did not clearly reject language-switch requests.
I added the same language-lock principle to the normal interview paths: opening, follow-up, angle transition, closing, and final wrap-up. Technical terms and proper nouns can remain in English when natural, but learner-visible content should stay in the session language. For an interview product, language is not just presentation. It affects evaluation consistency and user trust.
Provider errors needed the same treatment. A transient Gemini or Vertex limit should not leak raw RESOURCE_EXHAUSTED details into the UI. Structured calls received retry and backoff handling. Streaming calls only retry before the first chunk is emitted, because retrying after output has started can duplicate the assistant’s response and confuse persistence.
Retries are not automatically safe in streaming systems. Once bytes have reached the user, repeating the request is no longer invisible.
Looking back, that stretch of work can sound like a list of unrelated fixes: empty push-to-talk, noisy STT, late audio, duplicate transcripts, scroll behavior, language lock, and provider retry handling.
They were all versions of the same question: can the product trust this turn?
It should not trust silence as an answer. It should not trust turnComplete as proof that every audio packet has arrived. It should not assume a separate STT path is better because it is separate. It should not assume the model will preserve session language without a lock rule. It should not assume a provider error will arrive in a user-safe shape.
Live interviews expose these mistakes quickly because the user hears them in real time. Boundaries that can be cleaned up later in text mode become part of the immediate experience in voice.
Kovill can only pressure the candidate well if the product first captures the candidate’s answer as a stable turn. If that layer shakes, persona, evaluation, and feedback all shake with it.