Code › tail-villain
Voice Needed More Than TTS
Managing voice opt-in, waiting, playback, and cancellation as one product flow
It is easy to think a voice feature starts and ends with a speaker button.
Read the assistant’s message aloud. Let the user answer through the microphone. On a feature checklist, that looks like TTS plus STT. Once I put it into the study flow, the harder question was not whether audio could be generated. It was whether the user could tell what the product was doing at every quiet moment before the audio arrived.
Replacing the browser Web Speech API path with Gemini-based speech input and output gave the product more control over quality, cost, failures, and latency. It also created more states where the user had to wait. If the interface did not show whether it was listening, transcribing, or generating speech, the screen looked frozen.
So recording needed a waveform. TTS generation needed loading dots next to the persona name, not an ambiguous equalizer animation that looked like audio was already playing. Canceling a recording needed a real path that stopped before transcription cost was incurred. After transcription finished, focus had to return to the text area, which meant waiting for React to commit the enabled state before calling focus.
Those details sound small until audio is involved. In a voice interface, silence is still interpreted as product behavior.
The next problem was the audio format.
Gemini TTS returned raw PCM, not a browser-ready MP3. Wrapping the PCM as WAV made playback work, but the payload was too heavy. On mobile or slower connections, the user had to wait too long before hearing the response, and the voice feature started to feel slower than text.
I tried asking the Gemini endpoint for MP3 directly, but that generation path did not support the audio configuration I wanted. Compression had to happen on the server. After testing a few encoders, I moved to a WASM-based MP3 encoder that worked reliably in Node and kept the conversion cost low enough for the request path.
I kept the audio at 24 kHz instead of downsampling to 16 kHz. The smaller file was tempting, but the result sounded too much like phone audio. For a coaching product, the voice has to be clear enough that listening does not feel like a compromise. The final choice was a 24 kHz, 32 kbps MP3: not studio audio, but clear enough and much lighter than the original WAV path.
The microphone button had a related problem. When the user clicked it, there was a short gap while the browser acquired the device. If nothing changed during that gap, the click felt broken. I added a separate mic-acquiring state so the button reacts immediately, before recording actually starts.
That state matters because perceived responsiveness starts before the expensive work finishes.
Study mode also forced a product decision: voice had to be opt-in.
One possible design was to persist the TTS setting. That was wrong for this feature. Gemini TTS had cost and quota constraints, and a remembered voice preference could turn audio on in places where the user only expected text. Voice needed to be a per-entry choice, not a silent global preference.
A prominent “Start with Voice” button had the opposite problem. It pushed voice too hard. A separate modal also broke the existing study-entry flow. The final version stayed much smaller: a single AI Voice checkbox on the coach-selection page, below the existing start action.
By default, study starts as text. If the user checks the box, the start link carries ?voice=1, and the session page enables TTS for that entry only. It is deliberately low-emphasis. Tail Villain is still a conversation and pressure-practice product first. Voice is useful, but it should enter only when the user explicitly asks for it.
The last cleanup was playback ownership.
Clicking one message’s speaker and then another could make audio overlap. The reason was that playback did not have a single owner. Cached replay, the streaming speech queue, and the Gemini TTS strategy were separate channels. Stopping one path did not always stop the others.
I put every stop path behind one stopAllPlayback() function. Clicking the currently playing bubble toggles it off. Clicking a different bubble stops all channels first, then starts the new one. The streamed opening audio also uses the same loading and playing state as the message’s speaker control, so the UI does not pretend that separate audio systems exist.
This is not a flashy feature, but it is table-stakes behavior once sound is part of the product. Users rarely want two assistant messages talking over each other. One click should produce one sound, and the next click should end the previous one.
TTS was only one part of the voice feature.
The API call matters, but the experience also includes opt-in, listening state, transcription state, synthesis delay, first playback, cancellation, and interruption. If any of those are invisible or inconsistent, voice stops feeling helpful and starts feeling uncertain.
Adding voice was not just adding sound. It was turning silence, waiting, canceling, playing, and stopping into explicit product states.