Code › tail-villain
OpenAI Realtime Behind the Live Relay
Adding OpenAI text, speech, and Realtime paths without replacing the existing Gemini relay architecture
The OpenAI rollout was not supposed to become a second voice system.
Tail Villain already had Gemini-based text generation, TTS and STT, a Gemini Live path behind a backend relay, transcript persistence, and session state wrapped around all of it. If I added OpenAI by cutting a separate route through the product, the demo might work, but the product boundary would get weaker.
So the work was not a model swap. It was adding OpenAI text generation, TTS, STT, and Realtime behind provider-neutral contracts and the existing live relay.
The first boundary was text generation.
Interview and Study responses are not just strings returned from a model. They carry structured generation, streaming output, timeouts, retries, usage logging, and fallback metadata. If the application code keeps depending on the exact shape of a Gemini call, adding OpenAI turns into provider-specific branches spread across the codebase.
I moved the caller-facing shape toward a provider-neutral contract. The product asks for text generation. A routing policy chooses the provider. OpenAI structured generation and streaming providers were added directly, while Gemini stayed available as fallback. I did not route this through OpenRouter because I wanted product-owned behavior for failures, usage metadata, and the voice paths that sit next to text generation.
That change is not very visible in the UI. The learner still receives a Study response, and Kovill still asks the next interview question. Internally, though, the product now depends on Tail Villain’s generation contract instead of one provider’s request shape.
That matters because fallback is not just a backup model name. The product needs to know which provider was attempted, when it failed, where the request moved next, and whether the final response still fits the same session lifecycle.
Voice needed a stricter split.
AI Voice and Live Mode both produce sound, but they are not the same feature. AI Voice belongs to the text conversation path. The assistant response streams as text first. When the completed message is available, the app synthesizes that final text into one TTS payload and plays it. If audio preparation fails, the text answer still exists and the session can continue.
Live Mode is speech-to-speech. The browser sends microphone audio frames into the backend relay, the relay forwards them into a Realtime session, and model audio comes back as the conversation is happening. Push-to-talk, interruption, transcript persistence, session cancellation, and low-latency playback all live in the same path.
Treating AI Voice as a slower Live mode, or Live as a faster TTS mode, creates the wrong product contract. Full-response TTS is simpler and cheaper for typed interaction. Realtime is justified when spoken turn-taking itself is the experience.
OpenAI TTS and STT were added without erasing that distinction. AI Voice still runs text first, then one synthesized audio payload. OpenAI Realtime entered through a session driver behind the existing relay, so the relay kept owning the opening turn, transcript persistence, cancellation, and fallback behavior.
The browser still does not connect directly to the model. That was the rule when Gemini Live was introduced, and OpenAI did not change it. Credentials and long-lived WebSocket sessions belong behind the backend boundary, where the product can observe and control the conversation.
Putting OpenAI behind the relay did not make the UI problems disappear.
The first AI Voice opening failure was silence. Normal replies autoplayed after TTS generation, but the opening stream took a different path. It persisted text without invoking the page-owned TTS flow. Browser autoplay restrictions were also involved: if the user’s Start gesture did not unlock audio early enough, the generated opening audio could be ready and still fail to play.
The fix had two parts. The entry routes carried the tts=1 intent into the interview session, and the Start gesture unlocked audio before the asynchronous work began. Once the opening stream completed and the persisted message existed, the page synthesized that saved message and played it immediately.
The important detail was not to blur this with Live Mode. An AI Voice opening is text streaming followed by TTS autoplay. A Live opening is audio coming from the Realtime connection. Both can sound like the interviewer is speaking, but they have different failure modes and different state transitions.
Once the opening could speak, the rendering bugs became visible.
For a short moment, two opening bubbles appeared. One was the transient streaming bubble. The other was the persisted message inserted after final metadata arrived. The code waited for TTS preparation before calling clearInterviewStream(), so the finished text existed in both places while audio was being prepared.
Then a Thinking... bubble showed up under an already completed opening. The model had finished generating text, and the message had been saved, but the pending assistant state stayed active during TTS preparation. To the user, the interviewer had already spoken and was somehow still thinking.
That was not a copy issue. It was a state ownership issue.
Text generation, message persistence, and audio preparation are related, but they are not one lifecycle. When final text and persistence metadata arrive, the transient stream can be cleared. When the opening message becomes final, the assistant is no longer thinking. If TTS is still being prepared, that should be represented as audio loading, not model generation.
I split those states accordingly. Final metadata and the persisted message clear the transient stream. The pending assistant bubble is cleared when the opening message becomes final. TTS loading proceeds independently afterward. If audio fails, the saved text does not need to be regenerated. If it succeeds, playback attaches to the saved message.
Voice UI makes these boundaries visible. In text mode, a slightly broad loading flag may only look untidy. In voice mode, it can make the product appear to be thinking, speaking, and saving the same message at the same time.
Live rendering needed the same kind of authority rule.
Some branches still checked only voiceMode=1 in the URL. That is fine as an entry hint, but it is not the source of truth after the session exists. The persisted deliveryMode has to win. Otherwise, the same session can be treated as Live in one branch and ordinary voice in another.
I standardized the UI around isLiveDeliveryMode, which includes persisted session state, and removed an unused transcription flag that no longer represented the actual mode. That kept the UI aligned with the session contract: Live is real-time speech-to-speech; AI Voice is a text interview with TTS attached.
Provider fallback follows the same rule. OpenAI can fail and Gemini can remain available, but fallback should not create a different product. The relay can switch drivers only if transcript persistence, cancellation, opening behavior, and completion still look like the same interview flow.
That was the point of keeping OpenAI Realtime behind the existing relay rather than building a new browser path for it. The provider can change. The product boundary should not.
Adding a provider sounds like expansion. In practice, it tests whether the previous architecture was really an abstraction or just Gemini-specific code with nicer names. Focused tests could prove the new providers in isolation, but the full backend suite exposed old fixtures coupled to direct Gemini fetch calls and an earlier interview bootstrap lifecycle. The provider boundary was not real until those old assumptions were repaired.
The clearest lesson came from the three AI Voice opening bugs. No audio, duplicate bubble, and false Thinking... looked related because they all happened during the same opening turn. They actually belonged to three separate lifecycles: mode propagation and autoplay permission, transient stream cleanup, and pending assistant state.
Gemini remained available, OpenAI went behind the existing provider-neutral contracts, and the live relay continued to own the session. Text generation, persistence, and audio preparation now have separate lifecycles, so changing providers does not change the interview flow the learner sees.