Code › tail-villain
When a Prompt Refactor Broke the Engine
Recovering interview control flow lost during a prompt and grounding refactor
I was trying to make the interview prompt smaller, and I removed behavior the engine still needed.
The original goal was reasonable. Tail Villain needed better grounding. The villain should not ask follow-up questions from vague assumptions about the user. It should press on the resume, projects, and experience the user actually provided.
So I refactored the evidence hook structure. Then I tried to optimize the prompt at the same time, and the interview engine started losing pieces of itself.
An interview session in Tail Villain is not just a single prompt. The user answers, the villain stays in persona, the system decides what kind of follow-up should come next, evaluation signals are interpreted, and structured output has to be parsed safely. The visible part is chat, but the working part is closer to a small engine wrapped around an LLM.
The first change that day was about grounding. The old evidence hooks helped the interviewer refer to the user’s background, but the structure mixed different kinds of context. A detailed project and a general career summary should not carry the same weight in a follow-up question.
I split the shape into richer project context and broader experience context. Specific projects could carry concrete details. General experience could stay as a summary of the user’s background. When the villain drilled into a project, it would have a better chance of asking from the right evidence instead of inventing a plausible but unearned angle.
That part was the right direction. In an interview product, bad grounding is worse than a bland question. If the AI presses on something the user never actually did, the pressure stops feeling useful and starts feeling arbitrary.
I also added stability around LLM calls. Session startup could hit a temporary 503 from Gemini Flash. That is not the kind of failure a user should experience as a dead start if a short retry would recover it.
A retry with backoff made sense there. It did not make the model more reliable, but it made the product less brittle around transient demand spikes.
So far, the work was defensive. Better evidence boundaries. Safer calls to an external model.
Then I tried to make the prompts leaner.
The refactor moved toward layered system prompts: fewer repeated instructions, clearer no-assumption grounding rules, and less prompt bulk. That was a real concern. Long prompts cost more, take longer, and can bury the instruction that actually matters.
But the prompt did not contain only tone and style. Some of it was engine behavior. Flow control, JSON parsing expectations, evaluation handling, and persona-specific interview rules were sitting inside the same natural-language layer. During the cleanup, lines that looked like duplication were actually part of the control surface.
The prompt got shorter. The engine got weaker.
This kind of break is awkward because it does not always crash. The AI still says something. The UI still receives a response. From a distance, the session looks alive.
But the behavior changes. The interview flow is less controlled. Evaluation handling becomes less reliable. The persona can drift because the instructions that held it in place were treated like prose cleanup instead of product logic.
That is the dangerous part of prompt optimization. In an LLM system, some sentences are copy, some are policy, and some are executable behavior in all but name. Natural language makes those categories look the same until one of them disappears.
I rolled the prompt work back to the functional baseline first. Then I restored the useful parts surgically: the improved evidence structure stayed, the no-assumption grounding rule stayed, but the persona-based engine instructions and evaluation-handling behavior had to come back.
I also cleaned up the type boundary. After the schema changed, forcing mismatches through unsafe casts would have made the next failure harder to diagnose. Zod parsing and typed accessors made the structure explicit again. If the data shape breaks, I want that failure to be visible instead of hidden behind a cast.
The lesson was not that prompt optimization is bad. Tail Villain calls the model repeatedly during an interview, so response time and cost both matter. A prompt that grows forever becomes its own maintenance problem.
The mistake was treating all prompt text as equally removable.
Some lines shape the voice. Some lines protect grounding. Some lines preserve engine behavior. If I remove all three categories with the same cleanup motion, the result can look cleaner while the product becomes less correct.
After this, I started reading prompt diffs with a different question: is this line style, evidence policy, or engine control?
Only after that question is answered does deletion make sense.