Code › tail-villain
Building an Evaluator for Interviews
Evaluating full interview conversations with a typed LLM-as-a-Judge pipeline
After v1 went live, the next problem was not whether the interviewer could answer.
It was whether the interviewer was doing a good job.
A successful HTTP response and a visible follow-up question are weak evidence. The villain can still lose its persona halfway through a session. It can repeat itself. It can forget something the user said five turns ago. It can accept a contradiction that a real interviewer would immediately challenge.
Reading a few transcripts manually was enough while the product was small, but it would not scale once real sessions started accumulating. Interview quality needed a feedback loop inside the product.
The first version was an admin evaluation flow.
It selected interview sessions, sent the conversation to another LLM, and asked that model to judge the interviewer. This was LLM-as-a-Judge, but the target was not the candidate’s ability. The target was the AI interviewer itself: did it preserve context, challenge weak answers, stay in character, and produce feedback that matched the actual conversation?
The tempting approach was to clip or summarize the transcript. Long conversations cost more, and models can lose attention in the middle of a large context window. But clipping creates exactly the wrong failure mode for this product. Persona drift, repeated questions, and missed contradictions often happen in the middle of the session, not at the end.
So the evaluator received the full conversation.
Gemini Flash Lite made that practical enough: long context, low cost, and enough room to send the raw transcript instead of an edited summary. I added turn markers to each message so the judge could anchor its reasoning to specific points in the session. An interview is not judged by the last answer alone. It is judged by whether the interviewer remembers and uses what happened earlier.
The rubric could not be a loose prompt that returned a score.
Early runs produced the kind of contradiction that looks harmless in a demo and becomes painful in an operations tool: the same session receiving different scores, or a score of 5 with improvement points attached. If the score and evidence disagree, later prompt changes become hard to evaluate. You cannot tell whether the system improved or whether the judge simply changed its mood.
I tightened the JSON output order. The judge had to write step-by-step analysis first, then check for inconsistency, then decide whether a critical failure occurred, and only then choose a score. A score of 5 required zero gaps. If there were gaps, the score had to drop. If the interviewer looped, broke persona, or failed to recall context, the score was capped low regardless of surface-level strengths.
Sycophancy needed a specific rule too. If the user changes their story or gives an answer that conflicts with earlier statements, Kovill should push back. LLM interviewers can be too agreeable by default, so the evaluator had to check whether the interviewer challenged that contradiction. The interviewer prompt also gained a stronger directive to validate and then challenge inconsistent answers.
The useful part was not adding more negative instructions. It was turning the rubric into positive constraints. Five points requires no gaps. Critical failure caps the score. Contradiction requires challenge. Those rules are easier for the judge to follow than a long list of things not to do.
The admin dashboard also needed representative sampling.
Fetching the latest ten sessions would mostly show whatever users happened to do recently. If one session kind dominated that window, the dashboard would look healthy or broken based on user behavior rather than product quality. I changed the sampler to pull sessions across session kinds more evenly, then batched the judge calls in parallel so Evaluate Now did not become a long-running serial wait.
The UI moved toward an audit log instead of a set of decorative cards. Scores were color-coded. Critical failures were visually distinct. Empty strengths and gaps had explicit placeholders. The label changed from Critical Gaps to Improvement Points for stronger sessions, because a minor improvement note on a good interview should not read like a severe failure.
That gave me a practical loop: change the interview prompt, run the evaluator, compare the results, and inspect the underlying evidence.
The next day, a different bug exposed the same boundary problem.
The backend logs showed stamina values being prepared. The frontend received stamina as undefined. The mapping looked reasonable. The response shape looked reasonable. The missing link was the shared runtime schema.
The source Zod schema had been updated, but the compiled shared-types package was stale. Zod parse was silently stripping fields it did not recognize, so stamina, maxStamina, and reportStatus disappeared during response serialization. TypeScript looked fine in the editor, but the runtime schema was older than the source code.
That is a classic monorepo trap. Interfaces and runtime validators do not update together unless the build pipeline forces them to. I rebuilt the shared-types package and made the backend and frontend scripts build it before dev, start, and build flows.
The evaluator work had the same lesson in another form. A judge response is only useful if its schema is enforced. A session state is only useful if the frontend receives the fields the backend produced. The product cannot rely on intent crossing a boundary. It has to validate the shape that actually crosses it.
Building the evaluator changed the way I thought about the LLM stack.
The villain interviewer generates the pressure. The judge evaluates whether that pressure was coherent. The product stores the score and evidence. The admin dashboard turns it back into something a human can inspect. Every step needs a boundary, because every step can be plausible and still be wrong.
Interview quality does not become reliable because one prompt is good.
It becomes improvable when the product can measure the whole conversation, sample sessions fairly, enforce scoring rules, and keep typed data from disappearing between packages. The evaluator was not just a dashboard feature. It was the first piece of infrastructure that let me improve the interviewer without relying on vibes.