Code › tail-villain
Evaluating Only Observed Behavior
Separating transcript scope, eligibility, and evidence states so partial interviews are not scored as complete sessions
While re-running Tail Villain’s interview evaluation, I found a score that made no sense. The transcript contained only the interviewer’s opening question, but contextual follow-up received a 1 out of 5.
My first reaction was to blame the prompt. Maybe the judge was too strict. Then I looked at the transcript again and the problem became simpler: there was no candidate answer. If the candidate has not answered, the interviewer has not had a chance to follow up.
The system was treating unobserved behavior as bad behavior.
That is a different class of bug from a harsh rubric. A bad follow-up score should mean the interviewer responded poorly to something the candidate said. In this case, nothing had been said yet. The score was not describing interview quality. It was exposing a flaw in the evaluation contract.
A number was too small a shape
The earlier evaluator was built around numeric dimensions. Persona professionalism, question specificity, contextual follow-up, probing depth, personalization, grounding, coherence, repetition, and session focus could each become a 1-to-5 score.
That works better for completed sessions. Once a candidate has answered, the interviewer has asked follow-up questions, and several turns have passed, there is actual behavior to inspect. A judge can look at whether the interviewer stayed in character, followed the candidate’s reasoning, pressed on weak points, and avoided looping.
An opening-only transcript is different. The opening question can be evaluated. Follow-up quality cannot. Probing depth cannot. Repetition cannot, at least not in any meaningful way. Forcing every dimension into a number made the result look complete even when the transcript was not.
I changed the rubric so each dimension has one of three states. scored means the behavior is observable and can be rated from 1 to 5. insufficient_evidence means the dimension matters for this session, but the transcript does not yet contain enough behavior to judge it. not_applicable means the dimension does not apply to the available context.
Those two non-scored states are not softer versions of a low score. They mean different things. Insufficient evidence says, not yet. Not applicable says, not for this context. In an opening-only transcript, contextual follow-up, probing depth, and non-repetition should not quietly become failures just because they have no data.
Evaluation still needs to be strict. It just cannot invent evidence in order to be strict.
Eligibility belonged in the backend
At first, this looked like a prompt wording problem. Tell the judge not to evaluate follow-up when there is no candidate answer. Tell it not to penalize missing evidence. Add a few examples. Make the instruction louder.
But the backend already knew several facts more reliably than the judge could infer them. It knew whether the transcript contained candidate input. It knew whether that input was a substantive answer or just a clarification. It knew how many interviewer turns existed. It knew whether Goal or Background reference context was attached to the session.
Those facts became an eligibility contract before the judge ran.
If there is no interviewer turn after candidate input, contextual follow-up has insufficient evidence. If there is no substantive candidate answer, probing depth has insufficient evidence. If there are fewer than two interviewer turns, non-repetition has insufficient evidence. If no Goal or Background context exists, personalization and evidence grounding are not applicable.
The judge only receives dimensions that passed that backend contract. For dimensions decided deterministically, the response schema does not include an applicable field at all. The model does not get a second chance to contradict state that the program already established.
This was not about trusting the model less for everything. It was about giving the model the part of the job that actually needs judgment. The model can assess professionalism, specificity, coherence, and whether an interviewer meaningfully followed a candidate’s answer. It should not be asked to rediscover whether a candidate answer exists in the first place.
Opening, in progress, and complete are different claims
Partial transcripts are dangerous because their numbers can look too confident. A polished opening question can make an average look strong. Missing follow-up opportunities can make another transcript look weak. Neither number should be presented as whole-session quality.
I added explicit evaluation scope: opening, in_progress, and complete.
Opening evaluations are diagnostic. The evaluator can inspect the first question, but the overall score stays null. A good opening is useful evidence, not proof that the full interview would remain good.
In-progress sessions can receive a provisional score only after the required dialogue dimensions become observable. Until then, the evaluator keeps the aggregate empty rather than turning partial evidence into false precision. Completed sessions receive a final aggregate only when the required dimensions have been scored.
Critical failures also needed a separate rule. If the interviewer loops, contradicts the evidence, or otherwise fails in a way that undermines the session, the aggregate score cannot remain high just because several smaller dimensions looked acceptable. The backend can cap an otherwise available score at 2, and repeated loops can be detected deterministically even when the judge misses the failure flag.
After this change, the same numeric dimension means less without its scope. An opening evaluation is first-question diagnostics. An in-progress evaluation is a provisional signal. A completed evaluation is a final session judgment. They should not share the same interpretation just because they came from the same judge.
Personalization is not grounding
This work happened in the same production pass as RAG observability and landing-page messaging, but the important lesson here was not about dashboards or copy. Retrieval can show whether relevant context was found. The interview evaluator still needs to decide how that context affected the generated question.
I kept personalization and evidence grounding separate because they answer different questions.
Personalization asks whether available learner context was used when it would help the interview. If the system knows the candidate has a specific employer, project, or background, did the interviewer make the question more relevant because of it?
Evidence grounding asks whether candidate-specific factual claims are supported. If the interviewer says the candidate worked on a particular project or had a specific experience, that claim needs evidence. Failing to use relevant context may hurt personalization, but it is not automatically a grounding failure. Inventing a candidate fact is a grounding failure even when the turn sounds personalized.
The backend added deterministic support for high-confidence matches such as employer and project names. Broad technology or target-role terms do not count as deterministic evidence matches, because words like backend, SaaS, or a role title can appear naturally in a generic interview question.
That distinction also prevents not_applicable from becoming an escape hatch. If the interviewer makes an unsupported candidate-specific claim, grounding is applicable and should be scored poorly. The judge cannot hide that failure by saying grounding did not apply.
Evaluations needed transcript snapshots
One more issue showed up around historical results. A session can continue after an evaluation runs. If the UI later shows the current full transcript beside an old result, the evidence no longer matches the score.
That matters most for partial evaluations. Imagine an opening-only evaluation saved with no overall score. Later, the candidate answers and the interviewer follows up. If the old evaluation page displays the full current conversation, the reader sees evidence that did not exist when the judge ran. The score becomes hard to explain even if it was correct at the time.
The fix was to store the transcript boundary with the evaluation. Each LlmQualityEval records the evaluated message count, scope, rubric version, judge model, per-dimension evidence, scores, and critical-failure state. When the admin UI opens an old evaluation, it displays only the message prefix that existed at evaluation time.
That turns an evaluation into a snapshot instead of a floating interpretation of a growing session. The transcript can continue, but the evidence behind the old score does not move.
Targeted re-evaluation became part of the same cleanup. I should not need to create a new interview every time the rubric changes. Running a new rubric against an existing session makes the change easier to inspect, especially when the transcript snapshot is stable.
The rule I kept after this pass was simple: evaluate only observed behavior. A missing follow-up is not a bad follow-up when no answer exists. A dimension that does not apply is not a hidden zero. A score is useful only when it carries its evidence boundary with it.