Code › tail-villain
Profile Memory and Detail RAG
Separating stable learner memory from query-specific retrieval after correct chunks ranked first but missed the score threshold
Tail Villain’s Study coach had a strange failure mode. Retrieval found the right chunks. The learner’s background ranked first and second. No unrelated document beat it.
Then the threshold removed everything.
By the time the prompt reached the model, the retrieved context was empty, and the coach answered as if it knew nothing about the learner. At first, this looked like a normal RAG tuning problem: improve the query, contextualize the embedded documents, remove noisy persona names, or lower the score threshold. Each helped one slice. None fixed the contract I actually needed.
A Study coach should not remember the learner only when vector search clears a score gate.
The first fix was to separate search intent from lesson context
The next failure came from the query itself. In an IELTS Study session, the learner asked about their resume and career background. The retrieval query combined the Study topic, the topic description, and the latest learner message. The latest message pointed at Background, but the full query was pulled toward the IELTS Goal.
The result made sense after I looked at it. The Goal ranked high, while the correct Background chunks sat just under the admission threshold. The issue was not missing data. The search vector had been asked to represent too many jobs at once.
I changed retrieval to embed only the latest learner message. The Study topic and conversation state still belonged in the generation prompt. They did not need to distort the vector used to decide which private document to retrieve.
That improved explicit resume and career questions, but it did not solve broader profile questions. When the learner simply asked what the coach knew about them, the correct Background chunk ranked first and still scored below 0.65.
Ranking had succeeded, but context admission had failed.
Better embeddings helped, but conversation still drifted
Background chunks are often not self-describing. A resume paragraph may mention a project, a company, or a technology stack without saying that it is learner profile information. For embedding, that missing frame matters.
I added deterministic context to the document side before embedding Background chunks. The stored source text stayed unchanged, but the embedding text described the content as learner profile, resume, career, skill, education, project, and prior-experience information.
Generic profile questions moved above the threshold. Explicit Background questions became more stable. Unrelated questions stayed below the cutoff.
Then a persona name exposed another edge. A learner might address Socrates directly before asking a personal question. When that vocative entered the retrieval query, the Background scores dropped. Removing the selected persona name from the retrieval query raised the scores again while preserving the original message for storage and generation.
That kind of fix is tempting because it is visible and concrete, but real conversation is not a clean search box. A learner can mention a friend, laugh, address the persona, and only then ask what the coach knows about them. With that more realistic message, the correct Background chunks still ranked first and second, but both fell below the threshold again.
Rewriting the query into an explicit retrieval intent recovered the same chunks with much stronger scores. That proved the documents were retrievable. It also proved the category problem. A single query vector was being asked to compress filler, names, social context, the persona vocative, and the actual profile question into one representation.
A rewrite call on every Study turn would reduce some of that drift, but it would add latency, cost, and another failure mode. More importantly, it would still treat stable learner identity as optional search evidence.
Lowering the threshold solved one case and broke another
The obvious alternative was to lower the score threshold. If 0.65 removed the right chunks, maybe 0.60 or 0.55 was a better production value.
I measured that instead of guessing. For the long conversational profile question, lowering the threshold admitted part of the Background at 0.60. Lowering it further admitted both Background chunks and the Goal. On that case, the change looked attractive.
But the same threshold also applies to vague turns. For a question like why is this necessary, both Background and Goal produced scores that were plausible enough to enter at lower cutoffs. That would give the coach private learner context even when the current question did not clearly ask for it.
This is not just answer noise. Goal and Background data can include sensitive personal context. Injecting that material into unrelated turns increases distracting personalization, unsupported assumptions, and accidental exposure. The final generation model is not a guaranteed relevance filter.
So I kept the 0.65 threshold as a detail-evidence noise guard. The important change was to stop using that threshold as the memory contract.
RAG could decide whether a specific chunk was safe enough to add for this turn. It should not decide whether the coach knows the learner at all.
Stable profile memory became a separate layer
Tail Villain already creates a Roadmap analysis from the learner’s Goal and Background. Instead of relying on vector search to rediscover the learner every turn, I built a bounded learner profile from that analysis and made it always available to the coach.
The profile is not an unlimited memory dump. It contains a compact overview, target direction, strengths, gaps, focus areas, representative projects, selected experience, and requirements. It gives the coach enough stable context to avoid sounding blank, while keeping detailed evidence out unless the current question calls for it.
Then query-specific RAG sits beside it. If the learner asks a broad question like what do you know about me, the stable profile can answer at the level of target, strengths, gaps, and project summaries even when retrieval returns no chunks. If the learner asks what caused duplicate processing in a particular project and how they fixed it, the profile is not enough. Detail RAG needs to retrieve the source chunk that contains the actual incident and mechanism.
That distinction made the product behavior easier to reason about. Profile memory answers who this learner is broadly. Detail RAG answers which precise source fact is useful right now.
Both layers are still untrusted reference data. The profile is derived from user-provided material, and retrieved chunks come from user-controlled documents. The model may use facts from them, but it must not follow embedded instructions or expose hidden labels, scores, and source metadata.
Study, text Interview, and Live Interview needed the same contract
Once the profile existed, the next issue was consistency across product surfaces. Study had depended heavily on RAG. Text Interview already used analysis summaries and evidence hooks, but each prompt surface formatted them differently. Live Interview had its own initialization path, creating another chance for evidence to disappear between text and voice modes.
I moved the learner profile into a shared builder and used the same bounded shape across Study, streamed Study, Interview openings, follow-up questions, angle transitions, completion reports, and Live Interview setup.
That part matters more than it sounds. Tail Villain is a general-purpose practice platform. A learner may move from reading a Study explanation to answering an interviewer out loud. The system should not know a different person just because the delivery mode changed.
The profile also gives future evaluation a clearer boundary. If retrieval supplies zero chunks, the coach still has stable profile context. If retrieval supplies chunks, the prompt has profile plus raw detail evidence. The user-visible answer should not expose those internals, so debugging has to use backend events.
Memory and retrieval should not be the same promise
The useful lesson from this work was not that 0.65 is the right threshold. It is only a measured noise guard for the current Roadmap-scoped detail retrieval. Cosine scores are not calibrated truth, and they are fragile around conversational phrasing, mixed intent, multilingual text, and chunks that need framing before they are semantically legible.
The more important lesson was architectural. Stable learner identity and precise evidence retrieval have different failure modes. If they share the same gate, one vague sentence can make the coach forget the learner. If the gate is lowered too far, unrelated turns receive private detail they did not need.
Separating profile memory from detail RAG gives each layer a narrower job. The profile stays always on, bounded, and general. RAG stays selective, thresholded, and evidence-oriented. The coach can remember the learner without flooding every prompt with raw documents, and still pull exact details when the current question requires them.
There is still a boundary to revisit. The current profile includes enough project problem and solution text that a response can look detail-grounded even when RAG supplied nothing. I may need to make the always-on profile smaller and push more project-specific mechanism back into retrieval.
Personalization no longer depends on whether retrieval happens to clear a score threshold. A small profile keeps the learner’s broad context available, while RAG retrieves deeper evidence only when the current question needs it.