Code › tail-villain
Validating RAG Beyond Retrieval
Building a source-scoped RAG pipeline and validating its effect on real Study responses
Tail Villain’s Study responses needed access to the learner’s goals and background. The same topic should lead to different explanations and exercises for someone preparing for JLPT N2, someone struggling with articles in IELTS Writing Task 2, and someone reviewing a production incident from their own work.
That required retrieval, but retrieving a relevant document and putting it into a prompt did not prove that personalization worked. I needed separate evidence that the ranking was correct, tenant data stayed isolated, the generated answer actually used the retrieved facts, and the lesson continued in the right direction.
I split the implementation and evaluation along those boundaries.
Proving retrieval first
The source material came from the Goal and Background attached to a learner’s Roadmap. The original text became a KnowledgeDocument, while retrieval-sized pieces became KnowledgeChunks. Vertex AI produced a 768-dimensional embedding for each chunk, and PostgreSQL stored those vectors through pgvector.
The complete path was explicit enough to inspect one stage at a time.
Goal / Background
-> document persistence
-> paragraph- and sentence-aware chunking
-> document embedding
-> pgvector persistence
-> query embedding
-> cosine-distance search
-> retrieved context in the Study promptSource persistence and embedding generation had different failure contracts. Saving a Goal or Background is core product behavior, so it must not fail because an external embedding provider is temporarily unavailable. The source and its chunks remain stored, while missing embeddings can be retried later.
Ownership checks also belonged inside retrieval rather than at the UI boundary. A result had to match the owner on both the KnowledgeChunk and its joined KnowledgeDocument. These documents can contain private goals, resumes, or work history, so owner scoping is a data-isolation requirement before it is a ranking concern.
The first end-to-end smoke used two synthetic owners. Owner A had an IELTS Goal and a speaking Background. Owner B had an unrelated AWS Goal. Queries from Owner A ranked the correct Goal or Background first, and Owner B’s data never appeared. The script then removed the synthetic users and every derived knowledge row.
That proved the real chain: source services, Vertex embeddings, vector persistence, cosine search, owner isolation, and cleanup.
Refusing to turn one score into truth
The retrieval service exposed a score derived from cosine distance. It used one minus the distance, but that number was not a probability that the result was correct. A score of 0.65 could behave differently across languages, document structures, and query specificity.
The first labeled set contained ten cases. At a threshold of 0.65, precision and recall were both 90%. I then froze that value and ran it against a separate ten-case holdout. Precision dropped to 81.8%, even though top-one accuracy, hit@3, and mean reciprocal rank remained perfect.
The distinction mattered. Ranking still placed a relevant source first, but semantically adjacent noise also crossed the threshold. In another case, a useful cross-language Background scored 0.62 and was excluded. Raising the threshold would remove more noise by removing useful context at the same time.
Instead of changing the model or tuning the number again, I narrowed the candidate set. A Study session already knows which Goal and Background belong to its Roadmap. Searching those source IDs first prevents unrelated documents from competing, even when they are semantically similar.
An independent source-scoped holdout produced 90.9% precision and 100% recall. Cross-language facts and late facts from a second chunk remained retrievable, while another owner’s exact semantic match remained inaccessible.
This still did not make 0.65 a universal threshold. It established a narrower conclusion: restricting retrieval to the Roadmap’s linked sources reduced noise without sacrificing the useful context in that evaluation.
Retrieval success was not answer success
Once the retrieval path passed independently, I tested it through the authenticated Study API. Each pair used the same topic, persona, and learner message. The only difference was whether the Roadmap linked the Goal and Background.
Five topics produced ten real Study requests.
| Check | RAG on | RAG off |
|---|---|---|
| Responses containing at least one predefined personal signal | 5/5 | 0/5 |
| Total exact personal-signal matches | 12 | 0 |
| Responses exposing retrieval mechanics | 0 | 0 |
The RAG-enabled responses used concrete learner facts: a Kafka trace-context failure, a previous JLPT N3 result, IELTS article errors, PMP CPI and SPI, and duplicate records involving Airflow and dbt. The unlinked responses started with generic discovery questions because they had none of that context.
There was a measurable cost in this run. RAG-enabled requests took about 1.1 seconds longer on average and used roughly 133 additional input tokens per response. Each condition only ran once per case, so those numbers were observations, not stable performance estimates. They still confirmed that personalization adds retrieval latency and prompt context.
Then the IELTS response exposed the next failure boundary. It correctly mentioned Writing Task 2 and the learner’s article errors, but ended by asking about spoken English. Retrieval had succeeded. Grounding had succeeded. The teaching direction had not.
I added a separate six-point answer rubric: two points for grounding in known learner facts, two for alignment with the fixed first-step direction, and two for giving a concrete exercise the learner could perform immediately. RAG responses averaged 3.4 out of 6, compared with 1.0 without RAG. Yet only one of the five RAG responses passed the strict quality bar.
Retrieval created personalization. It did not automatically create a good lesson.
Making failures diagnosable
Connecting pgvector was not the hardest part of this work. The difficult part was defining enough evaluation boundaries to identify which layer had failed.
When retrieval returns the wrong source, the embedding path, candidate scope, and ranking need investigation. When retrieval is correct but the facts never reach the answer, the prompt-injection path is the next boundary. When the facts appear but the response chooses the wrong exercise or question, the generation policy and Study progression rules need to change.
Calling all three cases RAG quality would hide the repair point. I kept retrieval evaluation, grounding evidence, and instructional-quality evaluation separate, then compared fixed cases and scoring criteria instead of relying on a general impression that one answer sounded better.
RAG in Tail Villain is not a mechanism for putting as much learner data as possible into a prompt. Its job is to retrieve the Goal and Background relevant to the current Study session, preserve the ownership boundary, and turn that context into the correct next explanation or exercise.
The July 10 work established that retrieval could carry private learner context into a real personalized response. Turning that evidence into consistently useful teaching behavior remained a separate problem, which was exactly what the evaluation was designed to reveal.