Code › tail-villain
Observing RAG Without Private Text
Adding privacy-safe structural telemetry for production RAG calls, admission, scores, latency, and source mix
Tail Villain already had RAG in the production path. Study responses could retrieve the learner’s Goal and Background before generating an explanation or exercise. Text Interview turns could use the same evidence before asking a more personal follow-up question. The previous round had proved the basics: retrieval worked, owner scoping held, and personalized signals appeared in real responses.
Then the operational gap showed up. I could not easily answer how often RAG was being called, how many vector candidates it found, how many of those candidates passed the admission threshold, or whether the slow cases came from Study or Interview. The feature existed, but the retrieval layer was mostly invisible once it reached production.
That invisibility was not just inconvenient. RAG sits next to private learner text. Goals, resumes, project history, interview answers, and weak spots are exactly the material that makes personalization useful. Storing those strings again inside logs would turn observability into a second copy of sensitive data.
So the work became less about seeing more text and more about seeing the right structure without text.
The useful signals were structural
The first questions I needed to answer were straightforward. Was retrieval called or skipped? If it was called, did pgvector return successfully? How many candidates came back? How many were admitted into the prompt context after minScore filtering? What was the top candidate score? What kind of source was admitted, Goal or Background? How long did the retrieval step take?
None of those questions required the learner’s actual query. I did not need the resume sentence to know the candidate count. I did not need the retrieved snippet to measure duration. I did not need a source title or document ID to see whether admitted context mostly came from Goals or Backgrounds. A score distribution can be useful without storing the text behind each score.
Retrieval now writes a separate usage-log event with operation rag_retrieval and provider pgvector. It is not an LLM call, but the existing usage log already collects operational events by time, status, and operation. Keeping retrieval there also made it possible to inspect it beside embedding usage without mixing the two.
The event records whether retrieval was called, why it was skipped, which surface invoked it, whether it succeeded or errored, total duration, configured topK and minScore, candidate count, admitted count, top candidate score, admitted rank, admitted source kind, and admitted score. The surface matters because Study and Interview use the same retrieval layer for different product jobs. Study needs a next explanation or exercise. Interview needs a pressure-tested follow-up question.
The event deliberately does not record the learner query text, source title, source ID, document ID, chunk ID, or retrieved snippet text. Correlation identifiers can connect operational events, but the admin view should not become a place where private learner prose is readable.
That boundary was the point. Observability should help reduce operational uncertainty. It should not silently create a new private-data store.
Retrieval needed its own success record
Without retrieval telemetry, it is tempting to infer the retrieval state from the generated answer. If the answer is generic, maybe RAG was skipped. Maybe there were candidates but none passed the threshold. Maybe context entered the prompt and the model ignored it. Maybe the model used it, but then chose the wrong teaching move.
Those are different failures. They need different fixes.
The retrieval event is written independently from the final generated response. If retrieval is skipped, the skip reason is recorded. If pgvector fails, the error state is recorded. If there are candidates but no admitted context, the gap between candidate count and admitted count remains visible.
Admission count turned out to be one of the most useful operational numbers. A candidate count of ten does not mean the prompt received ten useful facts. It only means the vector search found possible neighbors. If admitted count is zero, the learner receives a response without retrieved context. If admitted count is consistently high, the prompt may be carrying extra cost and noise. Looking at candidate and admission counts together gives a clearer handle on source scope and threshold behavior.
Scores also needed separation. The top candidate score describes the nearest result in the search space. The admitted score describes evidence that actually entered the prompt. When those diverge, I can see whether the system is finding nearby material but rejecting it, or admitting low-confidence context.
Duration belongs in the same structural event. In Study, extra retrieval time makes the learner wait longer for an explanation. In Interview, latency can break the pressure of the exchange before the next question arrives. Measuring pgvector retrieval duration separately from model generation latency makes the boundary visible.
Telemetry persistence is best effort. A logging failure should not break a learner’s Study answer or Interview turn. If retrieval and generation succeed, the user gets the response.
Aggregates beat raw private evidence
Raw events are useful for debugging, but an admin dashboard needs distribution, not a pile of rows. I added a RAG metrics endpoint under the Admin Eval area to aggregate recent retrieval events. The default window is seven days, the requested range is clamped from one to ninety days, and processing is bounded to the latest 5,000 rows.
The dashboard reports retrieval hit rate, called count, skipped count, error count, P50 and P95 duration, average top score, admitted source-kind counts, and per-surface breakdowns. That is enough to answer the first operational questions without exposing what anyone wrote.
If Study has a low hit rate while Interview looks healthy, the next place to inspect is the Study-side wiring or its injection conditions. If both surfaces call retrieval but admitted count is low, the threshold, candidate scope, or query embedding path becomes more interesting. If P95 duration climbs, the vector query, search scope, and data size need attention. If Background is rarely admitted compared with Goal, then the system may be underusing the learner’s experience context.
None of those investigations start by reading a resume. They start from counts, scores, source kinds, and durations.
This dashboard does not prove answer quality. The earlier RAG work already showed that a response can use the right learner facts and still choose the wrong next teaching action. Retrieval can succeed while the Study lesson drifts. It can also succeed while an Interview question stays too generic or fails to probe deeply enough.
The dashboard’s job is narrower. It tells me whether the retrieval layer is being called, whether it is admitting evidence, what kind of evidence it admits, and how much latency it adds. Once that layer looks healthy, quality work can move to prompt policy and session progression. If the layer is unhealthy, tuning a prompt is the wrong first move.
That separation matters in production because generic symptoms tend to collapse into one bucket. If a user says the answer felt generic, structural RAG metrics answer the first question: did the system retrieve and admit context for this surface at all?
Privacy changed the debugging posture
The easiest debugging system would store the failed query, every candidate snippet, and the final admitted context. That would make individual failures convenient to inspect. It would also preserve exactly the text Tail Villain should be most careful with.
The source material for RAG is not abstract documentation. It is the learner’s goal, background, work history, and sometimes a private story about why an interview is difficult. In Interview, the text can get more personal because candidates are practicing under pressure and revealing gaps.
So I accepted a less convenient system. Candidate count, admitted count, top score, admitted scores, source kind, duration, and surface are enough for the first production dashboard. They do not explain every single retrieval miss, but they reveal the shape of the system without making private text readable.
There is a trade-off here. If one exact query fails, I cannot open the dashboard and read the query or snippet directly. Reproducing that case requires a narrower path with the right authority and probably a synthetic or user-approved fixture.
It is also the safer default. Logs tend to outlive the debugging session that created them. Once private text enters that layer, removing it later is harder than never storing it there.
This work did not make the retrieval model smarter. It did not change the core ranking strategy or guarantee that a personalized answer would be good. What it changed was the operational visibility of the RAG layer: calls, skips, errors, candidate counts, admission counts, score distributions, source mix, duration, and the difference between Study and Interview.
For Tail Villain, personalization should not mean looking at more private text by default. It should mean safely bringing the necessary structure into the right moment. The July 16 RAG work extended that rule into observability: watch the system closely, but do not turn user words into operational exhaust.