Code โบ tail-villain
Putting a Budget Around LLM Costs
Enforcing LLM cost limits in the backend and modeling their operational impact
An LLM feature can work perfectly and still be unsafe if the product does not know how expensive it can become.
Tail Villain talks a lot. It generates roadmaps, asks interview questions, evaluates answers, and writes study reports. The more a user practices, the more valuable the product becomes, but the number of calls and the amount of context also grow. If I simply attach a strong model and let every conversation expand naturally, cost can grow with usage in a way the product cannot absorb.
So the work from April 26 to April 28 was less about making the AI smarter and more about putting a budget boundary around it.
The first boundary was the token cap.
LLM calls cost money on both input and output. Output is especially easy to ignore because it feels like the model is just answering. But if the backend does not constrain response length, one turn can become much larger than expected. Repeated across users, that becomes operational risk.
I added required max output limits to backend LLM operations and put schema validation around the inputs. The important part is that this lived on the server. Frontend limits help the interface, but they are not financial controls. A user can bypass the browser, and frontend code can drift from backend behavior.
This is the same shape as authorization. Hiding a button is not the same as checking permission on the server. For LLM cost, a polite UI is not the same as a backend-enforced budget.
The second boundary was context length.
As an interview session gets longer, it is tempting to send the full conversation back to the model every time. More context feels like better questioning. The problem is that this makes each later turn more expensive than the previous one. The most engaged users become the most expensive users.
I added a sliding context window to the interview service. Recent messages are sent forward, while durable summary and session state carry the information that needs to survive longer. The product should not need to resend every raw message on every turn to keep the conversation coherent.
There is a trade-off. Trimming context can lose exact wording from earlier turns or make old contradictions harder to catch. But this was not blind cost cutting. It was a choice to keep long sessions viable. Persist what the product truly needs as state, and limit what every turn has to reread.
Once cost became a product concern, the admin surface mattered more.
On April 26, I added user management work alongside a cost simulator. User management is expected in an admin panel, but for an LLM product, a unit-economics view is closer to operational equipment. I needed to see which model was used, how input and output tokens affected cost, and what happened to margin as usage increased.
The early simulator had too much financial logic in the frontend. That makes the UI feel responsive, but it leaves business rules in the wrong place. Pricing, margin, and profitability calculations should not depend on client-side code that can drift or be tampered with.
On April 28, I moved financial calculation logic to the backend. The frontend collects inputs and renders results. The server owns the rules. I also added DTO validation and error handling so the simulator behaves like an admin tool backed by product policy, not just a local calculator.
The April 27 simulator refactor widened that boundary.
A simulator that assumes a few hardcoded vendors gets stale quickly. I removed vendor-specific filters from model synchronization, excluded free models, and made the model dropdown expose clearer pricing and profitability signals.
The point was not to show as many models as possible. The point was to make model choice a financial decision as well as a quality decision. Different models move output quality, latency, input cost, and output cost in different directions. The simulator should make those trade-offs visible instead of leaving them to instinct.
I also removed a redundant leaderboard-style table and added sorting controls where they helped the decision. Admin UI does not need to look impressive. It needs to make the next operational choice easier.
The simulator UX was tied to trust in the numbers.
If the cost screen flickers or lags in confusing ways, the numbers feel less reliable. I tightened state updates around the browser paint cycle and added debounced auto-simulation when inputs change. I also added a usage-rate slider because margin is not only about per-session cost. It changes with how many users actually consume the feature.
Loading skeletons and spinners look like small UI details, but they matter in an admin surface. The operator needs to know whether data is still loading, calculation failed, or the answer is truly zero. Ambiguous cost UI leads to ambiguous operational decisions.
This work made LLM cost control feel less like an accounting feature and more like part of the product architecture.
Token caps, context windows, model selection, simulator rules, and admin UX all shape how the product runs. The user should experience a natural conversation. Internally, the system still has to decide how much each turn may cost, how much context to carry, which model to use, and what those choices do to margin.
AI should feel magical to the user. The cost structure cannot be magical to the builder.
The real goal was not simply spending less. It was bringing LLM cost under product control.