Tag
1 post tagged with serving.
A system design practice note on separating real-time LLM serving from asynchronous feedback collection at 20K peak TPS