Separating LLM Serving from the Feedback Pipeline

I recently worked through a system design exercise for a global LLM serving architecture. The user was a developer using a coding agent, and the request flow covered prompt input, LLM inference, response streaming, tool calling, conversation history, and feedback collection.

I used 20K peak TPS as the scale assumption. The exact number matters less than the design pressure it creates: at that scale, the LLM inference path and the log or feedback collection path should not be treated as the same kind of workload. The goal was to keep TTFT, or Time To First Token, under 1500 ms while still collecting conversation history and feedback for later training or analysis. Since this was a system design exercise, I assumed the session and cache layers would be designed explicitly instead of being handed off to a fully managed data layer.

I am not going deep into GPU sizing or token throughput per model in this post. In a real design, that would require the model size, average prompt length, average output length, GPU type, and continuous batching efficiency. Here, I focused on the request path and where the queue should sit.

Problem definition

The requirements can be summarized like this.

- Global users use an LLM-based coding agent
- Peak traffic is around 20K TPS
- The first token should be returned in under 1500 ms
- Responses are streamed over WebSocket or SSE
- Prompts, responses, tool calls, feedback, and token usage are collected
- Collected data later moves into a training or analytics pipeline
- Global sessions and conversation history must be maintained
- Session and cache layers are designed explicitly for the exercise

At first, I thought of it like a familiar large-scale traffic problem. Put an API gateway at the front, send the request through the main API server, enqueue the request in SQS, and let LLM workers consume it. That structure came to mind quickly, but it misses an important property of LLM serving: the user is not waiting for a background job to finish later. They are waiting on an open session for the first token to start flowing back as soon as possible.

Where the queue belongs

For LLM chat serving, the first thing to notice is that the user is waiting for a streamed response over an open connection. Before the first token appears, the user has very little signal that the system is actually doing work. If that gap gets long, the perceived latency becomes high even if the full answer eventually completes.

If the inference request itself goes through an asynchronous queue such as SQS, queueing latency becomes part of the user-facing response path. With WebSocket or SSE, the system also needs to keep a real-time mapping between the open client connection and the token stream coming back from the worker. A worker that eventually picks up a message from a queue is a good fit for background processing, but not for a serving path that needs to show the first token within 1500 ms.

So the design separates the paths.

• Fast Path: user request -> session lookup -> LLM worker call -> token streaming
• Slow Path: logs, feedback, and token usage -> queue -> consumer -> data lake

Overall architecture

The simplified architecture looks like this.

Instead of forcing group boxes into the diagram, I kept the whole flow visible in one graph. Blue nodes are the real-time serving path where the user is waiting for the first token. Orange nodes are the asynchronous path for logs and feedback. Purple nodes are the storage path for sessions and conversation history.

The gateway layer handles authentication, rate limiting, and routing. In a real system, this could be split across API Gateway, ALB, CloudFront, or a WebSocket gateway. For this post, I treat it as the entry layer that accepts external requests and forwards them to the main API server.

The main API server can run in a stateless environment such as ECS Fargate. It does not run model inference directly. Its job is to keep the user connection open, load session and conversation context, forward the request to an LLM worker, and stream generated tokens back to the user.

The LLM worker layer belongs on a separate GPU cluster, such as EC2 GPU instances. If the workers use a serving engine such as vLLM or TensorRT-LLM, continuous batching and KV cache management become important. But even if the worker layer is optimized well, a long queueing delay in front of it can still break the TTFT target.

Fast Path

The Fast Path is the part the user directly feels.

User
-> Gateway
-> Main API Server
-> LLM Worker
-> Main API Server
-> User

The main API server should stay relatively light on this path. It handles authentication and authorization checks, rate limiting, session lookup, request validation, and the streaming connection. Conversation context should be read from a local cache first, with a fallback to the persistent store when the cache misses.

For communication with the LLM worker, I would avoid creating a heavy new connection for every request if possible. A persistent internal connection or gRPC streaming path is a better mental model. As soon as the worker produces the first token, the API server should pass it back to the user.

Several things have to line up to meet the TTFT target.

- Remove unnecessary waiting in the gateway and API server
- Keep session lookup fast
- Minimize network hops to the LLM worker
- Watch worker queue depth and saturation
- Limit prompt and context length
- Use continuous batching and KV cache management inside the worker layer

In other words, keeping TTFT under 1500 ms is not only a matter of running more servers. The design has to keep checking where the request waits on the user-facing path.

Slow Path

The Slow Path is the data collection path that the user does not need to wait for.

Main API Server
-> SQS
-> Feedback Consumer
-> S3 Data Lake

This path receives prompts, generated responses, tool call records, token usage, latency, error information, and user feedback. The data matters for model improvement, product analytics, cost monitoring, and incident analysis, but it should not block the first token.

The main API server can emit log events while it streams the response. Those events go into SQS, and a consumer normalizes them before storing them in something like an S3 data lake.

Queue depth is still an important signal on this path. If the queue backs up, the real-time response path should keep working, but feedback data will arrive late and incident analysis may also lag. The point of separating this path is to keep that delay out of the user’s TTFT.

Session and conversation history

Sessions and conversation history are also central to LLM serving. A coding agent is not just answering one-off questions. It needs to carry previous prompts and responses, tool call results, and the current working context across turns.

The simplest shape is to combine a regional Redis cache with a persistent store. The store is not just a narrow session table. It is closer to a persistent store for conversation history and session metadata. In a real implementation, that could be PostgreSQL, MySQL, or a distributed key-value store, but the important part here is the role split between cache and durable storage.

- Regional Redis Cluster: cache for recent conversation context
- Persistent Store: conversation history and session metadata
- Cross-region replication: conversation history replicated across regions

If the user stays in the same region, the API server can read recent context from local Redis, write the completed turn to the persistent store, and replicate it asynchronously to other regions.

Failover is the difficult case. If a user moves from region A to region B after an outage, region B may not have the session in its local Redis cache. In that case, the API server needs a fallback that reads from the latest original store or the freshest replica, then rebuilds the regional Redis cache for later requests.

That creates a trade-off. Asynchronous replication keeps the serving path fast, but it introduces replication lag. Strong synchronous replication improves consistency, but it can increase global latency. Under these requirements, local cache plus asynchronous replication with a failover fallback looks more realistic.

Monitoring points

I kept monitoring scoped to the requirements. In a global distributed system, at minimum I would watch latency, error rate, token usage, and queue length.

- latency
- error rate
- token usage
- queue length

Latency should not be reduced to a single average. On the user-facing path, TTFT is the key metric. On the feedback collection path, SQS queue length is the more direct signal. Token usage connects to cost and capacity planning, while error rate should be separated across the API server, LLM worker, and feedback consumer so the failure domain is visible.

Takeaway

The design became clearer once I treated queue placement as part of the latency boundary.

Real-time inference stays on the Fast Path, while logs and feedback move to the Slow Path. SQS acts as a buffer for data collection and model improvement. User-facing inference requests should not wait there.

To satisfy the TTFT requirement, the path the user waits on has to stay short. The API server needs to load the session quickly, forward the request to the LLM worker, and stream the first token back over WebSocket or SSE as soon as possible. Logs and feedback are important, but if they block the first token, they are competing with the purpose of the real-time serving path.