· Amol Umbarkar
How we serve tokens reliably at scale
The architectural decisions behind BlazeInfer's reliability — two-layer routing, GPU-pinned fleets and model-aware monitoring — and why redundancy sits at the heart of all of them.
In a cloud system, nothing stays still. The software changes, configs get adjusted, and hardware is rotated in and out. Serving tokens reliably in that environment comes down to a handful of architectural decisions, and at the heart of most of them is one tool: redundancy.
This post walks through three of those decisions — routing, the GPU fleet and monitoring — and why we made them the way we did.
Routing
The router sits in front of every request, which makes it an easy single point of failure.
There are several open-source inference routers, and in our experience each does part of the job well. Some are good at prefix-aware routing; others handle sessions and load balancing well. Some are being rewritten from Python to Rust. The category is changing quickly, and that churn tends to make an installation unstable.
What an inference router needs to do
An ideal inference router makes its decisions hierarchically, cheapest check first:
- If the session is already known, it doesn't need to compute prefix-token overlap at all.
- It enforces affinity limits, so one busy session can't overload a single worker.
- It notices when a pod is serving slowly because its GPU is running hot.
- It knows where to send long prompts and where to send short ones.
Much of this is specific to the environment and the workloads it serves. So rather than learning to work around pre-built routers — which often add extra network hops — we built a simple router in-house, shaped around our environment and what we know about our users' workloads. With today's coding tools, building this layer is not that hard. Owning it also let us integrate tools like NVIDIA NeMo Switchyard for model selection.
Two layers of routing
Our routing happens in two layers.
Client request
Any OpenAI-compatible SDK or agent
Central router · runs at the edge
Validates the request and picks a healthy fleet that meets the latency objective. Scales horizontally, deployed blue-green, can run in multiple regions.
Fleet router · fleet A
Picks the worker. Tracks cached prefixes per worker.
Fleet router · fleet B
Picks the worker. Tracks cached prefixes per worker.
Each worker is pinned to its own GPUs, so a hot GPU can be taken out on its own.
The central router checks that a request is valid, then decides which fleet to send it to. Often several fleets serve the same model, so the router can choose among them against a latency objective: it tracks how quickly each one is responding — time to first token and time per token — and sends the request to one that is meeting the target.
Alongside latency, the central router keeps a view of each fleet's health: whether it is reachable, its error rate, and how much capacity it has left. A fleet that is degraded or failing is taken out of rotation and its traffic shifts to healthy fleets, so no single fleet is a single point of failure. The same health checks work when fleets are spread across regions, so the router can evaluate and fail over between fleets in a multi-region setup too.
The central router is built to run at the edge: it can be deployed in multiple regions, can sit in the same cluster as a fleet, and scales horizontally. Config changes and deployments roll out blue-green: the new version comes up alongside the one serving traffic, traffic moves over only once it checks out, and rolling back is just switching traffic back.
The fleet router receives the request and picks the worker that will run it. It also scales horizontally, and uses NVIDIA Dynamo's Flash Indexer to track which prefixes are cached on which worker. It makes its decision step by step and stops as soon as it can:
- If overall load is low and the request belongs to a known session, it goes to that session's worker. That worker already holds the conversation's KV cache, so no prefix lookup is needed.
- Each worker has a session-affinity maximum. When sticky sessions keep a pod past that limit, the pod is running hot, and new requests are sent to another pod instead of queueing behind it.
- Otherwise the router weighs prefix overlap against load and picks the better of two candidates (power-of-two choices). Over time this keeps a hot prefix cached on two pods. When one gets busy, the other can take the request — even if it holds only part of the latest prompt, it reuses that part and recomputes only the rest.
- 1yes →
Is load low and the session already known?
Yes: Send it to the session's worker. Its KV cache is already there.
- 2yes →
Is that worker over its session-affinity limit?
Yes: It's running hot. Send the request to another worker.
- 3then →
Otherwise: which two workers hold the most of this prefix?
Then: Pick the less loaded of the two. A partial prefix match is still reused.
The fleet
You can't build a reliable system without assuming the hardware will fail at some point, and with GPUs that is more likely than most.
On Kubernetes, the norm is one Deployment with several replicas. We've found replicas to be a problem for inference, because a Deployment treats them as interchangeable. If one GPU runs hot, there's no clean way to take that specific replica out: delete its pod and the controller recreates it, and scaling down doesn't let you choose which replica goes.
So we run multiple Deployments instead. Each one pins its worker to specific GPUs, and any one can be shut down quickly to take it out of the work queue. On top of that, the fleet router sends fewer requests to a worker that is performing worse than its peers.
Running several fleets for the same model gives us redundancy in software as well as hardware. Each fleet can run a different configuration — a different serving stack such as Dynamo, SGLang or vLLM, or the same stack with different tuning parameters — and they all serve traffic in parallel. If one configuration or stack fails, say after an upgrade or on a request pattern that trips a bug, the central router moves traffic to the other fleets and we scale them up to take the load.
This matters because inference engines change constantly, with new releases, new features and new model support arriving every few weeks. With multiple fleets, a new engine version comes up as its own fleet alongside the current one. We send it a share of traffic, watch how it behaves, and move more over as it proves itself. If something goes wrong, rolling back is just shifting traffic back to the fleet that was already running.
Taking a pod out shouldn't cost its users their context either. A pod's KV cache normally lives in GPU memory and disappears when the pod restarts, so every conversation it held would have to be prefilled again from scratch. We offload KV cache to a distributed store, so a restarted or replacement pod can fetch a session's cached context instead of recomputing it.
The fleet router also places workers on a hash ring. When a pod drops out, only the sessions it owned move — to the next pod on the ring — and every other session stays where it is. Combined with the offloaded cache, a session moves to a new pod and picks up roughly where it left off.
Monitoring
We have been running inference in production for several years. There are plenty of good monitoring tools; the hard part is deciding what to monitor, and that keeps changing.
Our dashboards track and alert on the key abnormalities in the central and fleet routers and in the worker pods, whichever engine or serving setup they run — Dynamo, SGLang, prefill-decode disaggregation and so on.
GPU health needs a different approach. Temperature and power on their own say little: a GPU running a heavy prefill workload will run hotter and draw more power than one serving light traffic, and both can be perfectly healthy. A reading only means something next to other GPUs under the same conditions. So we group GPUs running the same workload, track the group's average temperature and power and the standard deviation around it, and flag any GPU that drifts well outside its peers. That is usually the first sign of a failing GPU, often well before it throws errors.
Over time we've learned that monitoring also has to be specific to the model architecture. Recently we saw spikes in request evictions on Gemma while KV cache usage looked low. Gemma mixes sliding window attention (SWA) layers with full-attention layers, and inference engines keep the two layer types' KV cache in separate pools. The SWA pool can fill up and force evictions while the overall KV usage figure still looks healthy — and our dashboard wasn't tracking the SWA pool at all.
Inference engines are changing faster than ever, so keeping up with new parameters and new model implementations is now part of the monitoring job.
What's next
We also have more mechanisms for token accounting — but that's a post for another day.