Part 2 of 3 – giving each phase its own hardware, and turning repeated context into a memory lookup.
This is the second part of the three-part series How To Solve “Broken” LLM Inference In Agentic Coding. Part 1 established two things: prefill is compute-bound while decode is memory-bandwidth-bound, and a KV-aware router can stop a Kubernetes Service from causing avoidable cache misses. But routing alone leaves the deeper problem untouched: Both phases still compete for the same GPU.
In this part, we will dive deeper into separating prefill and decode into dedicated GPU pools, and explore how this separation leads to a more usable architecture.
Both phases compete for the same GPU
For long-context coding requests – a repo-scale input of 32K–128K tokens producing a long, multi-thousand-token completion – co-locating prefill and decode on the same GPU and batch means the two phases get in each other’s way by design:
- A long prefill for one request blocks decode steps for others, causing ITL/TPOT jitter, and TTFT grows as a result.
- A shared tensor-parallel configuration and batch can’t be tuned per phase – compute sits idle during decode, and bandwidth sits idle during prefill.
PD Disaggregation (xPyD) splits the two onto separate GPU pools entirely. Dedicated, compute-optimized, high-TP prefill workers deliver fast prefill and low TTFT. Dedicated, memory-bandwidth-optimized, high-batch decode workers deliver smooth and stable ITL/TPOT. Because the pools are separate, they scale independently – you can pick the right number of GPUs for each phase, instead of one setup serving both.
Running both phases together creates a built-in bottleneck. Splitting them lets you tune each stage on its own, which lowers latency and uses the GPUs better.
Moving KV between the pools
Splitting the phases raises an obvious question: how does the KV cache computed by a prefill worker reach the decode worker that needs it? That’s handled by NIXL, and the transport path depends on topology.
Within one node, over NVLink or NVLink Switch, KV transfer goes directly GPU→GPU at roughly 900 GB/s, bypassing the CPU entirely. Across nodes it moves over UCX on InfiniBand or RoCE via RDMA at roughly 50–100 GB/s, with metadata synced through etcd or a discovery service.
A topology-aware routing layer decides, for every request, which decode worker a given prefill worker should hand off to. It applies three checks in order:
- Compatibility – same model, same tensor-parallel size, same KV layout, same block size and dtype. Only compatible decode workers are candidates.
- Topology – prefer a decode worker in the same node or NVLink domain, since that path is a fast, cheap VRAM→VRAM transfer. A different node means falling back to RDMA over InfiniBand, slower but still viable.
- Load – among the remaining candidates, pick whichever has the fewest active sequences or the most free KV blocks.
LMCache: stop rebuilding what never changed
Splitting the pools solves the hardware competition problem. It does not stop the wasted recompute – and for agentic coding, that’s the change that pays off the most.
Look at what a coding agent actually sends on turn 5. The system prompt is the same as turn 4. The tool schemas are the same. The repository files it pulled in are the same. Only the new question at the end is different – often around 10% of the prompt.
A standard setup does not know that. It takes the whole prompt and runs it through the GPU from scratch, producing a KV cache that is identical to the one it produced last turn and then threw away.
LMCache changes this in two steps. First it hashes the prompt in blocks, so it can tell which blocks it has seen before. Second, instead of keeping those blocks only in VRAM – where they get pushed out within a few turns – it keeps them in host RAM or on NVMe, which is far larger. When the same blocks come back, they are read from memory rather than computed again.
The result is that only the new tokens reach the GPU. In production this removes 50–75% of prefill work, and the effect compounds:
- Faster first token – skipping compute for the repeated part is what brings TTFT down.
- Less GPU work per request – the same card serves many more requests per second.
- A better fit for split pools – once the repeated part is free, the prefill pool is limited by memory speed rather than compute, so it handles far more users without adding GPUs.
That last point is the one that matters. Offloading doesn’t only make single requests faster – it changes what limits the prefill pool. Without it, splitting prefill and decode just moves the queue somewhere else.
The full request path
Putting it together: a request arrives at the frontend; the KV-aware router selects a prefill worker based on topology and cache state; the prefill worker looks up the prefix, loads any cached KV from the LMCache offload tier, and computes KV only for genuinely new tokens; the newly computed KV is stored back to CPU RAM offload; then a decode router selects a decode worker and triggers the NIXL VRAM→VRAM transfer before token generation and streaming begin.
What’s still missing
The architecture is now usable, but it isn’t yet operable. Prefill and decode pools need to scale independently without crowding out other workloads or being split up while running, and a newly spawned pod needs to start serving fast enough for autoscaling to be worth anything at all.
For that to happen, Part 3 will cover Grove and KAI Scheduler, KEDA-driven autoscaling, GPU-to-GPU weight streaming for fast cold starts – and the benchmark numbers showing what the whole stack is actually working well or not.
—————-
Author:
Duong Nguyen Thai is an AI Platform Engineer at FPT AI Factory, where he focuses on building and operating large-scale AI inference platforms. He is also a CNCF Kubestronaut, having earned the full set of Kubernetes certifications, including CKA, CKAD, KCNA, KCSA, and CKS. His work centers on scalable AI infrastructure, Kubernetes, and production-grade systems for serving AI models.
