Why Agentic Coding Breaks a Naive LLM Deployment?

Part 1 of 3 — the tokenomics problem, and routing requests by cache state instead of load.

Agentic coding is not chat. A chatbot session might drift from 2K to 4K tokens; a coding agent re-sends nearly everything it has already seen, every single turn. Five turns on a large codebase can grow from 64K to 480K tokens of resent context — roughly 1.25M tokens processed in total. That non-linear blow-up is why a naive vllm serve deployment collapses: a 500K-token repository context pushes time-to-first-token past 30 seconds and concurrency under 10 users.

This three-part series is a blueprint for doing that on Kubernetes. Part 1 covers the fundamentals and the routing layer; Part 2, PD disaggregation and cache offloading; Part 3, scheduling, autoscaling, and the benchmark numbers.

Prefill and decode are opposite workloads

Every LLM inference request runs through two phases with fundamentally different resource profiles, and understanding this split is the key to everything that follows.

Prefill is compute-bound. The full prompt is processed in parallel in a single forward pass — exactly the kind of workload that maximizes GPU FLOPS utilization.

Decode is memory-bandwidth-bound. Tokens are generated one at a time, and each step re-reads the entire KV cache from HBM. There’s very little arithmetic per byte moved, so the GPU spends its time waiting on memory bandwidth rather than computing.

These two phases don’t just have different needs — they actively compete for the same resource pool when co-located. On a single instance, a long prefill blocks decode steps for every other in-flight request, showing up to users as inter-token latency (ITL) jitter.

A stock Kubernetes Service is cache-blind

Layered on top of that is a routing problem. Round-robin or least-connections load balancing has no concept of which pod already holds a request’s KV cache in memory, so it routes requests to the wrong pod and manufactures cache misses that didn’t need to happen.

For chat traffic that’s a minor inefficiency. For agentic coding — where most of every prompt is context the system has already seen — it’s the difference between a cache hit and recomputing hundreds of thousands of tokens.

Routing by cache state and load together

The fix is a router that scores each candidate worker on two signals at once, then dispatches to whichever comes out cheapest.

Signal 1 — KV overlap. A KV indexer tracks tiered prefix hits by hashing incoming request tokens into block hashes, estimating how much of the prompt is already cached on a given worker. More overlap means less prefill work, which directly lowers TTFT.

Signal 2 — live load. A slot tracker watches each worker’s active prefill and decode sequences, estimating current load so the router doesn’t pile more requests onto an already-saturated worker.

These combine into a routing cost function:

cost = overlap_score_weight × prefill_blocks + decode_blocks

Source: https://docs.nvidia.com/dynamo/dev/knowledge-base/modular-components/router/routing-concepts

prefill_blocks is the number of tokens still needing prefill divided by block size — it shrinks as more of the prefix is already cached on that worker. decode_blocks is estimated from input tokens and the worker’s active sequence count, updated as requests complete. With the weight set to 1, the router balances cache-hit priority evenly against load balancing.

Consider four candidate workers with cache overlaps of 8/10, 6/10, 2/10 and 0/10, carrying active decode loads of 12, 4, 3 and 1 sequences:

W1 has the best cache hit rate but is heavily loaded. W2 wins because it balances a decent cache hit against low current load exactly the trade-off the cost function is designed to make. In practice this is a single flag, –router-mode kv, with the engine emitting KV cache events on block create and remove so the router’s state stays current.

What routing alone doesn’t fix?

KV-aware routing removes the manufactured cache misses, but it can’t touch the underlying contention: prefill and decode still share the same GPUs and the same batch. A long repo-scale prefill will still stall decode for everyone else on that worker.

In Part 2, we’ll explore how prefill-decode disaggregation can eliminate this contention by separating the two workloads across dedicated GPU pools.

—————-

Author:

Duong Nguyen Thai is an AI Platform Engineer at FPT AI Factory, where he focuses on building and operating large-scale AI inference platforms. He is also a CNCF Kubestronaut, having earned the full set of Kubernetes certifications, including CKA, CKAD, KCNA, KCSA, and CKS. His work centers on scalable AI infrastructure, Kubernetes, and production-grade systems for serving AI models.