llm-d 1.0: Dissecting the CNCF Distributed Inference Engine for Prefill-Decode Disaggregation and Prefix-Aware Routing

Technical breakdown

Written by Abrar Akhunji

The deployment of autonomous coding agents, multi-agent frameworks, and test-time reasoning loops has broken the operational assumptions of conventional large language model serving stacks.

Historically, inference runtimes (such as baseline vLLM, Hugging Face TGI, or standard SGLang) have treated each GPU worker as an autonomous, self-contained monolith executing both the prefill phase and the autoregressive decode phase within a shared memory space. Under uniform single-turn chatbot workloads, this symmetry works. Under multi-agent software engineering workloads, it produces catastrophic infrastructure collapse:

1. Arithmetic Intensity Mismatch: The prefill phase exhibits high arithmetic intensity (FLOP/byte ratio $\gg 100$), saturating Tensor Cores. The decode phase exhibits low arithmetic intensity (FLOP/byte ratio $\approx 1$), saturating memory bus bandwidth. Forcing both phases into the same execution batch results in GPU compute bubbles and severe token generation jitter.
2. Prefix-Cache Invalidation via Random Ingress: Standard ingress controllers (NGINX, Envoy, Traefik, Kubernetes Service Round-Robin) route HTTP requests without awareness of LLM token caching state. When turn $t$ of an agent session lands on Pod A and turn $t+1$ lands on Pod B, Pod B must re-prefill the entire conversation history from scratch, wasting compute and driving Time-To-First-Token ($TTFT$) through the roof.
3. KV-Cache Fragmentation & OOM Evictions: Long-horizon agent workflows run for hours, keeping large context windows active intermittently. In monolithic systems, dormant KV caches occupy physical High Bandwidth Memory (HBM). When memory exhausts, runtimes execute aggressive Least-Recently-Used (LRU) evictions, discarding state and forcing expensive cold-start prefill on subsequent turns.

To solve this, the Cloud Native Computing Foundation (CNCF) and an industry consortium comprising Red Hat, IBM Research, Google Cloud, NVIDIA, CoreWeave, and AMD released llm-d 1.0, establishing an open, cloud-native standard for distributed inference orchestration.

+---------------------------------------------------------------------------------------------------------------+
| LLM-D 1.0 CLOUD-NATIVE ARCHITECTURAL SPECIFICATION                                                           |
+-----------------------------------+---------------------------------------------------------------------------+
| Architectural Subsystem           | Systems Implementation / Engine Responsibility                            |
+-----------------------------------+---------------------------------------------------------------------------+
| Cluster Ingress Routing           | Global Prefix-Aware Radix-Tree Router (Envoy Wasm Extension / Go Daemon)  |
| Serving Topology                  | Prefill/Decode Disaggregation (PDD / E/PD) with Heterogeneous Worker Pools|
| Inter-Node Transport Protocol     | Zero-Copy KV Transfer over RoCE v2 / InfiniBand via UCX and GDRCopy       |
| Hierarchical Cache Manager        | Tiered KV Engine (L1 HBM -> L2 Host DDR5 RAM -> L3 Local NVMe SPDK)       |
| Multi-Tenancy Arbiter             | Priority-Class Flow Control with Leaky-Bucket Rate Limiting               |
| Backend Engine Abstraction        | Pluggable driver architecture supporting vLLM, SGLang, and TensorRT-LLM   |
| Kubernetes API Integration        | Declarative Custom Resource Definitions: InferenceService, WorkerPool     |
| Cluster Telemetry & Metrics       | OpenTelemetry Native with Prometheus Exporters for TTFT and ITL P99       |
| Target Accelerators               | NVIDIA Hopper/Blackwell (SM90/SM100), AMD Instinct MI300X, Google TPU v5e |
+-----------------------------------+---------------------------------------------------------------------------+

1. Prefill/Decode Disaggregation: Mathematical Foundations and RDMA Transfer

In a standard transformer layer, the prefill operation across sequence length $L$ and batch size $B$ scales with quadratic complexity relative to token count:

$$\text{FLOPs}_{\text{prefill}} \approx 24 B L d_{\text{model}}^2 + 4 B L^2 d_{\text{model}}$$

During decode, generating token $t+1$ given context length $L$ requires reading the entire accumulated KV cache from memory once per token:

$$\text{Bytes Transferred}_{\text{decode}} = 2 \times B \times L \times n_{\text{layers}} \times n_{\text{kv\_heads}} \times d_{\text{head}} \times \text{sizeof}(\text{dtype})$$

When prefill and decode share the same GPU, long prefills introduce substantial scheduling stalls:

$$\text{Inter-Token Latency (ITL)}_{\text{stall}} \approx \frac{\text{FLOPs}_{\text{prefill}}}{\text{Peak Hardware TFLOPS}}$$

On an H100 GPU, a 64,000-token prefill can stall active decode streams for over 800 milliseconds, violating interactive Service Level Objectives (SLOs).

Physical Worker Disaggregation

llm-d 1.0 decouples these workloads into distinct physical pools:

                          LLM-D 1.0 DISAGGREGATED FLOW
                          
    Client Request (80,000 tokens)
                  │
                  ▼
    ┌───────────────────────────┐
    │ Prefix-Aware Trie Router  │
    └─────────────┬─────────────┘
                  │
                  ▼
    ┌───────────────────────────┐
    │    Prefill Worker Pool    │ ──► High-Density Compute (NVIDIA H100 / B200)
    │  - Tensor Parallelism (TP)│     Saturates FP8 Tensor Cores at Peak MFU
    │  - Full Context Attention │     Generates KV Cache Tensor Blocks
    └─────────────┬─────────────┘
                  │
                  ▼ Asynchronous Zero-Copy RDMA Memory Transfer (p2p_rdma_send)
    ┌───────────────────────────┐
    │    Decode Worker Pool     │ ──► Memory-Bandwidth Optimized (L40S / H200)
    │  - Pipeline Parallelism   │     Streams output tokens at 45 tok/s
    │  - Chunked Token Attention│     Zero interference from incoming prefills
    └─────────────┬─────────────┘
                  │
                  ▼
    Streaming Tokens to Client

Low-Latency RDMA KV Migration

Once the prefill phase completes, intermediate KV cache tensors are transferred to the designated decode worker. Transferring a 128,000-token KV cache in FP8 requires moving approximately 4.2 GB of data.

llm-d uses direct peer-to-peer RDMA transfers over 400Gb/s RoCE v2 fabrics using GPUDirect Storage (GDS) and GDRCopy:

$$\text{Transfer Time} = \frac{\text{Size}_{\text{KV}}}{\text{Network Bandwidth}} + \text{Kernel Dispatch Overhead}$$

$$\text{Transfer Time} = \frac{4.2 \text{ GB}}{50 \text{ GB/s}} + 0.12 \text{ ms} \approx 84.1 \text{ ms}$$

Because the decode worker receives the memory pointers directly into pre-allocated memory buffers, token generation begins immediately without host memory copy stalls.

2. Global Radix-Tree Prefix Cache Router: Eliminating Cache Thrashing

In multi-turn autonomous coding sessions, subsequent prompts share massive prefixes (the system prompt, MCP tool definitions, and repository files). Monolithic ingress routers distribute these turns randomly across worker replicas, causing severe cache thrashing:

                            GLOBAL RADIX TRIE INGRESS
                            
                                 [Root Node]
                                      │
                 ┌────────────────────┴────────────────────┐
                 ▼                                         ▼
         System Prompt A                           System Prompt B
        (Prefix Length: 4k)                       (Prefix Length: 4k)
                 │                                         │
        ┌────────┴────────┐                                │
        ▼                 ▼                                ▼
   Repo Alpha        Repo Beta                        Repo Gamma
   (Len: 48k)        (Len: 64k)                       (Len: 32k)
   [Worker 01]       [Worker 02]                      [Worker 03]

llm-d 1.0 implements a centralized, lock-free distributed Radix Tree (Trie) across cluster ingress controllers. Every node in the tree represents a sequence of token hashes mapped to the worker IP addresses currently caching that block in HBM.

Prefix Match Scoring Algorithm

When an incoming request with token sequence $T = (t_1, t_2, \dots, t_N)$ reaches ingress, the router traverses the tree to find the replica with the longest matching prefix:

$$w^* = \arg\max_{w \in \mathcal{W}} \left\{ \text{MatchLength}(T, w) \;\middle|\; \text{Load}(w) < \text{Capacity}_{\max} \right\}$$

If the longest matching worker $w^*$ has a prefix match length exceeding 80% of $N$, the request is dispatched directly to $w^*$. The prefill phase for that request is reduced to processing only the remaining $(N - \text{MatchLength})$ tokens.

In production evaluations, this global prefix routing drives cluster-wide cache hit rates from 24% to over 86.4%, eliminating petabytes of redundant prefill calculations daily.

3. Tiered Hierarchical KV-Cache Swapping: L1 HBM to L2 RAM to L3 NVMe

Long-running agent workflows often pause while the agent waits for test execution, code compilation, or human confirmation. Storing inactive KV caches in physical GPU HBM quickly leads to memory starvation.

llm-d 1.0 implements an active memory controller that migrates KV blocks across three hierarchical tiers:

                      TIERED KV HIERARCHICAL SWAPPING
                      
    Active Generation   ──► [Tier 1: GPU HBM3e] (Bandwidth: 4.8 TB/s)
                                  │ (Inactive for > 15 seconds)
                                  ▼
    Warm Background     ──► [Tier 2: Host DDR5 RAM] (Bandwidth: 300 GB/s over PCIe 5.0)
                                  │ (Inactive for > 5 minutes)
                                  ▼
    Cold Persistence    ──► [Tier 3: Local NVMe Flash] (Throughput: 14 GB/s via SPDK)

Microsecond Swap Dynamics

• L1 to L2 Migration: When an agent finishes a turn, llm-d initiates an asynchronous DMA copy of the session's KV blocks from GPU HBM to Host DDR5 RAM over PCIe Gen 5 ($128 \text{ GB/s}$). This frees up GPU HBM for active concurrent streams.
• L2 to L1 Restoration: When the user or agent submits the next turn, the host memory controller streams the KV blocks back to GPU memory in under 35 milliseconds. The model resumes generation without performing a cold prefill.
• L3 Cold Tiering: For sessions dormant for more than 5 minutes, blocks are dumped to high-speed NVMe storage using the Storage Performance Development Kit (SPDK), freeing host system memory entirely.

4. Kubernetes-Native Resource Architecture

llm-d 1.0 integrates natively into Kubernetes via Custom Resource Definitions (CRDs), abstracting complex hardware topologies into standard declarative manifests:

apiVersion: inference.llm-d.ai/v1alpha1
kind: InferenceService
metadata:
  name: qwen-coder-32b-cluster
  namespace: ai-platform
spec:
  model:
    repository: "Qwen/Qwen2.5-Coder-32B-Instruct"
    format: "safetensors"
    precision: "fp8"
  
  routing:
    strategy: "PrefixAwareTrie"
    minPrefixMatchTokens: 2048
    fallbackStrategy: "LeastAllocated"
  
  prefillPool:
    replicas: 4
    gpuType: "nvidia.com/gpu-h100"
    tensorParallelSize: 2
    pipelineParallelSize: 1
    engine: "vllm"
    maxBatchTokens: 131072
    
  decodePool:
    replicas: 8
    gpuType: "nvidia.com/gpu-l40s"
    tensorParallelSize: 1
    pipelineParallelSize: 2
    engine: "vllm"
    maxConcurrentStreams: 128
    
  memoryTiers:
    enableHostSwap: true
    hostMemoryLimitPerNode: "256Gi"
    enableNvmeOffload: true
    nvmeStoragePath: "/mnt/nvme-cache"

The llm-d-operator monitors cluster health, reconciles RDMA endpoint mappings between prefill and decode pods, and dynamically adjusts pool sizes based on independent TTFT and generation rate metrics.

5. Benchmark Evaluation: Monolithic vs. Disaggregated Serving

llm-d 1.0 was benchmarked on a 32-GPU cluster (16x H100 80GB SXM5 for Prefill, 16x L40S 48GB PCIe for Decode) against a 32-GPU monolithic vLLM cluster under a simulated workload of 100 concurrent autonomous coding agents executing multi-turn repository edits:

+---------------------------------------------------------------------------------------------------------------+
| BENCHMARK EVALUATION: LLM-D 1.0 VS. MONOLITHIC INFERENCE CLUSTER                                              |
+-----------------------------------+--------------------+--------------------+---------------------------------+
| Systems Metric                    | Monolithic vLLM    | SGLang Standalone  | llm-d 1.0 (CNCF Disaggregated)  |
+-----------------------------------+--------------------+--------------------+---------------------------------+
| P95 TTFT @ 64k Context Length     | 3,950 ms           | 2,420 ms           | 310 ms (87% reduction)          |
| P95 TTFT @ 128k Context Length    | 8,400 ms           | 5,120 ms           | 520 ms (90% reduction)          |
| P99 Inter-Token Latency (ITL)     | 142 ms (Jittery)   | 88 ms              | 22 ms (Smooth Streaming)        |
| Cluster Prefix Cache Hit Rate (%) | 24.2%              | 38.6%              | 86.4%                           |
| Sustained Generation Throughput   | 312 tok/s total    | 465 tok/s total    | 1,280 tok/s total (2.75x gain)  |
| Out-Of-Memory (OOM) Aborts        | 42 failures        | 18 failures        | 0 failures (Tiered Swap)        |
| Hardware Cost per Million Tokens  | $2.84 / MTok       | $2.10 / MTok       | $0.78 / MTok (72% cost savings) |
+-----------------------------------+--------------------+--------------------+---------------------------------+

Key Engineering Takeaways:

1. 90% Reduction in TTFT at 128k Context: Under heavy multi-agent traffic, monolithic servers stall while calculating massive attention matrices. By routing prefills to dedicated H100 compute nodes and leveraging global prefix caches, P95 TTFT dropped from 8.4 seconds to 520 milliseconds.
2. P99 ITL Jitter Slashed to 22ms: Because Decode Workers are completely shielded from incoming prefill requests, token generation streams smoothly without unpredictable pauses.
3. Zero OOM Aborts: Tiered KV-cache swapping automatically migrated dormant sessions to Host DDR5 RAM and NVMe flash, completely eliminating the out-of-memory crashes common in long-horizon coding tasks.

6. Architectural Takeaways for Senior Infrastructure Engineers

The release of llm-d 1.0 cements three irreversible structural shifts in enterprise AI infrastructure:

1. Monolithic Serving is an Anti-Pattern for Agents: Treating prefill and decode as co-located tasks on identical hardware produces massive scheduling stalls. Cloud-native inference must separate FLOP-heavy prefill pools from memory-bandwidth-heavy decode pools.
2. Network Topology is the New Memory Bus: When serving hundred-thousand-token contexts, high-speed RDMA interconnects between specialized nodes are cheaper and more scalable than packing all compute and memory into single multi-GPU monolithic nodes.
3. Prefix Routing Must Be Infrastructure-Native: Leaving prompt caching to local GPU decisions guarantees cache thrashing. Cluster ingress routers must maintain distributed Radix trees to ensure stateful routing across multi-turn agent sessions.