Push-Based KV Transfer for Disaggregated Inference in vLLM

As modern Large Language Models like DeepSeek-V3 and Claude 3.5 Sonnet scale to hundreds of billions of parameters, deploying them efficiently requires architectural innovations beyond simple cluster scaling. Prefill-decode disaggregation has emerged as a cornerstone technique, separating compute-heavy prompt processing (prefill) from memory-bandwidth-bound token generation (decode). However, this architectural split introduces a critical bottleneck: transferring the Key-Value (KV) cache between prefill and decode engine instances. In this technical review, we examine the newly introduced push-based KV transfer mechanism using the NIXL connector in vLLM, analyzing how it eliminates coordination overhead to dramatically lower Time to First Token (TTFT) and improve Time Per Output Token (TPOT).

The Bottleneck of Disaggregated LLM Inference

In a standard monolithic LLM serving setup, a single GPU instance handles both the prefill phase (processing the input prompt in parallel) and the decode phase (generating response tokens sequentially). These two phases have radically different hardware utilization profiles. Prefill is compute-intensive, requiring massive parallel matrix multiplications, whereas decode is memory-bandwidth-bound, limited by how fast weights and KV cache can be loaded from high-bandwidth memory into processor cores.

By disaggregating prefill and decode into separate instances, infrastructure engineers can optimize hardware allocations specifically for each workload. For instance, prefill nodes can utilize high-throughput setups, while decode nodes focus on high memory bandwidth. Yet, moving the generated KV cache from the prefill engine to the decode engine over the network creates significant latency. Traditional pull-based architectures require the decode instance to request and wait for the cache, placing data transfer directly on the critical execution path and stalling inference pipelines.

Enter the Push-Based NIXL Connector

To solve this coordination overhead, vLLM integrates a push-based KV transfer mechanism powered by the NIXL connector. Instead of forcing the decode engine to poll for data, the prefill engine actively pushes the KV cache to the target decode instance as soon as computation completes.

This design shift decouples network transfer coordination from the primary inference loop. By leveraging high-performance networking fabrics such as AWS Elastic Fabric Adapter (EFA) on Amazon EC2 P5en instances, the push-based NIXL connector bypasses traditional kernel network stacks, utilizing remote direct memory access (RDMA) semantics to place tensor data directly into the target GPU's memory buffer.

Architectural Comparison of KV Transfer Mechanisms

Benchmarking Performance on AWS EC2 P5en

When deployed on Amazon EC2 P5en instances equipped with NVIDIA H100 GPUs and AWS EFA, the push-based NIXL architecture yields substantial performance gains. Benchmarks comparing legacy pull-based setups against the new push-based vLLM integration demonstrate profound improvements across key metrics.

1. Time to First Token (TTFT): Reduced by up to 3x under heavy concurrent request loads. Because the KV cache arrives at the decode node preemptively, the decode engine can start generating tokens almost instantaneously.
2. Time Per Output Token (TPOT): Improved by up to 30%. Eliminating transfer stalls ensures steady, predictable token generation rates, vital for real-time AI agents and interactive chat applications.

For developers orchestrating complex workflows using frameworks like LangChain or building custom multi-agent systems via the n1n.ai API ecosystem, these latency reductions translate directly into smoother user experiences and lower infrastructure costs.

Implementing Push-Based KV Transfer in vLLM

Configuring vLLM to utilize disaggregated inference with the push-based NIXL connector requires specific initialization parameters across your prefill and decode engine instances. Below is a practical configuration template for deploying this setup in a distributed Python environment.

from vllm import AsyncEngineArgs, AsyncLLMEngine

# Configuration for the Prefill Engine Instance
prefill_engine_args = AsyncEngineArgs(
    model="deepseek-ai/DeepSeek-V3