Transferring KV Cache Between LLMs Without Re-Prefill for 25x Faster Inference

In the rapidly evolving landscape of Large Language Model (LLM) deployment, efficiency is the ultimate currency. As developers move away from monolithic single-model architectures toward complex multi-model pipelines, a new bottleneck has emerged: the cost of switching models mid-conversation. Whether you are implementing model cascading, dynamic routing, or escalating complex queries from a small model to a large one, the overhead of re-prefilling the context is a significant drag on performance.

Platforms like n1n.ai provide access to a wide array of models, enabling these advanced strategies. However, the technical challenge of reusing the Key-Value (KV) cache between different models has remained largely unsolved—until now. A recent breakthrough by Heo et al. (2026) introduces a method to transfer KV caches across models in the same family using a closed-form linear mapper, achieving speedups of 2.7x to 25x compared to re-prefilling from scratch.

The Hidden Cost of Model Handoffs

Modern LLM deployments are no longer static. Three primary patterns dominate high-performance production environments:

1. Model Cascading: Routing simple queries to smaller, cheaper models (like Qwen-7B) and escalating difficult ones to larger models (like Qwen-72B).
2. Mid-conversation Switching: Starting a dialogue with a fast model for low-latency responses and switching to a stronger model as the context window grows or complexity increases.
3. Dynamic Routing: Using a classifier to select the best model for a specific request on the fly.

In every one of these scenarios, the receiving model currently has to re-run the entire prefill process. Prefill involves processing every input token through every layer to generate the KV cache. For Retrieval-Augmented Generation (RAG) pipelines or long-context applications with thousands of tokens, this process can take hundreds of milliseconds on high-end GPUs like the H100. This latency is repeated every single time a handoff occurs, negating many of the benefits of using smaller models in the first place.

Why KV Cache Reuse Was Considered Impossible

The reason we haven't been able to simply "copy" the KV cache from one model to another is rooted in architectural variance. Even within the same model family, a 14B model and a 32B model differ in:

• Layer Counts: More parameters usually mean more transformer blocks.
• Hidden Dimensions: The width of the vector space changes.
• Attention Heads: The number of heads and the dimensionality per head often vary.

Because the internal representations are different, the KV cache generated by a source model is essentially "gibberish" to a target model. However, the researchers discovered a surprising property: KV caches within the same model family exhibit a strong linear structure. In a Qwen3 14B to 32B transfer, a single source layer can explain 56% of the variance in the target model's keys. By combining multiple source layers, this predictability jumps to 79% for keys and 65% for values.

The Solution: A Closed-Form Linear Mapper

The proposed method uses ridge regression to map the source KV cache into the target model's space. This is a three-step process designed for maximum speed and minimum accuracy loss.

Step 1: Layer Selection

Not all source layers are equally useful for predicting a specific target layer. The researchers use R² (coefficient of determination) to identify the top-k most predictive source layers for each target layer. This ensures the mapper focuses on the most relevant information.

Step 2: RoPE Stripping

Modern LLMs use Rotary Positional Embeddings (RoPE) to inject positional information into keys. This makes the keys position-dependent. To create a universal mapper, the researchers strip the RoPE encoding before mapping and re-apply it afterward. This makes the linear mapper position-free and reusable across any context length.

Step 3: Ridge Fit

The mapper is trained using a simple ridge regression on a small calibration set (e.g., 500 sequences from FineWeb-Edu). The formula is a standard closed-form solution:

$W^* = (X^\top X + \lambda I)^\{-1\} X^\top Y$

Where $X$ represents the source KV cache and $Y$ is the target. This one-time offline calculation produces a weight matrix that can be used instantly during inference.

Implementation Guide

Below is a simplified Python implementation using PyTorch to demonstrate the core logic of the transfer process.

import torch
from typing import List

def rotate_half(x: torch.Tensor) -> torch.Tensor:
    x1 = x[..., : x.shape[-1] // 2]
    x2 = x[..., x.shape[-1] // 2 :]
    return torch.cat([-x2, x1], dim=-1)

def strip_rope(keys: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor) -> torch.Tensor:
    """Remove RoPE encoding to get position-free keys."""
    # Logic: keys = keys_orig * cos + rotate_half(keys_orig) * sin
    # We solve for keys_orig using the inverse rotation
    return keys * cos - rotate_half(keys) * sin

def fit_ridge_mapper(
    source_kvs: List[torch.Tensor],  # List of (N, D) tensors from top-k source layers
    target_kv: torch.Tensor,          # (N, D) tensor from the target layer
    lambda_reg: float = 1e-4
) -> torch.Tensor:
    """Fit closed-form ridge regression mapper."""
    X = torch.cat(source_kvs, dim=-1)       # Shape: (N, k*D)
    XtX = X.T @ X
    XtY = X.T @ target_kv
    reg = lambda_reg * torch.eye(X.shape[1], device=X.device, dtype=X.dtype)
    W = torch.linalg.solve(XtX + reg, XtY)  # Resulting mapper: (k*D, D)
    return W

def transfer_kv(
    source_kvs: List[torch.Tensor],
    W: torch.Tensor,
    cos: torch.Tensor = None,
    sin: torch.Tensor = None,
    is_key: bool = True
) -> torch.Tensor:
    """Transfer KV cache from source to target model space."""
    if is_key and cos is not None:
        # Keys need positional stripping
        kvs = [strip_rope(kv, cos, sin) for kv in source_kvs]
    else:
        kvs = source_kvs

    # Project to target space
    predicted_kv = torch.cat(kvs, dim=-1) @ W
    return predicted_kv

Benchmarking the Results

The performance gains are most visible in the Qwen family, which is widely available through n1n.ai. The following table summarizes the findings across different model pairs:

The 25x speedup in the Qwen3 14B to 32B case is transformative. It means that the latency penalty for upgrading a query to a larger model becomes almost negligible, enabling much more fluid model-switching logic in production.

Pro Tips for Production Deployment

1. Integration with vLLM: If you are using vLLM or other PagedAttention-based systems, this mapper can be integrated as a pre-processing step. Before the target model starts its "prefill," you inject the transferred KV cache into the paged memory pool. The engine then skips the prefill for the transferred tokens.
2. Addressing Value Handoffs: The research shows that Values are harder to transfer than Keys. If you notice a drop in accuracy, consider using a small non-linear MLP instead of a linear mapper for the Value cache. While this adds a small amount of overhead, it can recover up to 37 percentage points in tasks like HellaSwag.
3. Cross-Family Limitations: Currently, this method requires the source and target models to share the same KV head count and per-head dimension. You cannot transfer from a Llama model to a Qwen model using this specific linear mapping. For diverse model needs, using an aggregator like n1n.ai remains the best way to manage multiple disparate APIs.

Conclusion

The ability to transfer KV caches without re-prefilling marks a significant step toward truly elastic LLM infrastructure. By reducing the cost of model handoffs, developers can build more responsive, cost-effective, and intelligent applications. As LLM serving becomes more complex, tools that bridge the gap between models will be essential.

Get a free API key at n1n.ai