DeepSeek Multi-Head Latent Attention: Calculating KV Cache Costs at 1M Tokens

When evaluating large language model (LLM) serving costs and architectural scalability, advertised context window lengths often mask massive underlying infrastructure demands. A context length specification is fundamentally a statement about positional encoding bounds. The KV cache size per token, however, determines whether serving that sequence length is economically and hardware-feasible.

While discussions around upcoming models like DeepSeek-V4 generate significant interest, analyzing established open architectural weights provides concrete insights into memory footprint dynamics. This article analyzes the public architectural specifications of DeepSeek-V3 and contrasts its Multi-Head Latent Attention (MLA) design with Grouped-Query Attention (GQA) implementations such as Llama 3.1 70B.

1. The Mathematics of Multi-Head Latent Attention (MLA)

Traditional Multi-Head Attention (MHA) caches full Key and Value tensors for every attention head at every layer. Grouped-Query Attention (GQA) reduces this overhead by sharing Key and Value heads across query head groups.

Multi-Head Latent Attention (MLA), introduced by DeepSeek, takes compression further by projecting Key and Value states into a low-rank latent vector prior to caching. Instead of caching high-dimensional key and value vectors per head, the model caches a single compressed latent representation.

MHA Layout (Per Token, Per Layer):
[ Key Head 1 | Key Head 2 | ... | Key Head H ] + [ Value Head 1 | Value Head 2 | ... | Value Head H ]

GQA Layout (Per Token, Per Layer):
[ Grouped Key 1 | Grouped Key G ] + [ Grouped Value 1 | Grouped Value G ]

MLA Layout (Per Token, Per Layer):
[ Compressed KV Latent Vector (kv_lora_rank) ] + [ Decoupled RoPE Key (qk_rope_head_dim) ]

In DeepSeek-V3's config.json, two key parameters dictate the KV cache footprint per layer:

• kv_lora_rank: 512 (the dimension of the compressed latent KV vector)
• qk_rope_head_dim: 64 (the decoupled Rotary Position Embedding key dimension)

Because rotary position embeddings are position-dependent and cannot be folded directly into the static low-rank compression matrix, the 64-dimensional RoPE key vector must be cached separately.

Calculating Per-Token Memory Footprint

To compute the memory required per token for DeepSeek-V3:

$\text\{Elements per Token per Layer\} = \text\{kv\_lora\_rank\} + \text\{qk\_rope\_head\_dim\} = 512 + 64 = 576$

With 61 hidden layers (num_hidden_layers = 61) and using standard 16-bit brain floating-point precision (bf16, 2 bytes per element):

$\text\{Bytes per Token per Layer\} = 576 \times 2 \text\{ bytes\} = 1,152 \text\{ bytes\}$

$\text\{Total Bytes per Token\} = 1,152 \text\{ bytes/layer\} \times 61 \text\{ layers\} = 70,272 \text\{ bytes\} \approx 70.27 \text\{ KB\}$

Scaling this to a 1,000,000 token context window for a single inference sequence:

$\text\{KV Cache Size\} = 1,000,000 \times 70,272 \text\{ bytes\} = 70,272,000,000 \text\{ bytes\} \approx 70.27 \text\{ GB\}$

At 1 million tokens, a single unquantized user session requires roughly 70 GB of VRAM solely for the KV cache, before factoring in model weights, activation memory, or batching.

2. Comparative Analysis: MLA vs. GQA vs. MHA

To evaluate the efficiency of DeepSeek's MLA, compare it against Llama 3.1 70B, which uses Grouped-Query Attention (GQA).

Llama 3.1 70B Configuration:

• num_hidden_layers: 80
• num_key_value_heads: 8
• head_dim: 128
• Precision: bf16 (2 bytes)

For Llama 3.1 70B, each layer caches both Key and Value vectors across all KV heads:

$\text\{Elements per Layer\} = 2 \times (\text\{num\_key\_value\_heads\} \times \text\{head\_dim\}) = 2 \times (8 \times 128) = 2,048$

$\text\{Bytes per Token\} = 2,048 \text\{ elements\} \times 80 \text\{ layers\} \times 2 \text\{ bytes\} = 327,680 \text\{ bytes\} \approx 327.68 \text\{ KB\}$

At 1,000,000 tokens:

$\text\{Llama 3.1 70B KV Cache\} = 1,000,000 \times 327,680 \text\{ bytes\} \approx 327.68 \text\{ GB\}$

DeepSeek-V3's MLA configuration reduces KV cache memory consumption by ~78.5% compared to GQA on Llama 3.1 70B, and over 96% compared to standard MHA.

For developers deploying production workloads, accessing high-throughput models via unified API platforms like n1n.ai simplifies managing these underlying infrastructure trade-offs.

3. Dynamic Calculation Implementation

The following Python script computes the exact KV cache memory requirements across context lengths and precision formats for arbitrary model configurations:

dataclass
class LLMConfig:
    name: str
    layers: int
    kv_lora_rank: int = 0
    qk_rope_head_dim: int = 0
    num_kv_heads: int = 0
    head_dim: int = 0
    is_mla: bool = False

def calculate_kv_cache_gb(config: LLMConfig, context_tokens: int, bytes_per_element: float = 2.0) -> float:
    if config.is_mla:
        elements_per_layer = config.kv_lora_rank + config.qk_rope_head_dim
    else:
        # Standard MHA / GQA: 2 * (num_kv_heads * head_dim) for Key + Value
        elements_per_layer = 2 * (config.num_kv_heads * config.head_dim)
    
    total_bytes = elements_per_layer * config.layers * bytes_per_element * context_tokens
    return total_bytes / (1024 ** 3)

# Configurations
deepseek_v3 = LLMConfig(name="DeepSeek-V3