← All papers

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

DeepSeek-AI (Liu, Feng, Wang et al.) · 2024 · arXiv 2024

EfficiencyRead on arXiv

Introduces Multi-head Latent Attention (MLA), which compresses the KV cache by 93.3% through low-rank key-value projection into a shared latent vector, achieving 5.76× faster generation throughput while maintaining model quality.

Key Idea

Standard Multi-Head Attention (MHA) stores separate key and value vectors for every head at every position, creating massive KV caches that bottleneck inference. Grouped Query Attention (GQA) reduces this by sharing KV heads across query heads, but still stores full-dimensional KV vectors. Multi-head Latent Attention (MLA) goes further: compress all key-value information into a single low-rank latent vector per position, then reconstruct head-specific keys and values on the fly during attention.

How It Works

  • Low-rank KV compression: Instead of caching separate K and V matrices, MLA projects them into a shared compressed latent vector c_t of much smaller dimension
  • On-the-fly reconstruction: During attention computation, K and V are reconstructed from c_t via up-projection matrices, different for each head
  • Decoupled Rotary Position Embedding (RoPE): Since RoPE is incompatible with low-rank KV compression (it breaks the factorization), MLA uses a small additional set of decoupled key vectors that carry positional information separately
  • KV cache = just c_t: Only the compressed latent vector (plus the small RoPE keys) needs to be cached per token, not full K and V matrices

Architecture Context

MLA is one half of DeepSeek-V2's innovations (the other being DeepSeekMoE for sparse computation). Together they enable a 236B-parameter model with only 21B activated parameters per token.

Why It Matters

  • 93.3% KV cache reduction compared to standard MHA, dramatically reduces memory requirements during inference
  • 5.76× generation throughput improvement over the prior DeepSeek 67B model
  • Outperforms GQA and MQA in quality while being more memory-efficient, MLA achieves better perplexity than GQA at equivalent cache sizes
  • Became the attention mechanism for DeepSeek-V3 and influenced subsequent efficient attention research
  • Demonstrates that KV cache compression can be done through learned low-rank projection rather than just head-sharing heuristics

Key Takeaways for Interviews

  • MLA = compress KV cache into a low-rank latent vector, reconstruct K/V per head on the fly
  • Compare with GQA (shares KV heads) vs MLA (compresses all KV into a latent), MLA is more aggressive and more effective
  • The RoPE decoupling trick is essential: positional embeddings break low-rank factorization, so they need separate handling
  • Cache size per token: MHA caches 2×n_heads×d_head, GQA caches 2×n_groups×d_head, MLA caches d_latent (+ small RoPE component)
  • In system design: MLA enables serving much larger batch sizes under the same GPU memory budget, directly improving throughput and cost