← All papers

Mixtral of Experts

Jiang, Sablayrolles, Roux et al. (Mistral AI) · 2024 · arXiv 2024

ArchitectureRead on arXiv

Mistral AI's Mixtral 8x7B, a sparse Mixture-of-Experts model that matches or outperforms Llama 2 70B and GPT-3.5 while using only 13B active parameters per token, demonstrating that sparse MoE is a practical path to high quality at lower compute.

Key Idea

Mixtral 8x7B is a sparse Mixture-of-Experts (SMoE) decoder-only model. It has 47B total parameters but uses only ~13B active parameters per token, because each token is routed to only 2 of 8 experts in every MoE layer. This decouples model capacity from per-token compute cost.

Architecture

  • Based on the Mistral 7B architecture (sliding window attention, RoPE, GQA, SwiGLU), but with the feed-forward block in each layer replaced by an MoE block
  • 8 expert FFN blocks per MoE layer: each ~7B-equivalent
  • Top-2 routing: A lightweight gating network (linear layer + softmax) scores all 8 experts for each token, and the token's representation is computed as a weighted sum of the top 2 experts' outputs
  • Total params ≈ 47B (not 56B, because attention and embedding layers are shared across experts), active params per token ≈ 13B
  • 32K context length: with sliding window attention for efficiency on long sequences

Top-2 Routing in Detail

  • For each token at each MoE layer: gate_logits = x · W_gate (shape: [tokens, 8])
  • Pick the top-2 experts by gate score and renormalize their weights with softmax
  • Compute output: y = w_1 · Expert_{i_1}(x) + w_2 · Expert_{i_2}(x)
  • Routing decisions are made independently per layer per token, different tokens take different paths through the network

Performance

  • Outperforms Llama 2 70B on most benchmarks (MMLU, GSM8K, HumanEval, MT-Bench), despite having far fewer active parameters
  • Matches or beats GPT-3.5 on standard benchmarks
  • ~5× faster inference than a dense 47B model at the same total parameter count, because compute scales with active (not total) parameters
  • Mixtral 8x7B Instruct (fine-tuned with SFT + DPO) reached MT-Bench scores competitive with GPT-3.5

Expert Specialization

  • The paper analyzes routing patterns and finds no obvious topical specialization (experts aren't "the math expert" or "the code expert")
  • Instead, experts appear to specialize on syntactic and positional patterns, routing correlates with token identity and sequence position rather than high-level domain
  • This is a notable empirical finding that pushed back on intuitive assumptions about MoE behavior

Why It Matters

  • First widely-adopted open-source MoE at frontier quality, opened the playbook for everyone else (DeepSeek-MoE, Qwen MoE, Grok)
  • Demonstrates the compute-vs-parameters decoupling that makes MoE attractive: more knowledge in the weights without proportionally more compute per token
  • Released under Apache 2.0 with full weights and inference code, drove adoption in production serving stacks (vLLM, TGI added MoE support)

Key Takeaways for Interviews

  • Mixtral = 8 experts × 7B FFN with top-2 routing → 47B total, 13B active per token
  • Active parameters drive compute and latency; total parameters drive memory and capacity
  • MoE trades memory (must keep all experts in VRAM) for FLOPs (only run 2 of 8), relevant for serving cost models
  • Routing is per-token, per-layer, learned jointly with the model; load balancing losses (auxiliary loss) prevent expert collapse
  • Empirically, experts specialize on syntax/position, not topics, counterintuitive but consistent finding
  • MoE is harder to serve than dense models: expert load imbalance, all-to-all communication in distributed setups, memory pressure