← All papers

The Llama 3 Herd of Models

Grattafiori, Dubey, Jauhri et al. (Meta) · 2024 · arXiv 2024

ArchitectureRead on arXiv

Meta's Llama 3 technical report describes a herd of dense decoder-only models (8B, 70B, 405B) trained on 15T+ tokens with a heavy post-training pipeline. It established the strongest open-weight family in 2024, with practical details on data curation, scaling, long context, and multimodal extensions.

Key Idea

Llama 3 demonstrates that plain dense Transformers, scaled aggressively and over-trained beyond Chinchilla-optimal, plus a heavy post-training stack, can match frontier closed models. Three sizes (8B, 70B, and 405B) are trained on 15T+ tokens (well above Chinchilla-optimal for the smaller models, trading training compute for inference efficiency). Long context (up to 128K tokens) and multimodal capabilities (vision, speech, code) are added in staged extensions on top of the dense base.

How It Works

  • Architecture: Decoder-only Transformer with Grouped-Query Attention (GQA) for efficient KV cache, SwiGLU activations, RMSNorm, and RoPE positional embeddings. No MoE, no exotic tricks, bigger and better data is the story.
  • Tokenizer: 128K vocabulary (up from Llama 2's 32K), better compression, especially for non-English text and code.
  • Long context: 128K achieved by RoPE frequency scaling (rope_theta scaled from 500K to ~5M) and continued pretraining on long sequences.
  • Data: ~15T tokens of high-quality web data, heavily filtered with deduplication, quality classifiers, and annealing on small high-quality datasets at the end of training to boost benchmark performance.
  • Post-training: Multi-round pipeline of SFT → rejection sampling → DPO, with reward models trained on human preference data. No PPO, DPO is simpler and worked well enough.
  • Multimodal: Vision adapters and speech encoders trained separately and grafted onto the language backbone via cross-attention.
  • Infra: 405B trained on 16K H100 GPUs with 4D parallelism (tensor + pipeline + context + data); detailed reliability stats (hardware failures every ~3 hours at this scale).

Why It Matters

  • Llama 3 405B roughly matches GPT-4o and Claude 3.5 Sonnet on most benchmarks (MMLU ~88, HumanEval ~89), the first open-weight model to credibly compete at the frontier.
  • Llama 3.1 8B and 70B became the default starting point for fine-tuning, RAG, and downstream research across the open ecosystem.
  • Validated "over-training" small models as a deployment strategy: serving cost dominates training cost, so spending more tokens on smaller models pays off.
  • The detailed technical report (data mixing ratios, annealing schedules, failure modes) became a reference document for training large models.

Key Takeaways for Interviews

  • Llama 3 = dense Transformer + GQA + SwiGLU + RMSNorm + RoPE, scaled to 405B params and 15T tokens
  • Over-training smaller models past Chinchilla-optimal makes sense when inference cost matters more than training cost
  • Long context (128K) via RoPE frequency scaling + continued pretraining, not architectural changes
  • Post-training uses SFT + DPO + rejection sampling rather than PPO (simpler, more stable)
  • Multimodal capabilities are added as adapters on top of the LLM backbone, not joint pretraining
  • In system design: if asked to pick an open-weight base model, Llama 3.1 (8B/70B/405B) is the canonical choice; cite GQA + RoPE scaling for KV cache and long context