← All papers

Llama 2: Open Foundation and Fine-Tuned Chat Models

Touvron, Martin, Stone et al. (Meta) · 2023 · arXiv 2023

ArchitectureRead on arXiv

Meta's release of Llama 2, a family of openly available pretrained and fine-tuned chat models (7B, 13B, 70B) that closed much of the gap with closed-source models like GPT-3.5 and reshaped the open-source LLM ecosystem.

Key Idea

Llama 2 demonstrates that openly released foundation models, when carefully pretrained on 2T tokens and aligned via RLHF, can compete with proprietary systems like GPT-3.5. The release includes both base models and chat-tuned variants under a permissive license, catalyzing the open-source LLM ecosystem.

Architecture

  • Three sizes: 7B, 13B, and 70B parameters, covering deployment from single-GPU to multi-GPU clusters
  • Decoder-only Transformer with pre-normalization (RMSNorm), SwiGLU activations, and rotary positional embeddings (RoPE)
  • Grouped-Query Attention (GQA) in the 34B and 70B models, reduces KV cache memory by sharing keys/values across query heads, enabling faster inference
  • Context length of 4096 tokens: doubled from the original Llama
  • Pretrained on 2 trillion tokens of publicly available data, with extensive data cleaning and deduplication

Alignment Pipeline

  • Supervised Fine-Tuning (SFT) on ~28K high-quality human-written instruction examples
  • RLHF with two reward models: separate helpfulness and safety reward models trained on >1M human preference comparisons
  • Iterative refinement alternating between rejection sampling (sample N completions, fine-tune on the best) and PPO for the final stages
  • Ghost Attention (GAtt) trick to improve multi-turn instruction following

Safety

  • Extensive red-teaming by internal and external experts to surface harmful behaviors
  • A dedicated safety reward model trained on adversarial prompts ensures the model refuses unsafe requests
  • Released a detailed responsible use guide and model card

Why It Matters

  • Open weights changed the field, enabled research, fine-tuning, and downstream products without API gatekeeping
  • Llama 2-70B-Chat is competitive with GPT-3.5 on many benchmarks (MMLU, TriviaQA, helpfulness evals)
  • Spawned an enormous ecosystem: Llama.cpp, Code Llama, countless fine-tunes (Vicuna, Alpaca-style descendants), and became the default base for open RAG and agent systems

Key Takeaways for Interviews

  • Llama 2 = first widely-adopted open base model competitive with closed frontier models at the time
  • Know the GQA trick (memory-efficient attention) and RoPE (rotary embeddings, better length generalization than sinusoidal)
  • The alignment pipeline (SFT → reward modeling → rejection sampling + PPO) is the canonical RLHF recipe
  • Separate helpfulness and safety reward models is a notable design choice, single combined rewards often trade one off against the other
  • Open release vs closed release is a real systems-design tradeoff (control, safety, vs ecosystem and customization)