← All papers

Training Compute-Optimal Large Language Models

Hoffmann, Borgeaud, Mensch et al. · 2022 · NeurIPS 2022

TrainingRead on arXiv

Showed that most large language models were significantly undertrained: for a given compute budget, model size and training data should be scaled equally. Chinchilla (70B) outperformed Gopher (280B) with 4x fewer parameters.

Key Idea

The earlier Kaplan scaling laws suggested scaling model size faster than data. Chinchilla overturned this by showing that for compute-optimal training, model parameters and training tokens should be scaled in roughly equal proportion. Most existing LLMs were too large and undertrained.

The Scaling Law Revision

  • Kaplan (2020): for a 10x compute increase, scale model ~5.5x and data ~1.8x
  • Chinchilla (2022): for a 10x compute increase, scale model ~3.2x and data ~3.2x (roughly equal)
  • Rule of thumb: ~20 tokens per parameter for compute-optimal training
  • Gopher (280B params, 300B tokens) was 4x oversized, Chinchilla (70B params, 1.4T tokens) performed better

Experimental Approach

Three independent methods to estimate optimal allocation, all converging on the same conclusion:

  1. Fixed compute budgets with varying model/data ratios
  2. IsoFLOP curves (same total compute, different splits)
  3. Fitting a parametric loss function to all runs

Why It Matters

  • Changed how the industry trains LLMs: LLaMA, Mistral, and most modern models follow Chinchilla-optimal ratios
  • Showed that data is more valuable than previously thought, triggered investment in data quality and curation
  • Proved that smaller, well-trained models can beat larger, undertrained ones, more cost-effective for inference
  • Made compute-optimal planning a standard part of LLM development

Key Takeaways for Interviews

  • Know both scaling laws: Kaplan (favor model size) vs Chinchilla (equal scaling of model and data)
  • ~20 tokens per parameter is the Chinchilla-optimal ratio
  • In practice, many teams now "over-train" smaller models (e.g., LLaMA trained on far more tokens than Chinchilla-optimal) because inference cost matters more than training cost
  • The tradeoff: Chinchilla-optimal minimizes training compute, but smaller over-trained models minimize serving cost