Scaling Laws for Neural Language Models

Key Idea

Language model loss follows smooth power-law relationships with respect to model size (N), dataset size (D), and compute budget (C). These relationships are remarkably predictable and hold across several orders of magnitude.

The Scaling Laws

• Loss ∝ N^{-α}: performance improves as a power law of model parameters
• Loss ∝ D^{-β}: performance improves as a power law of training data size
• Loss ∝ C^{-γ}: performance improves as a power law of compute budget
• These relationships are smooth, with no sharp transitions or plateaus

Practical Insights

• Model size matters most: for a fixed compute budget, larger models trained for fewer steps outperform smaller models trained longer
• Data and model size should scale together: but model size should grow faster than dataset size
• Early stopping is efficient: you can predict final performance from early training curves
• Overfitting occurs when dataset size is too small relative to model size

Why It Matters

• Gave teams a quantitative framework for compute-optimal training decisions
• Influenced the development of GPT-3, GPT-4, and all frontier models
• Later refined by the Chinchilla paper (Hoffmann et al., 2022), which argued data should be scaled more aggressively
• Established that throwing more compute at larger models is a reliable path to improvement

Key Takeaways for Interviews

• Power-law scaling means diminishing returns, each 10x compute yields a fixed improvement
• Kaplan's laws favor larger models; Chinchilla favors more data, know both perspectives
• Scaling laws help predict training costs and performance before committing resources
• These laws hold for loss, but downstream task performance can be less predictable