Language Models are Few-Shot Learners

Brown, Mann, Ryder et al. · 2020 · NeurIPS 2020

TrainingRead on arXiv

Introduced GPT-3 and demonstrated that large language models can perform new tasks through prompting and in-context examples without any gradient updates. This paper launched the era of prompt engineering.

Key Idea

GPT-3 showed that scaling up a language model to 175 billion parameters enables remarkable in-context learning, the model can perform tasks it was never explicitly trained for, simply by being given a few examples in the prompt.

In-Context Learning Paradigm

Zero-shot: describe the task in natural language, no examples
One-shot: provide a single example of the desired input-output behavior
Few-shot: provide several examples (typically 10 to 100) in the prompt
No gradient updates or fine-tuning required, the model "learns" from the prompt alone

Scale is the Key

175B parameters, trained on 300B tokens from a filtered internet corpus
Performance scales smoothly with model size across most benchmarks
Larger models are disproportionately better at in-context learning
Training cost: estimated $4.6M at the time

Why It Matters

Proved that scale unlocks emergent capabilities not present in smaller models
Shifted the paradigm from fine-tuning to prompt engineering
Demonstrated that a single model can perform hundreds of diverse tasks
Launched the commercial LLM era (OpenAI API, ChatGPT, etc.)

Key Takeaways for Interviews

In-context learning is not fine-tuning, no weights are updated
Few-shot learning improves with model scale (emergent ability)
GPT-3 is a decoder-only autoregressive model (left-to-right generation)
The paper raised important questions about memorization vs generalization