← All papers

Training Language Models to Follow Instructions with Human Feedback

Ouyang, Wu, Jiang et al. · 2022 · NeurIPS 2022

TrainingRead on arXiv

Introduced InstructGPT and showed how reinforcement learning from human feedback (RLHF) can make language models more helpful, harmless, and aligned with user intent. This is the paper behind ChatGPT.

Key Idea

Raw language models are trained to predict the next token, which doesn't directly optimize for being helpful or safe. InstructGPT aligns a pretrained GPT-3 with human preferences using a three-step process: supervised fine-tuning (SFT), reward model training, and reinforcement learning with PPO.

The RLHF Pipeline

  1. Supervised Fine-Tuning (SFT): fine-tune GPT-3 on high-quality demonstrations written by human labelers
  2. Reward Model (RM) Training: collect human comparisons of model outputs (which response is better?) and train a model to predict these preferences
  3. RL Fine-Tuning (PPO): use the reward model as a reward signal to further fine-tune the SFT model using Proximal Policy Optimization

Results

  • InstructGPT (1.3B params) was preferred over GPT-3 (175B params) by human evaluators
  • Significantly reduced hallucinations, toxic outputs, and instruction-following failures
  • A 100x smaller model aligned with RLHF outperforms a 100x larger unaligned model

Why It Matters

  • Established RLHF as the standard alignment technique for production LLMs
  • Showed that alignment is more important than raw scale for user-facing applications
  • Directly led to ChatGPT (GPT-3.5 + RLHF) and influenced every major LLM release since
  • Opened research into DPO, RLAIF, and other alignment alternatives

Key Takeaways for Interviews

  • RLHF has three stages: SFT → Reward Model → PPO optimization
  • The reward model learns from human preference rankings, not absolute scores
  • Alignment tax is small, RLHF slightly reduces performance on some benchmarks but dramatically improves usability
  • DPO (Direct Preference Optimization) later simplified this by removing the explicit RM step