← All papers

Proximal Policy Optimization Algorithms

Schulman, Wolski, Dhariwal, Radford, Klimov · 2017 · arXiv 2017

RLRead on arXiv

Introduced a simple yet effective policy gradient method that uses a clipped surrogate objective to prevent destructively large updates. PPO became the go-to RL algorithm and is the backbone of RLHF for LLM alignment.

Key Idea

Policy gradient methods can be unstable, too large an update can destroy a good policy. TRPO (Trust Region Policy Optimization) solved this with constrained optimization, but was complex to implement. PPO achieves similar stability with a much simpler approach: clip the objective function to prevent the policy ratio from moving too far from 1.

The Clipped Objective

  • Policy ratio: r(θ) = π_θ(a|s) / π_θ_old(a|s) (how much the new policy differs from the old)
  • Clipped objective: L = min(r(θ)·Â, clip(r(θ), 1-ε, 1+ε)·Â)
  • ε is typically 0.1-0.2
  • If the advantage  > 0 (good action): don't let r(θ) go above 1+ε (limits reward for over-exploiting)
  • If the advantage  < 0 (bad action): don't let r(θ) go below 1-ε (limits penalty for over-correcting)

Why Clipping Works

  • Forms a pessimistic bound, takes the minimum of the clipped and unclipped objective
  • Prevents destructively large policy updates without the complexity of TRPO's KL constraint
  • Simple to implement: ~20 lines of code beyond standard policy gradient
  • Robust to hyperparameters, works well with default settings across diverse tasks

Why It Matters

  • Became the most widely used RL algorithm in both research and production
  • Powers RLHF: PPO is the default RL step in InstructGPT, ChatGPT, and most aligned LLMs
  • Used in OpenAI Five (Dota 2), robotic control, game playing, and LLM alignment
  • Simple enough to be a teaching algorithm, powerful enough for frontier research

Key Takeaways for Interviews

  • PPO = policy gradient + clipped surrogate objective to limit update size
  • The clip range ε controls how far the new policy can deviate from the old one
  • PPO is on-policy (uses data from the current policy), making it less sample-efficient than off-policy methods
  • In the RLHF pipeline: the reward model provides the reward signal, PPO optimizes the LLM policy against it
  • GRPO and DPO are newer alternatives that simplify or replace PPO for LLM alignment