← All papers

Direct Preference Optimization: Your Language Model Is Secretly a Reward Model

Rafailov, Sharma, Mitchell et al. · 2023 · NeurIPS 2023

AlignmentRead on arXiv

DPO reparameterizes RLHF so the policy is optimized **directly from preference pairs**, no separate reward model, no PPO loop, no on-policy sampling. The result matches or beats PPO-based RLHF at a fraction of the engineering complexity.

Key Idea

Classic RLHF has two heavy stages: (1) train a reward model on human preferences, (2) run PPO against that reward model with a KL penalty to the SFT reference. DPO collapses both into a single supervised loss on preference pairs by showing that, under the Bradley-Terry preference model, the optimal RLHF policy has a closed-form relationship with the reward, so you can solve for the policy directly.

How It Works

  1. Bradley-Terry preference model: The probability that human prefers response y_w over y_l given prompt x is σ(r(x, y_w) − r(x, y_l)).
  2. Closed-form optimal policy: The RLHF objective max E[r(x,y)] − β·KL(π || π_ref) has the analytic solution π*(y|x) ∝ π_ref(y|x) · exp(r(x,y)/β).
  3. Reparameterize: Invert that relationship to express r in terms of π* and π_ref, then substitute into the Bradley-Terry likelihood. The reward model cancels out.
  4. DPO loss is just −log σ(β · log(π(y_w|x)/π_ref(y_w|x)) − β · log(π(y_l|x)/π_ref(y_l|x))), a plain classification loss over preference pairs, trained with standard SGD.

Why It Matters

  • No reward model, no RL: a single training run on (prompt, chosen, rejected) triples, using the same infrastructure as SFT
  • Matches or beats PPO-RLHF on summarization (TL;DR), single-turn dialogue (Anthropic HH), and controlled sentiment, at lower cost and with more stable training
  • β plays the role of the KL coefficient: higher β keeps the policy closer to π_ref (safer, less drift); lower β allows bigger shifts (more reward, more risk of mode collapse)
  • Spawned a family of offline preference optimizers: IPO (fixes DPO overfitting on deterministic preferences), KTO (works with unpaired binary feedback), ORPO (combines SFT + preference loss, removes the reference-model requirement), SimPO, CPO
  • Modern open-weight chat models (Llama 3 Instruct, Mistral, Qwen, Zephyr) ship DPO/DPO-variants as the final alignment stage

Key Takeaways for Interviews

  • DPO = closed-form solution to RLHF when you assume a Bradley-Terry preference model, the reward is implicit in log(π/π_ref)
  • Trades two-stage RLHF (RM + PPO) for a single supervised loss on preference pairs, much simpler engineering
  • Offline and on-policy-free: you never need the current policy to sample, so it's trivially distributed and batched
  • β controls KL to the reference policy: too low and the model drifts/hacks the preferences; too high and it barely moves from SFT
  • When to prefer PPO over DPO: when you need an explicit reward model (multi-objective, reward shaping, online iteration with fresh samples) or when preferences are very noisy
  • In system design: cite DPO as the modern default for alignment fine-tuning; mention KTO/ORPO as variants for unpaired or reference-free settings