← All papers

DAPO: An Open-Source LLM Reinforcement Learning System

ByteDance Seed · 2025 · arXiv 2025

RLRead on arXiv

Identifies and fixes four critical failure modes in applying GRPO-style reinforcement learning to LLMs: entropy collapse, reward hacking through length, training instability from long outputs, and loss of code formatting. Achieves 50% on AIME 2024 with a 32B model through systematic RL engineering.

Key Idea

DAPO (Decoupled Clip and Dynamic sAmpling Policy Optimization) addresses the engineering reality of RL for LLM reasoning: naive GRPO-style training collapses within hundreds of steps due to entropy loss, reward hacking, and instability. The paper systematically diagnoses each failure mode and proposes targeted fixes, turning RL for reasoning from a fragile research prototype into a reproducible system.

Key Techniques

  • Decoupled clipping: Standard PPO/GRPO clips both positive and negative advantages symmetrically (e.g., [1-ε, 1+ε]). DAPO uses a larger upper clip (e.g., [1-0.2, 1+0.28]) to encourage exploration of new successful strategies while still penalizing failures. This prevents entropy collapse, the #1 failure mode.
  • Dynamic sampling: Skip prompts where all G samples are correct (nothing to learn) or all wrong (no positive signal). Only train on prompts with mixed outcomes where the model can learn from contrast. This improves sample efficiency by 2-4x.
  • Token-level loss: Instead of sequence-level loss (dividing by number of sequences), divide by total tokens in the batch. This prevents long outputs from dominating the gradient, stabilizing training on math/code tasks where output length varies wildly (100 to 16K tokens).
  • Overlong reward shaping: Assign a soft penalty score (e.g., -1.0) to outputs truncated at max length, rather than the hard 0 used by GRPO. This teaches the model to be concise without binary "all or nothing" signal.
  • Soft format enforcement: Instead of a binary format reward (0 or 1 for having ... tags), use a decaying format reward that starts at 1.0 and decreases to 0 over training. This prevents the model from getting stuck in format-hacking local optima.

Why It Matters

  • Reproducible RL for reasoning: Most RL reasoning papers (including DeepSeek-R1) don't share enough training details to reproduce. DAPO is fully open-source with training code, providing a blueprint.
  • 50% AIME 2024: A 32B model (Qwen2.5-32B base) achieves 50% pass@1 on AIME 2024, competitive with much larger reasoning models. Through RL engineering alone.
  • Practical lessons: Each technique addresses a real failure mode that practitioners encounter. The paper is essentially a troubleshooting guide for RL post-training.
  • Entropy preservation: The decoupled clipping insight is broadly applicable, any PPO/GRPO-style training that collapses to deterministic outputs can benefit.

Key Takeaways for Interviews

  • Entropy collapse: the primary failure mode of RL on LLMs, the model converges to a single output pattern and stops exploring. Fix: asymmetric clipping with larger upper bound.
  • Dynamic sampling: skip prompts where the model either always succeeds or always fails, only train on informative contrasts
  • Token-level vs sequence-level loss: when output lengths vary 100x (math proofs vs simple answers), sequence-level loss lets long outputs dominate gradients
  • DAPO vs GRPO vs PPO: DAPO = GRPO + decoupled clip + dynamic sampling + token-level loss + overlong shaping. No critic model (like GRPO), but more stable.
  • In system design: cite when discussing RL post-training pipelines, reasoning model training, or the engineering challenges of scaling RL beyond toy tasks