← All papers

DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

Shao, Wang, Zhu et al. (DeepSeek) · 2024 · arXiv 2024

RLRead on arXiv

Introduces Group Relative Policy Optimization (GRPO), a memory-efficient alternative to PPO that estimates baselines from group scores instead of a critic model, enabling scalable RL for mathematical reasoning.

Key Idea

GRPO (Group Relative Policy Optimization) eliminates the critic/value model used in PPO by estimating the baseline from group-level scores. For each prompt, it samples a group of responses, scores them with a reward model, and uses the group mean as the baseline (no separate value network needed). This cuts memory usage roughly in half while maintaining or improving performance.

How It Works

  • Group sampling: For each prompt x, sample G responses {y₁, ..., y_G} from the current policy
  • Group scoring: Score each response with a reward model to get {r₁, ..., r_G}
  • Relative advantage: Normalize rewards within the group: Â_i = (r_i - mean(r)) / std(r)
  • Policy update: Maximize the clipped PPO-style objective using group-relative advantages, with a KL penalty to the reference policy
  • No critic model → ~50% less GPU memory than PPO (only need policy + reference model, not policy + reference + critic + reward)

Why It Matters

  • Memory efficient: removes the critic model entirely, critical for RL on large LLMs where every model copy costs tens of GBs
  • DeepSeekMath results: GSM8K 82.9% → 88.2%, MATH 46.8% → 51.7% using GRPO on top of SFT
  • Foundation of DeepSeek-R1: GRPO is the core RL algorithm that trained DeepSeek-R1's reasoning capabilities, demonstrating that emergent chain-of-thought can arise from RL alone
  • Simpler than PPO: fewer hyperparameters, no value function to train, more stable optimization
  • Sparked a wave of PPO-free RL methods for LLMs: Dr. GRPO, GSPO, DAPO, and others

Key Takeaways for Interviews

  • GRPO = PPO without the critic, baseline estimated from group statistics instead of a learned value function
  • Key advantage: ~50% memory reduction vs PPO since you drop the critic model entirely
  • The group size G is a key hyperparameter (typically 8-64 responses per prompt), larger groups give more stable baselines but cost more compute
  • GRPO works because LLM rewards are relative, what matters is whether a response is better than peers, not its absolute score
  • Used in DeepSeek-R1 to train reasoning from scratch via RL, proving that CoT can emerge without supervised reasoning data
  • In system design: GRPO is the go-to when you want RLHF-style training but can't afford 4 models in memory (policy + ref + reward + critic)