← All papers

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

DeepSeek-AI · 2025 · arXiv 2025

ReasoningRead on arXiv

Demonstrates that pure reinforcement learning (without supervised fine-tuning on chain-of-thought data) can incentivize emergent reasoning in LLMs. DeepSeek-R1-Zero discovers chain-of-thought, self-verification, and reflection behaviors through RL alone, while DeepSeek-R1 adds a cold-start SFT phase to achieve OpenAI o1-level reasoning performance.

Key Idea

DeepSeek-R1 shows that reasoning is an emergent behavior of RL, not something that must be taught via supervised demonstrations. Starting from a base model with no chain-of-thought training (DeepSeek-R1-Zero), applying Group Relative Policy Optimization (GRPO) with only outcome-based rewards causes the model to spontaneously develop extended thinking, self-verification ("wait, let me check"), and structured problem decomposition.

How It Works

  • DeepSeek-R1-Zero: Apply GRPO directly to the base model (DeepSeek-V3-Base) with no SFT. Use simple reward signals, correctness for math/code (binary), format reward (ensure the model uses ... tags). The model naturally develops reasoning chains that grow from hundreds to thousands of tokens as training progresses.
  • Emergent behaviors: The model discovers reflection ("Hmm, wait, let me reconsider"), backtracking, and self-verification, none of which were taught. An "aha moment" occurs during training where the model learns to re-evaluate its initial answers.
  • DeepSeek-R1: To fix readability and language mixing issues from R1-Zero, a pipeline is used: (1) cold-start SFT with a small set of curated long-CoT examples, (2) reasoning-focused RL, (3) rejection sampling to create high-quality SFT data from the RL model, (4) final RL across all tasks (reasoning + general).
  • Distillation: R1's reasoning traces are distilled into smaller models (1.5B to 70B parameters based on Qwen and Llama), with the 32B distilled model outperforming OpenAI o1-mini on many benchmarks.

Why It Matters

  • RL alone produces reasoning: Overturns the assumption that step-by-step reasoning must be taught through demonstrations. The reward signal alone is sufficient.
  • Performance: R1 matches or surpasses OpenAI o1 on AIME 2024 (79.8% pass@1), MATH-500 (97.3%), and Codeforces (2,029 Elo rating).
  • Open weights: Unlike o1, DeepSeek-R1 is open-weight, enabling the research community to study and build on reasoning models.
  • Cost-effective: Built on the already efficient DeepSeek-V3 base (MLA + MoE), meaning reasoning capability is achievable without o1-scale compute.
  • Distillation works: Even small models (7B-32B) can acquire strong reasoning through distillation from R1, suggesting reasoning is more about training methodology than raw model size.

Key Takeaways for Interviews

  • GRPO: Group Relative Policy Optimization, samples multiple outputs per prompt, uses group-level mean/std for advantage normalization, no separate critic model needed
  • Emergent reasoning: the model discovers CoT, self-verification, reflection through RL alone, not programmed or demonstrated
  • Pipeline: cold-start SFT → reasoning RL → rejection sampling → general RL, each stage addresses specific weaknesses
  • Distillation finding: RL on small models directly is less effective than distilling from a large reasoning model, the large model discovers reasoning patterns that transfer well
  • Limitations: R1-Zero has readability issues (language mixing, poor formatting) that require the SFT cold-start fix in R1
  • In system design: cite when discussing reasoning-capable LLM serving (longer generation → higher latency/cost), test-time compute scaling, and RL post-training pipelines