Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations

Key Idea

Process reward models (PRMs) score each intermediate reasoning step, not just the final answer, allowing verification systems to catch errors early and guide search over reasoning paths. The core challenge is that step-level labels traditionally require human annotators to judge every step, prohibitively expensive at scale. Math-Shepherd solves this with automatic step-level annotation: complete the reasoning from each step multiple times, and if most completions reach the correct final answer, that step is labeled correct.

How It Works

• Step-level label generation: Given a solution with steps s₁, s₂, ..., sₖ, Math-Shepherd generates M completions starting from each step sᵢ. If the fraction of completions reaching the correct answer exceeds a threshold, step sᵢ is labeled as correct; otherwise it is labeled incorrect.
• PRM training: A reward model is trained on these automatically generated step-level labels to predict the correctness of each reasoning step. The PRM takes a partial solution (prompt + steps so far) and outputs a score for the latest step.
• Verification at inference: Given a problem, sample N candidate solutions from the LLM, score each solution's steps with the PRM, and select the solution with the highest minimum step score (or highest average score). This is best-of-N with process-level scoring.
• Reinforcement learning: The PRM can also provide step-level rewards for RL fine-tuning (PPO), giving the policy gradient signal at each reasoning step rather than only at the end, denser supervision leads to faster, more stable training.

Why It Matters

• PRM vs ORM distinction: Outcome reward models (ORMs) only check the final answer, they can't distinguish a solution that got lucky from one with sound reasoning. PRMs verify process correctness, catching errors at the step where they occur. This is critical for multi-step math, code generation, and any task where intermediate correctness matters.
• Automatic annotation removes the bottleneck: Human step-level labeling costs ~$1-2 per solution and doesn't scale. Math-Shepherd's completion-based approach generates millions of step labels using only final-answer verification, which is cheap and automatable.
• Enables test-time compute scaling: PRMs power more sophisticated search strategies at inference, beam search over reasoning steps, Monte Carlo Tree Search (MCTS) over reasoning paths, and best-of-N with step-level reranking. Each of these trades inference compute for accuracy, and PRMs make the search much more efficient than ORMs.
• Results: On GSM8K and MATH, PRM-guided search with Math-Shepherd substantially outperforms ORM-guided search at the same compute budget (e.g., best-of-N with PRM beats best-of-N with ORM by 5-10% at N=64). PRM-based PPO also outperforms ORM-based PPO.
• Connection to o1/o3: The reasoning models from OpenAI (o1, o3) and DeepSeek-R1 are widely believed to use process reward models internally for guiding their extended chain-of-thought. Math-Shepherd is the foundational open-source work demonstrating this approach.

Key Takeaways for Interviews

• Know the PRM vs ORM distinction: ORMs score final answers only; PRMs score each reasoning step, PRMs enable finer-grained search and catch errors earlier
• Automatic step labeling via completion: generate M completions from step k, check if they reach the correct answer, no human annotators needed for step-level supervision
• Connection to test-time compute scaling: PRMs are the value function that makes MCTS and beam search over reasoning paths effective, without a good step-level scorer, search is blind
• PRMs enable both inference-time search (best-of-N, beam search, MCTS) and training-time RL (step-level rewards for PPO/GRPO)
• In system design: cite Math-Shepherd when discussing reasoning verification, test-time compute scaling, or why o1-style models need process-level reward signals rather than just outcome-level