← All papers

RewardBench: Evaluating Reward Models for Language Modeling

Lambert, Pyatkin, Morrison et al. · 2024 · arXiv 2024

EvaluationRead on arXiv

Introduces a comprehensive benchmark for evaluating reward models, a critical but under-studied component of RLHF pipelines. Reveals that reward models struggle with subtle reasoning, safety boundary cases, and instruction following, even when they perform well on simple preference tasks.

Key Idea

Reward models (RMs) are the backbone of RLHF, they encode human preferences and guide policy optimization. Yet before RewardBench, there was no standardized way to evaluate them. This paper creates a benchmark with four categories: Chat (helpfulness), Chat Hard (subtle preference distinctions), Safety (refusing harmful requests while being helpful for benign ones), and Reasoning (math and code correctness). The benchmark reveals that even top RMs have significant blind spots.

How It Works

  • Evaluation format: Each example is a (prompt, chosen, rejected) triple. The RM should score the chosen response higher. Accuracy is the metric.
  • Chat: Standard helpfulness preferences from human annotators (relatively easy, top RMs get 95%+)
  • Chat Hard: Subtle distinctions where responses are close in quality. Tests whether the RM captures nuanced preferences (many RMs drop to ~65%)
  • Safety: Pairs of (helpful refusal, harmful compliance) for dangerous prompts, and (helpful response, unnecessary refusal) for benign prompts. Tests calibration of safety behavior.
  • Reasoning: Pairs where one response has correct math/code and the other has a subtle bug. Tests whether RMs can verify correctness vs just preferring fluent text.

Why It Matters

  • RM quality directly affects RLHF quality: if the reward model can't distinguish good from bad reasoning, RL will optimize for surface patterns (reward hacking)
  • Revealed weaknesses: even the best RMs (GPT-4-as-judge) struggle on Chat Hard and Reasoning categories, suggesting RLHF pipelines may be fundamentally limited by RM quality
  • Generative vs classifier RMs: LLM-as-judge (e.g., GPT-4, Claude) often outperforms trained classifier RMs, especially on reasoning tasks, but at much higher cost
  • Safety calibration: many RMs are either too aggressive (refusing benign requests) or too permissive, finding the right boundary is hard to learn from preference data alone

Key Takeaways for Interviews

  • RLHF bottleneck: the reward model is often the weakest link, if it can't evaluate reasoning, the policy will learn to game it
  • Four categories: chat (easy), chat-hard (subtle preferences), safety (refusal calibration), reasoning (correctness verification), RM performance varies dramatically across them
  • LLM-as-judge: using a strong LLM to evaluate responses often outperforms trained RMs, especially for reasoning, but has cost and latency implications for online RL
  • Process vs outcome RMs: RewardBench evaluates outcome RMs; process reward models (Math-Shepherd) that score intermediate steps are a promising direction for reasoning
  • In system design: cite when discussing reward model selection, RLHF pipeline design, or evaluation infrastructure for alignment