← All papers

Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

Bai, Jones, Kaplan et al. (Anthropic) · 2022 · arXiv 2022

AlignmentRead on arXiv

Anthropic's foundational RLHF paper that introduced the HH (Helpful and Harmless) dataset and methodology. It explored the tension between helpfulness and harmlessness, compared online vs offline RLHF, and released a preference dataset that became a standard alignment benchmark used across the field.

Key Idea

This paper systematically studies how to train a general-purpose conversational assistant that is simultaneously helpful and harmless using RLHF. It frames alignment as a dual-objective preference learning problem, collects two separate preference datasets (one for helpfulness, one for harmlessness), and shows that online iterated RLHF (repeatedly collecting fresh preference data on the current policy) produces substantially better assistants than a single round of offline RLHF.

How It Works

  1. Data collection: Crowdworkers chat with the assistant in two modes:
    • Helpfulness: workers try to get useful answers, choose between two model responses
    • Harmlessness: workers try to elicit harmful responses, choose the less harmful one (red-teaming)
      This produces the HH-RLHF dataset, tens of thousands of preference comparisons across both axes.
  2. Preference model (PM): A Transformer trained to predict which of two responses a human would prefer. Same architecture and pretraining as the base LM, fine-tuned on the comparison data.
  3. RL fine-tuning: PPO against the PM reward, with a KL penalty to the SFT policy to prevent reward hacking and distribution collapse.
  4. Online iteration: Periodically deploy the current RLHF policy, collect new preference comparisons on its actual outputs, retrain the PM, and rerun PPO. This is the iterated online RLHF loop, which closes the gap between the PM's training distribution and the policy's actual output distribution.
  5. Evaluation: Elo ratings from head-to-head human comparisons against baselines, plus per-axis helpfulness and harmlessness scores.

Why It Matters

  • Released the HH dataset publicly: it became a de facto standard benchmark for alignment research and is used as the training set in DPO, IPO, KTO, and many other follow-up papers.
  • Quantified the helpfulness/harmlessness tension: naively training for harmlessness alone produces an evasive, useless assistant; jointly training is non-trivial and the paper provides practical recipes.
  • Online RLHF > offline RLHF: showed that iterating data collection on the current policy is critical, motivating the staged training pipelines now standard at frontier labs.
  • Preference model scaling laws: larger PMs are dramatically more sample-efficient and produce better policies, alignment benefits from scale just like pretraining.
  • Established Anthropic's alignment research program and directly informed the design of Claude, while also providing a public reproduction of the RLHF recipe.

Key Takeaways for Interviews

  • HH-RLHF = the dataset (helpful + harmless preference pairs) and the methodology behind early Claude models
  • Helpfulness and harmlessness trade off: you need both axes in your preference data or the model collapses into "I can't help with that"
  • Online iterated RLHF beats single-round offline RLHF, the PM must see the policy's actual distribution
  • Preference models scale: bigger PMs give more reliable reward signal and better downstream policies
  • PPO + KL penalty to the SFT baseline is the standard RLHF objective; KL prevents reward hacking and distribution drift
  • In system design: cite HH-RLHF as the canonical example of a public RLHF dataset and methodology; contrast with DPO (offline, no PM needed) when discussing modern alignment pipelines