← All papers

Constitutional AI: Harmlessness from AI Feedback

Bai, Kadavath, Kundu et al. (Anthropic) · 2022 · arXiv 2022

AlignmentRead on arXiv

Constitutional AI (CAI) replaces most human harmlessness labels with **AI-generated feedback** guided by a written set of principles (the "constitution"). The result is a scalable, more transparent alignment pipeline, and the origin of RLAIF.

Key Idea

Scaling RLHF for harmlessness is bottlenecked by human labelers: producing red-team prompts and harm comparisons is slow, expensive, and ethically taxing. CAI shows that a capable model can critique and revise its own outputs against a small written constitution, then act as the preference labeler for RL, reducing human input to "helpful" labels and a few dozen principles.

How It Works

CAI runs in two phases:

  1. Supervised stage (SL-CAI): distill a "harmless-but-helpful" prior into the model:
    • Sample a response from a helpful-only RLHF model to a red-team prompt
    • Ask the same model to critique its response against a randomly drawn constitutional principle ("Identify ways this response is harmful, unethical, racist, …")
    • Ask the model to revise its response based on the critique
    • Fine-tune the model on the (prompt → revised response) pairs
  2. RL stage (RL-CAI / RLAIF): replace the human harmlessness preference model:
    • For each prompt, sample two responses from the SL-CAI model
    • Ask the model: "Given this principle, which response is better?", log-probs over (A) vs (B) give a preference distribution
    • Train a preference model on these AI labels, then run standard PPO against it (with a KL penalty to the SL-CAI reference)
    • Human labels are still used for helpfulness; harmlessness is fully AI-labeled

Why It Matters

  • RLAIF is born: shows that AI feedback can substitute for human feedback on subjective dimensions like harm, now standard practice (Anthropic, Google, OpenAI all use AI judges)
  • Pareto improvement on the helpful/harmless frontier vs human-only RLHF in Anthropic's evals, CAI models refuse less while also being safer
  • Transparency: the constitution is a short, human-readable document, much easier to audit and iterate than rater guidelines buried in contractor onboarding
  • Chain-of-thought reasoning helps: asking the model to reason about which principle applies before choosing improves label quality
  • Foundation of modern AI-judge / LLM-as-a-judge pipelines that scale eval and preference labeling

Key Takeaways for Interviews

  • CAI = RLHF for harmlessness with AI labels guided by a written constitution; humans only label helpfulness
  • Two phases: SL-CAI (self-critique + self-revision → SFT) then RL-CAI (AI preference model → PPO)
  • Reduces dependence on human red-teamers, makes the safety spec explicit and editable rather than implicit in rater guidelines
  • Started the RLAIF trend, AI judges are now used widely for both training and evaluation
  • Limitation: the model's biases propagate into its own labels; pairing with periodic human audits is standard
  • In system design: cite CAI when discussing scalable alignment, AI-judge eval pipelines, or when human labeling is the bottleneck