LLM EngineeringHard

🎯 RLHF & Alignment

Aligning LLMs with human preferences using RLHF, DPO, and modern preference optimization methods

What is RLHF and Why It Matters

Reinforcement Learning from Human Feedback (RLHF) is the dominant post-training technique for aligning large language models with human preferences. Pretrained LLMs are trained to predict the next token over web-scale text; they learn to mimic the distribution of internet text, including unhelpful, untruthful, or harmful patterns. RLHF shapes the model's behavior to be helpful, honest, and harmless by optimizing against human (or AI) preference signals rather than next-token likelihood.

RLHF was the key innovation behind InstructGPT (2022), ChatGPT, Claude, and Gemini. Without it, raw pretrained models like base GPT-4 are notoriously hard to use: they continue prompts rather than following instructions, and they readily produce harmful or hallucinated content. RLHF turns a "text predictor" into an "assistant."

The 3-Step RLHF Pipeline

The canonical RLHF recipe (from InstructGPT, Ouyang et al. 2022) has three stages:

Step 1: Supervised Fine-Tuning (SFT)

Start from a pretrained base model. Collect high-quality demonstrations of instruction-following from human annotators (or curated datasets like ShareGPT, OpenAssistant). Fine-tune on these (prompt, response) pairs with standard cross-entropy loss. The result is a model that follows instructions reasonably well but may still be inconsistent, verbose, or unsafe.

Step 2: Reward Model (RM) Training

Collect preference data: for each prompt, the SFT model generates multiple completions, and humans rank or choose between them. The reward model (typically the SFT model with a scalar head replacing the LM head) is trained to predict which response humans prefer. The standard loss is the Bradley-Terry pairwise loss:

L_RM = -E[(x, y_w, y_l)] [ log σ(r(x, y_w) - r(x, y_l)) ]

where y_w is the preferred (winning) response and y_l is the rejected (losing) response. The RM learns a scalar score reflecting human preference.

Step 3: RL Optimization (PPO)

The policy (the SFT model) is fine-tuned with Proximal Policy Optimization (PPO) to maximize the reward model's score, while a KL penalty against the SFT reference policy prevents the model from drifting too far from coherent language:

objective = E[r(x, y) - β · KL(π_RL(y|x) || π_SFT(y|x))]

The KL penalty is crucial. Without it, the policy "reward hacks" by producing degenerate text that the RM happens to score highly. The KL coefficient β trades off reward maximization against staying close to the SFT model.

PPO is complex: it requires four models in memory (policy, reference, reward, value/critic), is sensitive to hyperparameters, and is hard to stabilize at scale. This is the motivation for DPO and related methods.

RLHF's model budget: PPO's four models vs DPO/GRPO's two

Models resident in GPU memory (1 of 4)

Policytrainable

112 GB

Reference

not needed

Reward model

not needed

Critic

not needed

Total per-GPU footprint112 GB

Trainable (weights + grads + optimizer ≈16 B/param)Frozen (weights only ≈2 B/param)

Step 1 — SFT. Alignment starts by supervised fine-tuning the base model on good (prompt, response) demonstrations. Only one model is in memory and it’s trainable, so it carries weights, gradients, and optimizer state (~16 bytes/param). This is behaviour cloning — it teaches the format of a helpful answer and gives RL a sane starting point.

Step 2 — a reward model. To go beyond imitation you need a signal for what humans prefer. Collect pairwise comparisons and train a reward model with the Bradley-Terry loss −log σ(r(y_w) − r(y_l)), so it scores a winning response above a losing one. That frozen reward model now joins the picture as one of the models RL will need to keep resident.

Step 3 — PPO, and the four-model tax. Classic RLHF optimizes the policy against the reward model with PPO, plus a KL leash to a frozen reference so it can’t drift into gibberish that games the reward. That means four models in memory at once: the trainable policy and critic, and the frozen reference and reward model. The bar balloons — this is why PPO RLHF is so resource-hungry.

DPO collapses it to two. Direct Preference Optimization skips the reward model and the RL loop entirely: a closed-form loss trains the policy straight on preference pairs against the frozen reference, treating the model itself as the implicit reward. No reward model, no critic — two models, roughly half the memory, and a far simpler, more stable loop. This is why DPO became the default for preference alignment.

GRPO drops the critic. For reasoning with verifiable rewards, GRPO keeps PPO’s clipped objective but replaces the critic with a group baseline — the mean reward over several sampled responses. So it also runs on just two resident models (policy + reference), with the reward coming from a verifier rather than a resident network. Half the memory of PPO, and a natural fit for math and code. See the group baseline in the playground →

DPO: Direct Preference Optimization

DPO (Rafailov et al., 2023) is the most important alignment development of the last few years. It eliminates the reward model and RL training entirely by deriving a closed-form mapping between the optimal RLHF policy and the preference data. The DPO loss is:

L_DPO = -E[(x, y_w, y_l)] [ log σ(β · log(π_θ(y_w|x)/π_ref(y_w|x))
                                   - β · log(π_θ(y_l|x)/π_ref(y_l|x))) ]

Conceptually, DPO treats the LM itself as an implicit reward model. It increases the relative log-probability of preferred responses while keeping the model close to the reference via the implicit KL constraint encoded in β.

Advantages over PPO:

No separate reward model to train
No RL machinery (no rollouts, no value function, no PPO clipping)
More stable, simpler to implement (one training loop, two model copies)
Often matches or beats PPO on standard benchmarks

Disadvantages:

Sensitive to the reference model; drift from π_ref can be hard to control
Can over-optimize on chosen examples and reduce diversity
Less expressive than PPO when reward signal comes from non-preference sources

KTO, ORPO, and Modern Alignment (2024 to 2025)

The post-DPO landscape has rapidly diversified:

KTO (Kahneman-Tversky Optimization, 2024): Replaces pairwise preferences with binary "good/bad" labels per response, drawing on prospect theory. Useful when you only have thumbs-up/down feedback and no paired comparisons, which matches how production systems actually collect data.

ORPO (Odds Ratio Preference Optimization, 2024): Combines SFT and preference learning into a single stage by adding an odds-ratio preference term to the standard SFT loss. Removes the need for a separate reference model, cutting memory and complexity. Popular for fine-tuning open-weight models on consumer hardware.

IPO (Identity Preference Optimization, 2023): Addresses DPO's tendency to over-fit on preference data by replacing the log-sigmoid loss with a squared-error formulation that doesn't saturate.

SimPO (2024): Removes the reference model from DPO by using length-normalized log-probabilities, simplifying training further.

GRPO (Group Relative Policy Optimization, DeepSeek 2024): A PPO variant that drops the value network by using group-relative baselines (multiple samples per prompt). Used in DeepSeek-R1 reasoning training and has become a standard for RL on math/code with verifiable rewards. The group-relative advantage playground shows how a group of rewards becomes advantages with no critic — and how the signal collapses when every response scores the same.

As of 2025, the practical recipe is: SFT, then DPO (or ORPO if budget-constrained) for general alignment, plus GRPO with verifiable reward signals for reasoning capabilities.

Constitutional AI and RLAIF

Constitutional AI (CAI, Anthropic 2022) trains a model to critique and revise its own outputs against a written constitution (a list of principles like "be helpful" and "avoid harm"). The pipeline:

SFT on revised responses produced by the model itself critiquing initial outputs against the constitution
RLAIF: the model generates preference labels (instead of humans), trained against the constitution-following prompt

RLAIF (RL from AI Feedback) is now standard practice, using a strong "judge" LLM (often GPT-4 / Claude / Gemini-class) to label preferences at scale. It's vastly cheaper than human labeling and often comparable in quality for many domains. Hybrid pipelines are common: humans label the most difficult or safety-critical pairs, while AI handles the rest.

Practical Considerations

Preference Data Collection: The single biggest determinant of RLHF quality. Tips: high inter-annotator agreement (>70%), diverse annotators, careful prompt curation covering real use cases, length-balanced pairs (humans heavily prefer longer responses, which biases models toward verbosity).

Reward Hacking: The model finds patterns the RM scores highly but humans don't actually want: excessive hedging, sycophancy, gratuitous formatting, ASCII art, refusals on safe prompts. Mitigations: KL penalties, ensemble RMs, periodic RM retraining on new failure modes, length normalization.

Alignment Tax: RLHF often degrades capabilities measured on benchmarks like MMLU, code generation, or math. Causes include over-conservative refusals, distribution shift from SFT/RL data, and capacity used for safety patterns. Mitigations: mix pretraining/instruction data into the RLHF stage, careful reward model design, light-touch KL constraints.

Evaluation: Notoriously hard. Common approaches:

LLM-as-judge: AlpacaEval, MT-Bench, Arena-Hard. Cheap but biased toward verbose answers and the judge's style.
Human evaluations: Chatbot Arena (pairwise human comparisons), gold standard but expensive.
Safety: HarmBench, ToxicChat, jailbreak suites.
Capability preservation: Always run MMLU, HumanEval, GSM8K pre/post-RLHF to catch the alignment tax.