← All papers

Group Sequence Policy Optimization

Zheng, Liu, Li, Chen et al. (Qwen / Alibaba) · 2025 · arXiv 2025

RLRead on arXiv

Advances beyond GRPO by performing policy optimization at the sequence level rather than token level, achieving superior stability for large-scale MoE model training and powering the Qwen3 model family.

Key Idea

GSPO (Group Sequence Policy Optimization) redefines the importance ratio in policy optimization to operate at the sequence level instead of the token level. By using length-normalized sequence likelihoods and sequence-level clipping, GSPO avoids the instabilities that plague token-level methods (like GRPO) when training large Mixture-of-Experts models, where expert routing can cause wild token-level probability fluctuations.

How It Works

  • Sequence-level importance ratio: s_i(θ) = (π_θ(y_i|x) / π_old(y_i|x))^(1/|y_i|), the ratio of new-to-old policy likelihood, raised to the power of 1/sequence_length for normalization
  • Sequence-level clipping: clip the entire sequence ratio (not individual tokens), then apply rewards and optimize
  • Length normalization: the 1/|y_i| exponent unifies the numerical range across different response lengths, reducing variance
  • No Routing Replay needed: GRPO requires caching and replaying expert routing decisions for MoE stability; GSPO's sequence-level approach is inherently insensitive to token-level expert routing changes

Why It Matters

  • Powers Qwen3: GSPO is the core RL algorithm behind the Qwen3 model family, delivering significant improvements over GRPO-trained baselines
  • MoE stability: token-level methods like GRPO suffer when expert routing changes between the old and new policy (different tokens hit different experts). GSPO sidesteps this entirely by working at the sequence level
  • Simpler infrastructure: eliminates the need for Routing Replay, a complex caching workaround that GRPO requires for MoE training stability
  • Scalable: demonstrates continuous performance improvement with more training compute, regular query set updates, and longer generation lengths
  • Precision-tolerant: sequence-level likelihoods are more robust to FP16/BF16 precision differences, potentially allowing direct use of inference engine probabilities without recomputation

Key Takeaways for Interviews

  • GSPO = sequence-level GRPO, importance ratios and clipping operate on full response likelihood, not individual tokens
  • Key innovation: length-normalized sequence ratio with 1/|y_i| exponent, unifies scale across response lengths
  • vs GRPO: GRPO clips per-token, which is unstable for MoE (expert routing changes cause token-level ratio spikes). GSPO clips per-sequence, which is smooth and stable
  • Practical impact: no Routing Replay needed → simpler training infrastructure for MoE models
  • Know the RL alignment progression: PPO (critic-based) → DPO (no RL) → GRPO (no critic, token-level) → GSPO (no critic, sequence-level)
  • In interviews: cite GSPO when discussing scalable RL for large MoE models, it's the latest advancement (2025) and powers Qwen3