← All papers

Qwen2.5: A Party of Foundation Models

Qwen Team, Alibaba · 2025 · arXiv 2025

ArchitectureRead on arXiv

Presents the Qwen2.5 family of foundation models spanning language (0.5B-72B dense, plus Qwen2.5-Turbo at 100B+ MoE), code, math, vision, and specialized variants. Achieves GPT-4o-level performance on many benchmarks with open weights, trained on 18 trillion tokens with extensive post-training.

Key Idea

Qwen2.5 represents the state of the art in open-weight foundation models as of early 2025, closing the gap with proprietary models like GPT-4o across reasoning, coding, math, and multilingual tasks. The key advances are in data quality and scale (18T tokens, up from 7T for Qwen2), long context (128K native, 1M with YaRN), and a comprehensive post-training pipeline (SFT + multi-stage RLHF with both online and offline methods).

Architecture & Training

  • Architecture: Dense transformer with GQA, SwiGLU, RoPE. Sizes from 0.5B to 72B parameters. Qwen2.5-Turbo uses a MoE variant (similar to DeepSeek-V2's shared + routed expert design).
  • Pre-training data: 18 trillion tokens with heavy emphasis on synthetic data, math problems generated and verified by Qwen2-Math, code snippets extracted and validated, multilingual translations. Quality filtering via model-based scoring.
  • Post-training: Multi-stage RLHF pipeline: (1) SFT on high-quality instruction data, (2) DPO for initial alignment, (3) Online RLHF with reward model, (4) Final DPO/GRPO refinement. Separate reward models for different capabilities (helpfulness, safety, code, math).
  • Long context: Pre-trained with 32K, extended to 128K via YaRN, with 1M context demonstrated via dynamic NTK-aware RoPE scaling. NIAH (Needle in a Haystack) perfect recall up to 128K.

Why It Matters

  • Open-weight GPT-4o competitor: Qwen2.5-72B-Instruct matches GPT-4o on MMLU-Pro, HumanEval, MATH, and many others, making it the strongest open model at its release
  • Size range: From 0.5B (runs on phones) to 72B (server-grade), all from the same training pipeline, enabling study of scaling behavior
  • Specialized variants: Qwen2.5-Coder-32B is the strongest open code model; Qwen2.5-Math solves competition-level math; QwQ-32B adds test-time reasoning
  • Ecosystem: Served as the base model for DeepSeek-R1 distillation, demonstrating its strength as a foundation for reasoning research
  • Practical: Widely deployed in production due to permissive Apache 2.0 license, strong multilingual support (29 languages), and comprehensive tooling

Key Takeaways for Interviews

  • 18T tokens: data scaling + quality filtering (model-based scoring, synthetic data generation) is as important as model scaling
  • Post-training pipeline: SFT → DPO → online RLHF → GRPO/DPO, multi-stage iterative refinement with specialized reward models
  • Synthetic data: Qwen2.5 heavily uses model-generated math, code, and instruction data (validated and filtered) as a key training ingredient
  • MoE variant (Turbo): shared expert + routed expert design for cost-efficient serving of the largest model
  • Long context: YaRN + dynamic NTK-aware RoPE for 128K-1M context without quality degradation
  • In system design: cite as the reference open-weight model family for production LLM deployments, and for understanding the full pre-train → post-train → specialize pipeline