← All papers

Learning to Reason with LLMs

OpenAI · 2024 · OpenAI Blog / System Card 2024

ReasoningRead on arXiv

Introduced o1, a model trained with reinforcement learning to perform extended chain-of-thought reasoning before answering, achieving PhD-level performance on math, science, and coding benchmarks.

Key Idea

OpenAI o1 is trained to think before answering, it produces a long internal chain-of-thought (reasoning trace) before generating its final response. Unlike standard CoT prompting, o1's reasoning ability is trained into the model via reinforcement learning, allowing it to learn strategies like backtracking, error-checking, and trying alternative approaches within a single generation.

How It Works

  • Training: Large-scale RL (details undisclosed) teaches the model to generate productive reasoning chains that lead to correct answers. The model learns when to think longer on harder problems.
  • Inference: The model generates a hidden "thinking" trace (not shown to users by default), then produces the final answer. More thinking tokens = better answers on hard problems.
  • Test-time compute scaling: Unlike traditional scaling (more parameters, more training data), o1 scales by spending more inference compute, thinking longer on harder problems yields better performance.
  • Chain-of-thought is learned: not prompted, the model decides how and when to reason, unlike few-shot CoT.

Why It Matters

  • GPQA Diamond: 78% (PhD-level science), surpassing human PhD experts for the first time
  • AIME 2024: 83% on competition math (vs ~13% for GPT-4o)
  • SWE-bench Verified: 41% (coding agents)
  • Codeforces: 89th percentile in competitive programming
  • Established test-time compute scaling as a new axis, complementary to parameter scaling and data scaling
  • Spawned a wave of reasoning models: o3, DeepSeek-R1, Claude with extended thinking, Gemini 2.0 Flash Thinking
  • Changed interview expectations: candidates now asked about test-time compute tradeoffs

Key Takeaways for Interviews

  • o1 = RL-trained chain-of-thought reasoning, not just prompted CoT
  • Test-time compute scaling: spending more tokens thinking improves accuracy, especially on hard problems, a new scaling law
  • Tradeoff: much higher latency and cost (10-100× more tokens) vs standard models, use for hard problems, not simple queries
  • The reasoning trace is a form of process reward, the model is rewarded for good reasoning steps, not just correct final answers
  • In system design: route easy queries to fast models, hard queries to reasoning models (cost-aware routing)
  • Know the progression: CoT prompting → Self-Consistency → Tree of Thoughts → o1 (RL-trained reasoning)