← All papers

Self-Consistency Improves Chain of Thought Reasoning in Language Models

Wang, Wei, Schuurmans et al. (Google) · 2023 · ICLR 2023

PromptingRead on arXiv

Improves chain-of-thought prompting by sampling multiple reasoning paths and taking a majority vote over final answers, consistently outperforming single-path greedy decoding.

Key Idea

Instead of generating a single chain-of-thought and trusting that answer, Self-Consistency samples multiple diverse reasoning paths (using temperature > 0) and selects the most frequent final answer via majority voting. The intuition: correct reasoning paths are more likely to converge on the same answer, while incorrect paths tend to diverge.

How It Works

  1. Prompt the model with chain-of-thought exemplars (same as standard CoT)
  2. Sample N completions with temperature (e.g., T=0.7), generating N different reasoning chains
  3. Extract the final answer from each chain
  4. Majority vote: the answer appearing most frequently across the N samples is selected
  • No additional training or fine-tuning required, purely an inference-time technique
  • Typically N=5-40 samples; diminishing returns beyond ~20

Why It Matters

  • Consistent improvements: +5-18% accuracy over greedy CoT on arithmetic (GSM8K: 78%), commonsense (ARC), and symbolic reasoning benchmarks
  • Simple and universal: works with any LLM that supports CoT, no model changes needed
  • Pioneered the idea of test-time compute scaling, spending more inference compute for better answers, a precursor to o1-style reasoning
  • Showed that LLMs encode multiple valid reasoning strategies and sampling reveals this diversity
  • Became a standard technique in production LLM pipelines for high-stakes decisions

Key Takeaways for Interviews

  • Self-Consistency = sample N chain-of-thought paths + majority vote on final answer
  • It's a free lunch in accuracy at the cost of N× inference compute, classic latency/quality tradeoff
  • Works because correct reasoning paths converge while errors are diverse (random)
  • Key hyperparameters: temperature (0.5-0.7), number of samples (10-20 is usually enough)
  • In system design: use self-consistency for high-value decisions (medical, financial) where latency is acceptable; skip for real-time chat
  • Precursor to more sophisticated test-time compute methods: Tree of Thoughts, Best-of-N, o1 reasoning