🧪 Test-Time Compute & Reasoning Models

What is Test-Time Compute Scaling and Why It Matters

Test-time compute scaling is the paradigm of allocating additional computational resources during inference (rather than during training) to improve model performance on hard problems. Traditional scaling laws (Chinchilla, Kaplan et al.) focus on training compute: more parameters, more data, more FLOPs during pretraining yield better models. Test-time compute flips this: a fixed model can produce dramatically better answers by "thinking longer" at inference time, generating intermediate reasoning tokens before committing to a final answer.

The key insight, formalized by Snell et al. (2024) in "Scaling LLM Test-Time Compute Optimally," is that for many problems, spending 10-100x more inference FLOPs is more cost-effective than training a model 10-100x larger. A smaller model with extensive test-time search can match or exceed a much larger model answering immediately. For example, OpenAI's o1-preview (September 2024), a model estimated at GPT-4-class size, outperformed GPT-4 on competition math (83% vs 13% on AIME 2024) and PhD-level science (GPQA Diamond: 78% vs 56%) purely through test-time reasoning.

This represents a fundamental shift in how we think about AI scaling. The "pre-training scaling wall" (diminishing returns from simply making models bigger) has pushed frontier labs toward test-time compute as the next scaling axis. OpenAI's o1/o3, DeepSeek's R1, Google's Gemini 2.0 Flash Thinking, and Alibaba's Qwen QwQ/QwQ-32B all embody this paradigm. The era of "thinking models" has arrived, and understanding how they work is now essential for ML system design.

How Reasoning Models Work

Chain-of-Thought as Internal Monologue

Reasoning models generate an extended chain-of-thought (CoT), sometimes called "thinking tokens," before producing their final answer. Unlike standard CoT prompting (Wei et al., 2022), where the user explicitly asks the model to "think step by step," reasoning models are trained to do this automatically. The thinking process is an internal monologue: the model explores hypotheses, backtracks when it hits dead ends, verifies intermediate results, and synthesizes a final answer.

In OpenAI's o1 family, thinking tokens are hidden from the user (shown only as a summary). In DeepSeek-R1, the full reasoning trace is visible inside <think>...</think> tags. A typical R1 response on a hard math problem might generate 2,000-10,000 thinking tokens before a 200-token final answer. These thinking tokens are the mechanism through which the model "spends" additional compute: each token requires a full forward pass through the model.

Extended Thinking and Thinking Budgets

Modern reasoning models expose a "thinking budget": a configurable maximum number of tokens the model can use for internal reasoning. Anthropic's Claude with extended thinking allows budgets from 1,024 to 128,000 tokens. OpenAI's o1 and o3 have "reasoning effort" settings (low/medium/high) that control how many thinking tokens are generated. Qwen3 (2025) introduces a hybrid approach: the same model can toggle thinking on or off with a /think or /no_think flag, allowing users to choose between fast responses and deep reasoning per query.

The optimal thinking budget depends on problem difficulty. Simple factual questions need zero thinking tokens; a standard model suffices. Medium-difficulty problems (multi-step math, code debugging) benefit from 1,000-5,000 thinking tokens. Competition-level math or complex reasoning tasks may need 10,000-50,000+ tokens. Setting the budget too low wastes the model's reasoning potential; setting it too high wastes compute on easy problems and can even degrade performance through overthinking (the model second-guesses correct initial answers).

DeepSeek-R1 and RLVR: Reinforcement Learning with Verifiable Rewards

Pure RL Without SFT: Emergent Reasoning from Reward Alone

DeepSeek-R1 (January 2025) demonstrated a remarkable finding: reasoning ability can emerge purely from reinforcement learning, without any supervised fine-tuning on human-written chain-of-thought demonstrations. The R1-Zero experiment trained a base language model (DeepSeek-V3) directly with RL using only verifiable reward signals (correct/incorrect on math problems and pass/fail on code tasks) with no human demonstrations of "how to reason."

The result was striking: R1-Zero spontaneously developed extended chain-of-thought reasoning, self-verification ("let me check this"), backtracking ("wait, that's wrong, let me reconsider"), and even reflection on its own reasoning process. These behaviors emerged purely from the reward signal, not from imitating human demonstrations. On AIME 2024, R1-Zero achieved 71% accuracy using majority voting, rivaling OpenAI's o1-0912.

RLVR: The Training Recipe

The RLVR (Reinforcement Learning with Verifiable Rewards) approach used in R1 has two key components. First, the reward is binary and verifiable: for math, the answer is checked against the ground truth (correct = +1, incorrect = -1); for code, the solution is executed against test cases (all pass = +1, any fail = -1). No reward model is needed; the reward is computed directly. Second, the RL algorithm is GRPO (Group Relative Policy Optimization), which samples multiple responses per prompt (typically 16-64), computes group-relative advantages (subtract the group mean reward, divide by group standard deviation), and updates the policy without a separate value network. The GRPO objective is:

J_GRPO(θ) = E_q[1/G Σᵢ min(rᵢ(θ)Âᵢ, clip(rᵢ(θ), 1-ε, 1+ε)Âᵢ) - β·KL(πθ || πref)]

where G is the group size, rᵢ(θ) is the importance ratio, and Âᵢ is the group-normalized advantage.

R1-Zero vs R1: The Role of Distillation

R1-Zero, while impressive, had issues: mixed languages in reasoning, poor readability, and occasional repetitive loops. The full R1 model addresses this with a multi-stage pipeline: (1) cold-start SFT on a small set of long-CoT examples to seed the reasoning format, (2) reasoning-focused RL with verifiable rewards on math/code, (3) rejection sampling to collect high-quality reasoning traces from the RL-trained model, (4) SFT on the combined rejection-sampled reasoning data plus general instruction data, and (5) a final RL stage combining reasoning rewards with general helpfulness/harmlessness rewards. DeepSeek also distilled R1's reasoning into smaller models (1.5B to 70B Qwen/Llama bases), showing that even small models can reason well when trained on R1-generated reasoning traces. Distilled R1-Qwen-32B scores 72.6% on AIME 2024, comparable to o1-mini.

Process Reward Models (PRMs) vs Outcome Reward Models (ORMs)

ORM: Scoring the Final Answer

An Outcome Reward Model (ORM) evaluates only the final answer of a reasoning chain. Given a problem and a complete solution, the ORM outputs a scalar score predicting whether the final answer is correct. ORMs are simple to train (you need only (problem, solution, correct/incorrect) triples) and serve as the baseline for test-time search methods like Best-of-N sampling. However, ORMs provide no signal about where reasoning went wrong in an incorrect solution, limiting their ability to guide fine-grained search.

PRM: Scoring Each Reasoning Step

A Process Reward Model (PRM) assigns a correctness score to each individual step in a reasoning chain. Given a problem and a partial solution up to step k, the PRM predicts whether that step is logically valid and on a productive path toward the correct answer. This step-level supervision enables much more efficient search: you can prune bad reasoning branches early rather than waiting for the complete solution.

Training PRMs requires step-level labels, which are expensive to collect manually. OpenAI's PRM800K dataset (Lightman et al., 2023) contains 800K step-level human labels across 75K math solutions, each step labeled as positive, negative, or neutral. Math-Shepherd (Wang et al., 2024) automates this by generating multiple continuations from each step and labeling the step as correct if any continuation reaches the right answer (a Monte Carlo estimate of step correctness). This automated approach scales PRM training without human annotation.

PRMs Enable Better Search

The power of PRMs lies in their compatibility with tree search at inference time. With an ORM, you can only do Best-of-N: generate N complete solutions and pick the one the ORM scores highest. With a PRM, you can do beam search (expand the top-K partial solutions at each step), MCTS (Monte Carlo Tree Search with PRM as the value function), or step-level reranking. Empirically, PRM + beam search with N=100 candidates significantly outperforms ORM + Best-of-N with N=100 on GSM8K and MATH benchmarks. The PRM acts as a per-step heuristic that focuses compute on the most promising reasoning paths.

Search Strategies at Inference Time

Best-of-N Sampling (Majority Voting)

The simplest test-time compute strategy: generate N independent solutions with temperature > 0, then select the best one. Selection can be by majority voting (self-consistency: pick the most common final answer), ORM scoring (pick the highest-scored solution), or PRM scoring (pick the solution with the highest minimum step score). Self-consistency (Wang et al., 2023) is remarkably effective: on GSM8K, generating 40 samples and majority-voting improves accuracy from 78% (greedy) to 92% with PaLM 540B. The compute cost scales linearly with N, and there are diminishing returns: going from N=1 to N=10 helps a lot; N=100 to N=1000 helps much less.

Beam Search with PRM Scoring

At each reasoning step, maintain the top-K partial solutions (beams). After each step, score all candidates with the PRM, keep the top-K, and expand further. This focuses compute on promising paths and prunes dead ends early. The compute cost is O(K * S) where S is the number of steps, rather than O(N * S) for Best-of-N (where each of the N solutions is generated to completion). In practice, beam search with K=10 and a PRM often outperforms Best-of-N with N=100 at lower total compute, because it avoids wasting tokens on solutions that go wrong early.

Monte Carlo Tree Search (MCTS)

MCTS treats the reasoning process as a tree where each node is a partial solution and each edge is a reasoning step. It uses the UCB (Upper Confidence Bound) formula to balance exploration of new paths vs exploitation of promising ones, with the PRM providing the value estimate at each node. The algorithm repeats: select (follow UCB down the tree), expand (generate a new step), evaluate (PRM score), and backpropagate (update ancestor scores). MCTS is the most compute-intensive but also the most powerful search strategy: it was used in AlphaProof (DeepMind, 2024) to solve IMO-level math problems. However, it requires a strong PRM and is complex to implement efficiently for autoregressive language models.

Self-Consistency

A specific form of Best-of-N where the selection criterion is majority voting on the final answer. The intuition is that correct reasoning paths, while varied in their intermediate steps, converge on the same answer, so the most frequent answer across N samples is likely correct. Self-consistency requires no reward model at all, making it the cheapest search method to implement. It works best when the problem has a single verifiable answer (math, factual questions) and less well for open-ended generation.

Practical Considerations

Latency vs Accuracy Tradeoffs

Thinking models trade latency for accuracy. A standard Claude Sonnet response takes 1-3 seconds; the same query with extended thinking might take 10-60 seconds. For o1-pro on hard math, response times of 1-5 minutes are common. This makes reasoning models unsuitable for latency-sensitive applications like autocomplete, real-time chat, or streaming UIs that need first-token-fast. The design pattern is to use a router: classify incoming queries by difficulty, send easy ones to a fast standard model, and route hard ones to a thinking model with an appropriate budget.

When to Use Thinking Models vs Standard Models

Thinking models shine on problems requiring multi-step reasoning, planning, mathematical proof, complex code generation, and problems where being correct matters more than being fast. They underperform their cost on simple factual retrieval, creative writing, translation, summarization, and other tasks where the first intuition is usually right. A useful heuristic: if a human expert would need to "work through" the problem on paper, a thinking model will likely help; if a human would answer instantly from memory, save the compute.

Cost Implications

The cost of reasoning models is dominated by thinking tokens, which are typically billed at output-token rates. A query that generates 5,000 thinking tokens + 500 output tokens costs roughly 10x more than the same query answered in 500 tokens by a standard model. At scale, this is significant: a service handling 1M queries/day at an average of 3,000 thinking tokens per query generates 3B additional tokens/day. Strategies to manage cost include difficulty-based routing (only use thinking for hard queries), thinking budget caps, caching reasoning traces for repeated similar queries, and using distilled smaller reasoning models (DeepSeek-R1-Distill, Qwen QwQ-32B) instead of frontier models for most queries.

Hybrid Models: Toggle Thinking On/Off

Qwen3 (2025) pioneered a practical architecture: a single model that can operate in both "thinking" and "non-thinking" modes. Users toggle with a system prompt flag, and the model either generates <think>...</think> tokens before answering or responds directly. This eliminates the need for a separate router model and simplifies deployment: one model serves both fast and deep queries. The training recipe combines SFT on both thinking and non-thinking data, followed by RL that rewards both modes appropriately. This hybrid approach is likely the future: every model will have a "think harder" dial rather than reasoning being a separate model family.