← All papers

Measuring Massive Multitask Language Understanding

Hendrycks, Burns, Basart et al. · 2021 · ICLR 2021

EvaluationRead on arXiv

Introduced MMLU, a 57-subject multiple-choice benchmark that became the de facto yardstick for comparing frontier LLMs across STEM, humanities, social sciences, and professional exams.

Key Idea

MMLU (Massive Multitask Language Understanding) measures the breadth of a model's world knowledge and reasoning by testing it on 57 subjects spanning elementary math, US history, computer science, law, medicine, ethics, and more. All questions are 4-way multiple choice, scored by accuracy, and evaluated in a few-shot (typically 5-shot) setting.

Dataset Construction

  • ~15,900 questions total, hand-collected from real sources: practice exams (GRE, USMLE, bar exam, AP), textbooks, and online study material
  • Split into dev (5/subject), validation (~1.5K), and test (~14K)
  • Subjects grouped into four super-categories: STEM, Humanities, Social Sciences, Other (professional)
  • Designed so that random guessing yields 25% and a knowledgeable human expert reaches ~90%

Headline Results

  • At release (2020): GPT-3 175B scored ~43.9%, far below human experts
  • GPT-4 (2023): 86.4%, the result that effectively "saturated" the benchmark in the public eye
  • Claude, Gemini, and Llama-3-class models now routinely score in the 80 to 88% range
  • Smaller open models (7B to 13B) typically land in the 45 to 65% range, making MMLU a useful discriminator at the low end too

Limitations & Successors

  • Contamination risk: MMLU questions appear in scraped web data, inflating scores for models trained on recent crawls
  • Multiple-choice format tests recognition, not generation or calculation, models can pattern-match to (A)/(B)/(C)/(D)
  • Label noise and ambiguous questions have been documented (some questions have wrong "correct" answers)
  • Spawned successors: MMLU-Pro (10-way choices, harder distractors), MMLU-Redux (cleaned), GPQA (graduate-level science), BIG-Bench Hard

Why It Matters

  • Became the single most-cited number in LLM release announcements for several years
  • Forced labs to track broad knowledge coverage rather than narrow task accuracy
  • Made cross-model comparison tractable with one headline metric, at the cost of overfitting incentives

Key Takeaways for Interviews

  • 57 subjects, ~15K multiple-choice questions, 5-shot eval is the standard protocol
  • Human-expert ceiling ~90%, random baseline 25%, useful anchors when interpreting a score
  • Contamination is the dominant concern for any benchmark this widely scraped; always check whether a reported MMLU number used a decontaminated split
  • Multiple-choice ≠ real capability, pair MMLU with generation benchmarks (HumanEval, MT-Bench) and reasoning benchmarks (GPQA, MATH) for a fuller picture
  • Know the successors (MMLU-Pro, GPQA), interviewers like to ask "what would you use instead today?"