📊 Evaluation & Benchmarking

Why Evaluation Matters

Evaluation is how you know whether a model is good. It's also how you decide which model to ship, which research direction to pursue, and where the field is heading. Goodhart's Law ("when a measure becomes a target, it ceases to be a good measure") looms large: once a benchmark drives leaderboards and funding, labs optimize directly for it, and the score stops reflecting the underlying capability it was meant to measure. The history of ML is largely a history of benchmarks (ImageNet, GLUE, SuperGLUE, MMLU) driving progress, then being saturated and replaced.

Traditional ML Metrics

For classification: accuracy (overall correctness), precision (of predicted positives, how many are correct), recall (of true positives, how many we caught), and F1 (harmonic mean of precision and recall). For ranking and probabilistic outputs: AUC-ROC (probability the model ranks a random positive above a random negative) and log loss. For probabilistic forecasts: calibration (when the model says 70%, does it happen 70% of the time?), measured via Expected Calibration Error (ECE) or reliability diagrams.

Metric choice matters enormously when classes are imbalanced or when false positives and false negatives have asymmetric costs (fraud, medical diagnosis, content moderation).

LLM-Specific Benchmarks

• MMLU (Massive Multitask Language Understanding): 57 subjects, multiple choice, tests knowledge breadth. Largely saturated by 2024 (top models > 90%).
• HellaSwag: commonsense sentence completion. Saturated.
• HumanEval / MBPP: Python coding from docstrings. Pass@1 is the standard metric.
• GSM8K: grade-school math word problems. Tests step-by-step reasoning.
• ARC (AI2 Reasoning Challenge): grade-school science questions, harder split.
• MATH: competition-level math (AMC/AIME style).
• GPQA ("Google-Proof Q&A"): PhD-level science questions specifically designed to resist web search. The current frontier knowledge benchmark.
• SWE-bench / SWE-bench Verified: resolve real GitHub issues in large Python repos. Closest thing to "agentic coding" eval; the leaderboard moved from <5% in 2024 to >70% by 2026.
• FrontierMath: research-grade math problems written by professional mathematicians; the hardest current math benchmark, with top models still under 30%.

Benchmark Suites and Leaderboards

• HELM (Holistic Evaluation of Language Models, Stanford): standardized, multi-metric evaluation across many scenarios.
• Open LLM Leaderboard (Hugging Face): reproducible academic benchmark aggregation for open models.
• Chatbot Arena (LMSYS): crowdsourced pairwise human preference voting. Uses an Elo rating system: each vote updates competing models' ratings, producing a live ranking based on real user preferences rather than static test sets. Widely considered the most resistant to gaming because the prompts are unseen.

Eval Harnesses

The leaderboards above all sit on top of an eval harness: the engine that loads a model, runs it across benchmarks, scores the outputs, and produces a comparable result. lm-evaluation-harness (EleutherAI) is the de facto standard: it defines a benchmark taxonomy, handles tokenization quirks, manages few-shot prompting, and ships hundreds of benchmark implementations. The HF Open LLM Leaderboard runs lm-eval-harness under the hood, which is why its scores are reproducible across labs. Inspect AI (UK AISI) is the agent-aware successor, built for evaluating multi-step tool-using agents (not just single-turn completions), with first-class support for sandboxing, judge models, and dangerous-capability evals. OpenAI Evals and Anthropic's internal eval tooling fill similar roles inside frontier labs. The harness layer is itself a piece of engineering with its own design space (judge isolation, contamination controls, reproducibility guarantees); see Harness Engineering for the broader framing.

LLM-as-Judge

Using a strong model (GPT-4, Claude, Gemini) as an evaluator scales human-style judgment to thousands of examples cheaply. Two common patterns:

• Pairwise comparison: judge picks which of two responses is better (matches Chatbot Arena format). Reduces absolute-score bias.
• Rubric-based grading: judge scores a single response on defined criteria (correctness, helpfulness, safety).

Known biases: position bias (judges prefer the first response), length bias (judges prefer longer answers), self-preference bias (a model judges its own family's outputs more favorably). Mitigations: swap positions and average, normalize for length, use a different judge family than the candidate.

Benchmark Contamination

When test data leaks into training data, the model "remembers" answers rather than reasoning to them, and reported scores wildly overstate real capability. Sources: public benchmarks scraped during pretraining, paraphrased restatements in tutorials, even canary strings not being filtered. Detection: n-gram overlap between training and test sets, perplexity tests (a contaminated model has unusually low perplexity on test answers), and canary string checks (BIG-bench includes a deliberate UUID to test whether training data filtered it). Newer benchmarks like GPQA, SWE-bench Verified, and FrontierMath were designed specifically to be contamination-resistant: kept private, regenerated, or sourced from material the labs are unlikely to have scraped.

Red-Teaming and Safety Evaluation

Capability benchmarks don't measure harm. Safety evaluation involves:

• Jailbreak resistance: does the model produce harmful content when adversarially prompted? Benchmarks: HarmBench, AdvBench, JailbreakBench.
• Toxicity: RealToxicityPrompts measures unprompted toxic completions.
• Bias: BBQ (Bias Benchmark for QA), StereoSet, CrowS-Pairs test demographic stereotyping.
• Red-teaming: humans (or automated attackers) attempt to elicit policy violations. Usually performed pre-launch by dedicated safety teams.

RAG Evaluation

RAG systems need split evaluation because failures can come from retrieval or generation:

• Retrieval: hit rate, MRR (Mean Reciprocal Rank), NDCG, recall@k.
• Generation: faithfulness (is the answer supported by retrieved context?), answer relevance, context precision/recall.
• RAGAS is the most-used framework; it uses LLM-as-judge to compute faithfulness and relevance without requiring ground-truth answers. Alternatives: TruLens, DeepEval, Arize Phoenix.

Offline vs Online Evaluation

Offline (held-out test sets, benchmarks) is cheap, repeatable, and unblocks development, but proxy metrics often diverge from real user value. Online A/B testing routes a fraction of production traffic to the new model and measures business metrics: engagement, retention, task completion rate, user-reported satisfaction. A/B tests are the ground truth for "does this ship?" but they're slow, expensive, and require traffic.

Common stack: offline benchmark to gate the candidate, shadow deployment to compare on real traffic without user impact, then A/B test on a small slice before full rollout.

Custom Evaluation

Off-the-shelf benchmarks rarely capture what your product needs. Building a custom eval suite means: (1) collecting a representative set of real user inputs (or a synthetic equivalent), (2) defining task-specific success criteria (often graded by LLM-as-judge or human raters), (3) holding the eval set out of any training/fine-tuning loops, and (4) tracking the metric on every model change as a regression test. Strong product teams treat the eval suite as a first-class artifact: it is the contract that defines what "good" means for the product.