← Coding labs/

Build an LLM Evaluation Pipeline

Implementationmedium~35 min
Objective

Implement three evaluation metrics (perplexity, accuracy, LLM-as-judge) and combine them in a unified eval suite that returns structured results.

Background

Your team needs to evaluate a language model before deploying it. You have a MockLLM that simulates generation and log probabilities. Your job is to build the evaluation pipeline: compute perplexity over reference texts, measure exact-match accuracy on Q&A pairs, implement LLM-as-judge scoring for open-ended responses, and wire everything into a single run_eval_suite function that returns structured results. The MockLLM and data structures are provided. You only need to implement the four metric functions.

Requirements
  1. 1.Compute perplexity from log probabilities: PPL = exp(-average_log_prob)
  2. 2.Compute exact-match accuracy by comparing generated answers to ground truth
  3. 3.Implement LLM-as-judge that constructs a rating prompt and parses a 1-5 score
  4. 4.Wire all metrics into run_eval_suite and return a structured results dict
  5. 5.Handle edge cases: empty inputs, parse failures in judge scores
Evaluation (100 points)
Perplexity computation
Uses math.exp and model.get_logprobs with proper averaging
25pt
Accuracy computation
Calls model.generate and compares with expected answers
20pt
LLM-as-judge scoring
Constructs a rating prompt and calls judge_model.generate
20pt
Eval suite orchestration
Calls all three metrics in run_eval_suite
20pt
Structured results returned
Returns a dict with metric results
15pt
Hints
Select a file to start editing
Terminal
$
AI Assistant50K tokens left

Ask me about the code, bugs, or concepts.
I'll guide you in plain English, no code output.
Budget: 50K tokens per lab