Implement three evaluation metrics (perplexity, accuracy, LLM-as-judge) and combine them in a unified eval suite that returns structured results.
Your team needs to evaluate a language model before deploying it. You have a MockLLM that simulates generation and log probabilities. Your job is to build the evaluation pipeline: compute perplexity over reference texts, measure exact-match accuracy on Q&A pairs, implement LLM-as-judge scoring for open-ended responses, and wire everything into a single run_eval_suite function that returns structured results. The MockLLM and data structures are provided. You only need to implement the four metric functions.
Ask me about the code, bugs, or concepts.
I'll guide you in plain English, no code output.
Budget: 50K tokens per lab