After this lesson, you will be able to: Use standard NLP benchmarks responsibly and detect/avoid data contamination, the issue that silently invalidates LLM evaluations.
Benchmarks let the field compare systems, but in the LLM era they leak into pretraining data, inflating scores. This lesson covers the major benchmarks and how to keep your evaluation honest.
GLUE and SuperGLUE (language understanding) were the pre-LLM standards. MMLU (multitask knowledge), HellaSwag (commonsense), GSM8K (grade-school math), HumanEval (code), and BIG-bench stress different capabilities. HELM and the Open LLM Leaderboard aggregate many at once. For generation, task-specific sets (e.g. CNN/DailyMail for summarization, WMT for translation) dominate. Knowing which benchmark targets which capability is basic literacy: reporting MMLU for a summarization model is a category error a reviewer will flag.
For a research contribution you usually need an evaluation the model could not have memorized: a freshly collected set, a held-out split you control, an adversarial/perturbed version of an existing set, or a private test set. Document provenance, size, license, and any annotation process. A smaller, clean, well-documented eval set beats a large contaminated one. If you introduce a new benchmark, that itself can be the paper, with contamination resistance as a selling point.
Reporting benchmark numbers without checking the model's training cutoff against the benchmark's release date. Comparing your model to others' reported numbers obtained with different prompts/few-shot settings (the eval harness and prompt matter enormously). Using a benchmark outside the capability it tests. Treating a public leaderboard score as ground truth despite contamination risk. Failing to fix the evaluation prompt/format across models, so you are comparing prompts, not models.
Pick the most likely culprit.
Sign in and purchase access to unlock this lesson.