After this lesson, you will be able to: Use and critique the standard automatic metrics: perplexity, BLEU, ROUGE, METEOR, and BERTScore, knowing what each measures and where each fails.
Automatic metrics are cheap and reproducible, which is why every paper reports them, and limited, which is why every reviewer distrusts them. This lesson covers the main ones and their failure modes so you report them correctly.
Perplexity is exp(cross-entropy loss): roughly, how surprised the model is by held-out text, or the effective number of equally-likely choices per token. Lower is better. It is the natural metric for language modeling, but it only compares within the same tokenizer and test set (different tokenizers make it incomparable), and a lower perplexity does not guarantee better task performance. Report it for LM quality, never as a stand-in for downstream usefulness.
These compare a generated text to one or more references by n-gram overlap. BLEU (precision-oriented, with a brevity penalty) is the historic machine-translation metric. ROUGE (recall-oriented, ROUGE-N and ROUGE-L) is the summarization standard. METEOR adds stemming/synonym matching and correlates a bit better with human judgment. All share a fatal weakness: they reward surface word overlap, so a correct paraphrase with different words scores low and a fluent-but-wrong output sharing words scores high. Use them because the field expects them, but never alone.
The `evaluate` library gives you the standard, comparable implementations.
import evaluatebleu = evaluate.load("bleu")rouge = evaluate.load("rouge")bertscore = evaluate.load("bertscore")preds = ["the cat sat on the mat"]refs = [["a cat was sitting on the mat"]]print(bleu.compute(predictions=preds, references=refs))print(rouge.compute(predictions=preds, references=[r[0] for r in refs]))# BERTScore compares contextual embeddings, not surface words:print(bertscore.compute(predictions=preds, references=[r[0] for r in refs], lang="en"))# Always report which implementation + config (e.g. sacreBLEU) for comparability.
BERTScore compares the contextual embeddings of candidate and reference tokens, so it credits paraphrases that overlap metrics miss. It correlates better with humans on many tasks, but depends on which embedding model you use (report it), is slower, and still cannot judge factual correctness. For LLM outputs specifically, the field increasingly uses model-based judges (next lesson) precisely because n-gram metrics break down on long, open-ended generations.
Reporting BLEU without specifying the implementation (the infamous reason BLEU scores were not comparable across papers until sacreBLEU). Comparing perplexity across different tokenizers. Using a single reference when multiple valid outputs exist. Treating a 0.5-point ROUGE gain as meaningful without significance testing. Reporting only an automatic metric for a task (e.g. dialogue) where it is known to correlate poorly with quality. Choosing the metric that makes your method look best after seeing the results (metric-hacking).
Pick the correct reason.
Sign in and purchase access to unlock this lesson.