After this lesson, you will be able to: Design a human evaluation (rubric, annotation protocol, inter-annotator agreement) and use LLM-as-judge correctly, knowing its biases.
When automatic metrics fail (open-ended generation, dialogue, reasoning quality), you need human judgment or a model judge. This lesson covers how to run both so the results are trustworthy, not anecdotal.
A credible human eval needs: a clear rubric (define each dimension, e.g. fluency, faithfulness, helpfulness, on a fixed scale with anchored examples), multiple annotators per item, randomized and blinded presentation (annotators must not know which system produced which output), and a reported measure of agreement. Pairwise comparison (which of A/B is better) is often more reliable than absolute scoring. Document the annotator pool and instructions; reviewers will ask who judged and how.
If two annotators disagree wildly, the metric is meaningless, so you must report agreement. Cohen's kappa (two annotators) and Fleiss' kappa (more than two) measure agreement beyond chance; Krippendorff's alpha handles missing data and various scales. Rule of thumb: kappa above ~0.6 is reasonable, above ~0.8 is strong, near 0 means annotators are guessing and your rubric needs work. Low agreement is a signal to fix the rubric, not to hide the number.
Reporting human eval without inter-annotator agreement (so no one knows if the scores mean anything). Not blinding annotators to which system produced each output. Using LLM-as-judge without checking position/verbosity bias or validating against humans. Letting the same model be both a contestant and the judge (self-preference). Evaluating on too few items to detect a real difference. Changing the rubric after seeing results. Reporting a judge score as if it were human-validated when it never was.
Pick the correct answer.
Sign in and purchase access to unlock this lesson.