After this lesson, you will be able to: Build a reusable evaluation harness for a task that computes the right metrics with confidence intervals and significance tests, and apply it to compare two systems.
The Evaluation capstone: a harness you can point at any two systems and get a trustworthy verdict. This is the tool that makes every later claim in your research defensible, and reviewers love a paper whose evaluation is clearly rigorous.
Pick a task and build an evaluation harness that: loads a (decontaminated, documented) test set, runs one or more systems with a fixed prompt/template, computes appropriate metrics (an overlap metric plus an embedding or judge metric where relevant), reports per-example scores, and produces mean plus-or-minus variance across seeds with a paired significance test and confidence intervals between systems. Make it config-driven and seeded so a run is one command and fully reproducible.
A clean, reusable tool, not a one-off notebook.
Milestone 1: Choose the task + metrics; justify why those metrics fit (and their limits). Document the test set + contamination check.
Milestone 2: System runner: given a model + prompt template, produce predictions for the test set (fixed, logged).
Milestone 3: Metric layer: compute per-example scores for each metric via the evaluate library; store them.
Milestone 4: Statistics layer: mean + std across seeds, paired bootstrap p-value + 95% CI between two systems.
Milestone 5: (If applicable) an LLM-as-judge mode with order-swapping + a small human-validation subset.
Milestone 6: A report it generates automatically (table with means, CIs, significance) + a README and pinned environment so anyone can rerun it.
'I built an evaluation harness that runs systems on a decontaminated test set, computes overlap and embedding metrics plus an order-swapped LLM judge, and reports significance with a paired bootstrap across seeds.' That tells an advisor you understand the part of research that actually gates publication. Use it on your own fine-tuning project to produce the headline comparison, and reuse it for the reproduction in the Publishing sub-track.
Sign in and purchase access to unlock this lesson.