Statistical Rigor

After this lesson, you will be able to: Apply statistical rigor: significance testing, confidence intervals (bootstrap), multiple seeds, and ablations, so a reported improvement is real and attributable.

A number without uncertainty is not evidence. This lesson covers the statistics reviewers expect: is the gain bigger than the noise, and what actually caused it.

Prerequisites:Human Evaluation and LLM-as-Judge

Variance is real: report it

Run the same fine-tune with three seeds and the metric will move, often by more than the 'improvement' you are claiming. So a single number is not a result. Run multiple seeds (3-5 minimum), report the mean and the standard deviation (or a confidence interval), and show error bars on plots. If your method's mean plus-or-minus its spread overlaps the baseline's, you have not shown an improvement, you have shown noise. This one practice prevents most over-claiming.

Significance testing and confidence intervals

To claim system A beats system B, test whether the difference is statistically significant, not just numerically larger. Paired tests (the systems are evaluated on the same examples) are standard: a paired bootstrap or a paired permutation test makes few assumptions and suits NLP metrics well. Bootstrap also gives confidence intervals: resample the test set many times, recompute the metric, and report the 95% interval. If you run many comparisons, correct for multiple testing (e.g. Bonferroni) or you will find 'significant' results by chance.

A paired bootstrap significance test

Resample the test set to estimate whether A really beats B.

python

import numpy as np

def paired_bootstrap(scores_a, scores_b, n=10000, seed=0):
    # scores_a, scores_b: per-example metric for the SAME test items
    rng = np.random.default_rng(seed)
    a, b = np.array(scores_a), np.array(scores_b)
    diffs = a - b
    obs = diffs.mean()
    # how often does a resample reverse the sign of the observed difference?
    boot = np.array([
        rng.choice(diffs, size=len(diffs), replace=True).mean()
        for _ in range(n)
    ])
    p = (boot <= 0).mean() if obs > 0 else (boot >= 0).mean()
    ci = np.percentile(boot, [2.5, 97.5])
    return obs, p, ci   # mean diff, p-value, 95% CI
# Report: 'A beats B by X (95% CI [..], p < 0.05, paired bootstrap, 3 seeds).'

Common mistakes only experienced researchers catch

Reporting one seed as a result. Claiming an improvement that is within the seed-to-seed variance. Using an unpaired test when the systems were evaluated on the same examples (throwing away power). Running dozens of comparisons and reporting the significant ones without multiple-comparison correction (p-hacking). No ablations, so the cause of the gain is unknown. Confusing statistical significance with practical importance (a tiny but significant gain may not matter).

Quick Check

Your method beats the baseline by 0.4 points on one seed. What's missing before you can claim it's better?

Pick the most important missing piece.

←Human Evaluation and LLM-as-Judge

Back to Evaluation and Experimental Rigor

Reproducibility→