After this lesson, you will be able to: Explain instruction tuning and preference optimization (RLHF and the simpler DPO), and when each stage is worth running.
Fine-tuning on input/output pairs (SFT) teaches a model to follow instructions; preference optimization teaches it which of two answers people prefer. This lesson covers both, with DPO as the practical, RL-free method most students will actually run.
Supervised fine-tuning on (instruction, response) pairs is what turns a base completion model into something that follows directions. The objective is the same next-token cross-entropy, but the data is curated demonstrations of desired behavior, and the loss is usually computed only on the response tokens, not the prompt. The quality and diversity of the instruction data matters more than its size; small, clean instruction sets often beat large noisy ones. This is the first and most important alignment stage and is often enough on its own for research purposes.
RLHF (Reinforcement Learning from Human Feedback) trains a separate reward model on human preference comparisons, then uses reinforcement learning (PPO) to push the model toward higher reward while staying close to the SFT model. It works but is complex, unstable, and compute-heavy. DPO (Direct Preference Optimization) achieves a similar effect with a simple classification-style loss directly on preference pairs (chosen vs rejected), no reward model and no RL loop. For an SRW-scale project, DPO (or its relatives like ORPO, KTO) is the realistic way to do preference tuning; mention RLHF as the conceptual predecessor.
DPO and friends train on triples: a prompt, a preferred response, and a rejected one.
# A preference example (what DPO trains on){"prompt": "Explain why the sky is blue to a 10-year-old.","chosen": "Sunlight is made of many colors. Air scatters the blue ...","rejected": "Rayleigh scattering of electromagnetic radiation with ...",}# DPO's loss increases the model's relative likelihood of `chosen` over# `rejected`, regularized to stay near the reference (SFT) model, with NO# reward model and NO RL loop. Hugging Face TRL provides DPOTrainer.
Jumping to preference optimization before a solid SFT model exists (DPO/RLHF refine behavior; they do not create it). Computing SFT loss over the prompt tokens instead of only the response. Using preference data that conflates style with correctness, so you align the model to confident-sounding wrong answers. Forgetting the KL/regularization to the reference model, letting the model drift and degrade. Claiming 'RLHF' when you ran DPO, or vice versa, which misrepresents the method to reviewers.
Pick the correct answer.
Sign in and purchase access to unlock this lesson.