After this lesson, you will be able to: Compare full fine-tuning with parameter-efficient fine-tuning (PEFT), and do the memory math that decides what you can train on a given GPU.
Whether you can fine-tune at all comes down to GPU memory. This lesson covers what full fine-tuning costs, why PEFT exists, and how to estimate memory before you waste an hour hitting an out-of-memory error.
Full fine-tuning updates every weight. The memory cost is not just the weights: you also store gradients (same size as the weights) and optimizer state. Adam keeps two extra values per parameter (momentum and variance), so in fp32 the rule of thumb is roughly 16 bytes per parameter (4 weights + 4 gradient + 8 optimizer), plus activations. A 7B-parameter model in full fine-tuning needs well over 100GB, far beyond a single consumer or free-tier GPU. That is the problem PEFT solves.
PEFT freezes the pretrained weights and trains a small number of new parameters instead. Because the frozen weights need no gradients or optimizer state, memory drops dramatically. The dominant PEFT method is LoRA (next lesson), but the family also includes prefix/prompt tuning and adapters. The research payoff beyond memory: you can keep many small task-specific adapters for one base model, and PEFT often matches full fine-tuning quality on narrow tasks, which makes it the default for academic-budget experiments.
A back-of-the-envelope check that saves hours.
# Rough training memory (no PEFT), per parameter:# fp32 full fine-tune ~16 bytes (weights 4 + grad 4 + Adam 8)# mixed precision ~ less for weights but optimizer often still fp32# So a 7B model full fine-tune in fp32:# 7e9 params * 16 bytes ~= 112 GB (+ activations) -> not on one GPU## LoRA on the same 7B model:# base weights frozen (load in 4-bit ~= 3.5 GB for QLoRA)# only the adapter params get gradients + optimizer state (often <1% of params)# -> fits on a single 16 GB (sometimes 8 GB) GPU.## Always estimate first: params * bytes/param + activation overhead.
Assuming inference memory equals training memory (training is several times larger because of gradients + optimizer state + activations). Forgetting optimizer state when budgeting (Adam doubles it). Reporting 'we fine-tuned a 7B model' without saying full vs LoRA vs QLoRA, which changes both cost and how others reproduce you. Running out of memory mid-epoch because activation memory scales with batch size and sequence length, not just parameters. Comparing your PEFT result to a full-fine-tune baseline you never actually ran.
Pick the precise reason.
Sign in and purchase access to unlock this lesson.