Compute Realities for Fine-Tuning

After this lesson, you will be able to: Run fine-tuning jobs within a real compute budget: free/cheap GPUs, mixed precision, gradient checkpointing and accumulation, and tracking experiments.

Most NLP research now happens on a tight compute budget. This lesson is the practical layer: where to get GPUs, the memory tricks that let a job fit, and how to track experiments so results are reproducible.

Prerequisites:Data Curation for Fine-Tuning

Where students actually get GPUs

Free: Google Colab (T4, time-limited) and Kaggle Notebooks (T4 x2 or P100, 30 hours/week) are enough for LoRA on small models. Cheap on-demand: RunPod, Lambda, Vast.ai rent a single A100/H100 by the hour for a few dollars, which is how a lot of SRW-scale work gets done. University clusters if you have access. The research skill is fitting the experiment to the budget, not assuming a lab's worth of GPUs.

The memory tricks that make a job fit

Mixed precision (bf16/fp16) halves activation/weight memory and speeds up on modern GPUs. Gradient checkpointing trades compute for memory by recomputing activations in the backward pass instead of storing them, often the difference between fitting and not. Gradient accumulation simulates a large batch by summing gradients over several small batches before stepping, so you get a stable effective batch size on a small GPU. Combine these with QLoRA and a 7B fine-tune fits on a single 16GB card. Report the effective batch size (per-device batch x accumulation x devices), not just the per-device batch.

These knobs in the Trainer

The arguments that turn an out-of-memory job into a running one.

python

from transformers import TrainingArguments

args = TrainingArguments(
    output_dir="out",
    per_device_train_batch_size=1,
    gradient_accumulation_steps=16,   # effective batch = 1 * 16 = 16
    gradient_checkpointing=True,      # recompute activations to save memory
    bf16=True,                        # mixed precision (use fp16 on older GPUs)
    learning_rate=2e-4,               # higher LR is normal for LoRA
    num_train_epochs=3,
    logging_steps=10,
    report_to="wandb",               # track loss/metrics for reproducibility
    seed=42,                          # set + report the seed
)

Common mistakes only experienced researchers catch

Reporting per-device batch size while hiding that gradient accumulation made the effective batch much larger (or smaller). Not setting a seed, so the result cannot be reproduced. Forgetting gradient checkpointing and blaming the model for OOM. Running one seed and reporting the number as if it were stable (variance across seeds is real, see the Evaluation sub-track). Losing the config that produced a good checkpoint, making it irreproducible even to yourself.

Quick Check

Your GPU OOMs at batch size 8 but you need a larger effective batch. What's the fix that keeps memory low?

Pick the right technique.

←Data Curation for Fine-Tuning

Back to Fine-Tuning and Adaptation

Passion Project: Fine-Tune a Model→