Pretraining Objectives

After this lesson, you will be able to: Compare the pretraining objectives (causal LM, masked LM, seq2seq denoising) and explain the loss, the data, and what each objective is good for.

Architecture is half the story; the training objective is the other half. This lesson covers the three objectives behind GPT, BERT, and T5, and the cross-entropy loss they all minimize.

Prerequisites:Multi-Head Attention and the Transformer Block

Causal language modeling (GPT)

Predict the next token given all previous tokens. The model outputs a probability distribution over the vocabulary at each position, and the loss is the cross-entropy between that distribution and the actual next token, averaged over the sequence. Because the target at position i is just the input at position i+1, the data is unlabeled text, which is why this scales: the entire internet is training data. This objective produces generators and is what almost every modern LLM uses.

Masked language modeling (BERT) and denoising (T5)

Masked LM (BERT) hides ~15% of tokens and trains the model to predict them from both left and right context; it is great for producing representations for classification/extraction but cannot generate fluently. T5 reframes everything as text-to-text with a span-corruption denoising objective: mask spans, predict them in order. The trend since GPT-3 has been toward causal LM because it unifies generation and (via prompting) most other tasks, but masked models are still strong and efficient for pure understanding tasks, and the choice is a real experimental decision.

The cross-entropy loss, concretely

Every objective above minimizes cross-entropy between predicted and true tokens.

python

import torch, torch.nn.functional as F

# logits: (batch, seq, vocab) from the model; targets: (batch, seq) of token ids
# For causal LM, targets are the inputs shifted left by one.
def lm_loss(logits, targets):
    B, T, V = logits.shape
    return F.cross_entropy(
        logits.view(B * T, V),   # flatten positions
        targets.view(B * T),
        ignore_index=-100,       # skip padding / non-target positions
    )

# Perplexity, the standard LM metric, is just exp(cross_entropy_loss):
#   perplexity = torch.exp(loss)
# Lower is better; it measures how 'surprised' the model is by the real text.

Common mistakes only experienced researchers catch

Reporting raw loss across models with different tokenizers as if comparable (perplexity is only comparable within the same tokenization). Forgetting to shift targets for causal LM (off-by-one that silently trains the model to copy). Not masking padding/loss-ignored positions, which corrupts the loss. Claiming a base model 'can't do X' when it was simply never instruction-tuned for X. Confusing pretraining scale with fine-tuning scale when estimating cost.

Quick Check

Why can causal language modeling train on unlabeled text at internet scale?

Pick the correct reason.

←Multi-Head Attention and the Transformer Block

Back to Transformers and NLP Foundations

Passion Project: Build a Tiny Transformer→