After this lesson, you will be able to: Build, train, and sample from a small decoder-only transformer on a modest corpus, using a tokenizer you trained, and document it as a research-repo artifact.
This is the Foundations capstone: implement a working GPT-style model end to end. It is the single best way to make the architecture real, and it becomes the substrate for experiments in later sub-tracks.
Train a sub-word tokenizer on a small corpus (a public-domain book, a code dataset, anything a few MB), then implement a small decoder-only transformer (token + positional embeddings, a handful of pre-norm blocks, a final projection to vocab logits), train it with the causal-LM cross-entropy loss on a free Colab/Kaggle GPU, and sample text from it. nanoGPT is the canonical reference; the goal is that you understand every line, not that you invent a new architecture.
Work in a Git repo from the first commit; this is research hygiene.
Milestone 1: Train a BPE tokenizer on your corpus; commit the tokenizer.json + a notebook that shows example tokenizations.
Milestone 2: Implement the model (embeddings, N pre-norm blocks with causal multi-head attention + FFN, final norm + lm_head). Verify shapes on a tiny batch.
Milestone 3: Write the training loop (batching, the shifted-target causal loss, an optimizer like AdamW, gradient clipping). Overfit a tiny dataset first to confirm the loss can reach near zero.
Milestone 4: Train on the full small corpus; log train/val loss and val perplexity; save checkpoints.
Milestone 5: Implement sampling (temperature, top-k, top-p) and generate text. Show samples at several training checkpoints.
Milestone 6: Write a README: architecture, hyperparameters, data, loss curves, sample outputs, and what you would change with more compute.
In an interview or SRW mentorship chat: 'I implemented a decoder-only transformer from scratch, trained my own BPE tokenizer, and trained it to coherent character/sub-word level text on a single GPU. I can walk through attention, the causal mask, pre-norm residual blocks, and the cross-entropy objective, and I profiled where the time goes.' Be honest about scale (a small model on a small corpus) and specific about what you learned (e.g., the effect of the scaling factor, or how sampling temperature changes outputs).
Sign in and purchase access to unlock this lesson.