Learning Tracks
Your roadmap to real skills.
Many tracks and dozens of sub-tracks. Every track starts with a free intro lesson.
Learning Tracks
Many tracks and dozens of sub-tracks. Every track starts with a free intro lesson.
Tokenization, embeddings, attention, and the transformer from the math up.
~12 hours · 9 lessons
View track details →What LLM and NLP Research Is
Map the NLP research landscape, the venues (ACL/EMNLP/NAACL + SRW), and the shape of a research contribution.
Text as Data: Tokens and Vocabulary
Explain why text becomes a fixed vocabulary of sub-word tokens and the character/word/sub-word tradeoffs.
Training a Tokenizer: BPE, WordPiece, SentencePiece
Train a BPE tokenizer from scratch and distinguish BPE, WordPiece, and SentencePiece/Unigram.
Embeddings and Positional Encoding
Explain embeddings as a lookup table and the positional schemes (sinusoidal, learned, RoPE) you meet in papers.
Attention from the Math Up
Derive and implement scaled dot-product attention with the sqrt(d_k) scaling and causal masking.
Multi-Head Attention and the Transformer Block
Assemble multi-head attention, residuals, LayerNorm, and the FFN; distinguish encoder vs decoder stacks.
Pretraining Objectives
Compare causal LM, masked LM, and denoising objectives and the cross-entropy loss they minimize.
Passion Project: Build a Tiny Transformer
Implement, train, and sample from a small decoder-only transformer with your own tokenizer.
Foundations Research Readiness
Translate transformer foundations into resume signals and the architecture questions research roles screen on.