Transformers and NLP Foundations

Lessons

1. 1

   What LLM and NLP Research Is

   Free

   Map the NLP research landscape, the venues (ACL/EMNLP/NAACL + SRW), and the shape of a research contribution.

   Intermediate·30 min
2. 2

   Text as Data: Tokens and Vocabulary

   Explain why text becomes a fixed vocabulary of sub-word tokens and the character/word/sub-word tradeoffs.

   Intermediate·40 min
3. 3

   Training a Tokenizer: BPE, WordPiece, SentencePiece

   Train a BPE tokenizer from scratch and distinguish BPE, WordPiece, and SentencePiece/Unigram.

   Advanced·50 min
4. 4

   Embeddings and Positional Encoding

   Explain embeddings as a lookup table and the positional schemes (sinusoidal, learned, RoPE) you meet in papers.

   Advanced·45 min
5. 5

   Attention from the Math Up

   Derive and implement scaled dot-product attention with the sqrt(d_k) scaling and causal masking.

   Advanced·55 min
6. 6

   Multi-Head Attention and the Transformer Block

   Assemble multi-head attention, residuals, LayerNorm, and the FFN; distinguish encoder vs decoder stacks.

   Advanced·50 min
7. 7

   Pretraining Objectives

   Compare causal LM, masked LM, and denoising objectives and the cross-entropy loss they minimize.

   Advanced·45 min
8. 8

   Passion Project: Build a Tiny Transformer

   Implement, train, and sample from a small decoder-only transformer with your own tokenizer.

   Advanced·240 min
9. 9

   Foundations Research Readiness

   Translate transformer foundations into resume signals and the architecture questions research roles screen on.

   Intermediate·35 min