After this lesson, you will be able to: Explain how token ids become vectors (embeddings) and how transformers inject word order through positional encoding (sinusoidal, learned, and RoPE).
Tokens are integers; a transformer needs vectors, and it needs to know order because attention is order-blind by default. This lesson covers embeddings and the positional encoding schemes you will see named in every architecture paper.
An embedding layer is a lookup table: a matrix of shape (vocab_size, d_model). Token id 4217 just selects row 4217, a learned d_model-dimensional vector. These vectors are trained, so semantically related tokens drift toward similar directions. This is the modern descendant of word2vec and GloVe, with one key difference: in a transformer the embedding is only the starting point; attention layers then make each token's representation context-dependent, which is why the same word gets different vectors in different sentences (contextual embeddings).
Sinusoidal (original Transformer): fixed sine/cosine functions of position added to embeddings; no parameters, extrapolates somewhat to longer sequences. Learned absolute (BERT, GPT-2): a trainable position embedding per slot up to a max length; simple but cannot exceed the trained length. Rotary Position Embedding (RoPE, used by Llama and most modern LLMs): rotates the query/key vectors by an angle proportional to position, encoding *relative* position inside attention; it generalizes better to longer contexts and underlies context-extension tricks (position interpolation, YaRN). ALiBi is another relative scheme (a linear bias on attention scores). When a paper says 'we use RoPE with a base of 10000,' you now know what that means.
Assuming a model can handle inputs longer than its trained positional range without degradation (learned absolute positions simply have no embedding past the limit). Confusing the embedding dimension d_model with the vocabulary size. Forgetting that tying the input embedding and output projection (weight tying) is a real, common choice that saves parameters. Treating positional encoding as a solved detail when it is precisely where many long-context papers make their contribution.
Pick the correct reason.
Sign in and purchase access to unlock this lesson.