Lesson 28
The block
Build an LLM From Scratch, 53 lessons
Lesson 28: The block
Build an LLM From Scratch, Lesson 28 of 53

A transformer block is attention then a feed-forward network, each reading a normed copy of the stream and adding its answer back. The 2017 figure had an encoder, cross-attention and the norm after each add, because it translated. GPT keeps one text, so it drops the first two and moves the norm into each branch.
The question for this lesson: how do the parts fit into one block, and how does that differ from 2017?
Lessons 19 to 27 gave us every part: attention, several heads, the mask, the feed-forward network, adding the input back, and a norm. They are separate tools on a bench. This lesson puts them in order, and the order has a history.
The rest of this lesson is in Premium
Premium includes every lesson, with its notes, practice and progress.