Lesson 25
The feed-forward network
Build an LLM From Scratch, 53 lessons
Lesson 25: The feed-forward network
Build an LLM From Scratch, Lesson 25 of 53

Attention mixes words together. The feed-forward network then thinks about each mixed word on its own, with the same stored weights for every word.
The question for this lesson: what does a word do with what it gathered?
By lesson 24 every word carries its meaning and its place, and attention has let each one gather what it needs from the others. In lesson 19’s unmasked picture, “it” ended up part cat and part mat, and which part depended on the last word of the sentence.
BREAF Premium
The rest of this lesson is in Premium
Premium includes every lesson, with its notes, practice and progress.