Build an LLM From Scratch

Lesson 25

The feed-forward network

Build an LLM From Scratch, 53 lessons

Attention mixes words together. The feed-forward network then thinks about each mixed word on its own, with the same stored weights for every word.

The question for this lesson: what does a word do with what it gathered?

By lesson 24 every word carries its meaning and its place, and attention has let each one gather what it needs from the others. In lesson 19’s unmasked picture, “it” ended up part cat and part mat, and which part depended on the last word of the sentence.

BREAF Premium

The rest of this lesson is in Premium

Premium includes every lesson, with its notes, practice and progress.