Lesson 19
Attention
Build an LLM From Scratch, 53 lessons
Lesson 19: Attention
Build an LLM From Scratch, Lesson 19 of 53

Let every word look straight at the sentence's words and blend what they carry. Score, share, blend. That is attention, and it is the idea the rest of the course turns on.
The question for this lesson: how can a word use the other words in its sentence, without squeezing them all into one memory?
Last lesson gave a network a notepad. It reads one word at a time, writes what it knows into a small memory of fixed size, and hands that memory on. The memory is a bottleneck: everything the network has read has to squeeze through it. By the time the network reached “it” in “the cat sat on the mat because it was tired”, the cat had faded.
Read these two sentences.
- The cat sat on the mat because it was tired.
- The cat sat on the mat because it was wet.
The rest of this lesson is in Premium
Premium includes every lesson, with its notes, practice and progress.