Build an LLM From Scratch

Lesson 19

Attention

Build an LLM From Scratch, 53 lessons

Let every word look straight at the sentence's words and blend what they carry. Score, share, blend. That is attention, and it is the idea the rest of the course turns on.

The question for this lesson: how can a word use the other words in its sentence, without squeezing them all into one memory?

Last lesson gave a network a notepad. It reads one word at a time, writes what it knows into a small memory of fixed size, and hands that memory on. The memory is a bottleneck: everything the network has read has to squeeze through it. By the time the network reached “it” in “the cat sat on the mat because it was tired”, the cat had faded.

Read these two sentences.

  • The cat sat on the mat because it was tired.
  • The cat sat on the mat because it was wet.
BREAF Premium

The rest of this lesson is in Premium

Premium includes every lesson, with its notes, practice and progress.