Lesson 24
Word order
Build an LLM From Scratch, 53 lessons
Lesson 24: Word order
Build an LLM From Scratch, Lesson 24 of 53

Attention treats a sentence as a bag of words. Three ways give a word its place back, and they differ in what they cope with.
The question for this lesson: attention can’t see order, so how do models give it back?
Lesson 23 ran attention in several heads. Each head scores words by what they are. We ended on a worry: shuffle the words before a word, and the blend it gathers stays exactly the same.
“Dog bit man” is a Tuesday. “Man bit dog” is news. They hold the same three words, and only the order says who did the biting.
Think about the word “bit” in each, with no mask, so it reads all three words. Picture 1 has three tries.
[[breaf-artifact:1]] Do the three tries in Picture 1 before reading on.
The rest of this lesson is in Premium
Premium includes every lesson, with its notes, practice and progress.