Build an LLM From Scratch

Lesson 24

Word order

Build an LLM From Scratch, 53 lessons

Attention treats a sentence as a bag of words. Three ways give a word its place back, and they differ in what they cope with.

The question for this lesson: attention can’t see order, so how do models give it back?

Lesson 23 ran attention in several heads. Each head scores words by what they are. We ended on a worry: shuffle the words before a word, and the blend it gathers stays exactly the same.

“Dog bit man” is a Tuesday. “Man bit dog” is news. They hold the same three words, and only the order says who did the biting.

Think about the word “bit” in each, with no mask, so it reads all three words. Picture 1 has three tries.

[[breaf-artifact:1]] Do the three tries in Picture 1 before reading on.

BREAF Premium

The rest of this lesson is in Premium

Premium includes every lesson, with its notes, practice and progress.