Lesson 22
The mask
Build an LLM From Scratch, 53 lessons
Lesson 22: The mask
Build an LLM From Scratch, Lesson 22 of 53

While a model trains, every word is asked to guess the word after it, and attention can read that word. A mask sets those scores to minus infinity, so a word sees only what came before. A broken mask can look healthy for a thousand steps.
The question for this lesson: how do we stop the model reading the answer?
Lesson 19 let every word look at every other word. Lesson 21 kept the scores a calm size, so softmax gives real shares and not a single winner. The formula works. Every word looks at all the others, and that includes the ones after it.
That is fine for a reader who has the whole page. It is not fine for a model that is being trained to write. Here is why.
The rest of this lesson is in Premium
Premium includes every lesson, with its notes, practice and progress.