Build an LLM From Scratch

Lesson 22

The mask

Build an LLM From Scratch, 53 lessons

While a model trains, every word is asked to guess the word after it, and attention can read that word. A mask sets those scores to minus infinity, so a word sees only what came before. A broken mask can look healthy for a thousand steps.

The question for this lesson: how do we stop the model reading the answer?

Lesson 19 let every word look at every other word. Lesson 21 kept the scores a calm size, so softmax gives real shares and not a single winner. The formula works. Every word looks at all the others, and that includes the ones after it.

That is fine for a reader who has the whole page. It is not fine for a model that is being trained to write. Here is why.

BREAF Premium

The rest of this lesson is in Premium

Premium includes every lesson, with its notes, practice and progress.