Lesson 27
Keeping numbers steady
Build an LLM From Scratch, 53 lessons
Lesson 27: Keeping numbers steady
Build an LLM From Scratch, Lesson 27 of 53

Layer norm takes one word's own mean and spread and rescales that word to mean 0 and spread 1, so a layer can read a word's pattern on a steadier scale. Without it, changes in size can make attention too sharp or too flat.
The question for this lesson: what does layer norm do, and why?
Lesson 26 added each layer’s answer onto the word instead of replacing it. That keeps the word alive, and it has a side effect: every layer adds something, so the numbers in a word can drift. After a few dozen layers a word can be several times bigger than it started, and another word can be smaller. Nothing in the stream holds the size steady.
A layer reads numbers, so a change in their size can change its answer even when their pattern has not changed. That can make size look like meaning.
The rest of this lesson is in Premium
Premium includes every lesson, with its notes, practice and progress.