Build an LLM From Scratch

Lesson 27

Keeping numbers steady

Build an LLM From Scratch, 53 lessons

Layer norm takes one word's own mean and spread and rescales that word to mean 0 and spread 1, so a layer can read a word's pattern on a steadier scale. Without it, changes in size can make attention too sharp or too flat.

The question for this lesson: what does layer norm do, and why?

Lesson 26 added each layer’s answer onto the word instead of replacing it. That keeps the word alive, and it has a side effect: every layer adds something, so the numbers in a word can drift. After a few dozen layers a word can be several times bigger than it started, and another word can be smaller. Nothing in the stream holds the size steady.

A layer reads numbers, so a change in their size can change its answer even when their pattern has not changed. That can make size look like meaning.

BREAF Premium

The rest of this lesson is in Premium

Premium includes every lesson, with its notes, practice and progress.