Lesson 26
Adding the input back
Build an LLM From Scratch, 53 lessons
Lesson 26: Adding the input back
Build an LLM From Scratch, Lesson 26 of 53

A tall stack can lose information or become hard to train. Adding each layer's input back gives information and learning a direct path through the stack.
The question for this lesson: why do tall stacks fail, and how does adding the input back help?
Every block now has a part that mixes the words and a part that thinks about each one. A single block is not enough. A real model stacks dozens, so each word goes through the first block, then the second, then the third, and learning has to travel back through all of them.
BREAF Premium
The rest of this lesson is in Premium
Premium includes every lesson, with its notes, practice and progress.