Build an LLM From Scratch

Lesson 26

Adding the input back

Build an LLM From Scratch, 53 lessons

A tall stack can lose information or become hard to train. Adding each layer's input back gives information and learning a direct path through the stack.

The question for this lesson: why do tall stacks fail, and how does adding the input back help?

Every block now has a part that mixes the words and a part that thinks about each one. A single block is not enough. A real model stacks dozens, so each word goes through the first block, then the second, then the third, and learning has to travel back through all of them.

BREAF Premium

The rest of this lesson is in Premium

Premium includes every lesson, with its notes, practice and progress.