Lesson 35
A smarter walker
Build an LLM From Scratch, 53 lessons
Lesson 35: A smarter walker
Build an LLM From Scratch, Lesson 35 of 53

A plain step uses only the push it was just handed. See why that bounces and crawls, how remembering the last pushes helps, how Adam gives every knob a step of its own, and what weight decay does about a network that grows too sure of its text.
The question for this lesson: why does plain gradient descent struggle, and what does Adam do about it?
Last lesson drew the step size over a whole run. The best drawing ended at 1.81, and the lowest the network of lesson 15 can reach is 1.67. Whatever the step size, a plain step takes the push it was handed this step, multiplies it by the step size, and forgets it. It remembers nothing of the steps before. And it gives one step size to all 111,556 knobs, however big or small each knob’s own push is.
The rest of this lesson is in Premium
Premium includes every lesson, with its notes, practice and progress.