Lesson 5
Walking downhill
Build an LLM From Scratch, 53 lessons
Lesson 5: Walking downhill
Build an LLM From Scratch, Lesson 5 of 53

Turn the slopes into a machine that improves the network by itself. Learn the update rule, what a learning rate does when it is too small, right or too big, why some starts go nowhere, and why the hours are shifted and scaled first.
The question for this lesson: how does a network use slopes to improve itself, and how big should each step be?
Last lesson gave every connection an arrow. In lesson 4, Picture 3, you followed one of them: you turned Neuron 1 to output a whole step up, and the loss went from 256.25 to 301.42, higher than where it started. A half step would have brought it down to 20.54. The arrow was right about the direction and said nothing about the distance. Moving a knob can change the other slopes too, so an old reading may no longer fit.
The rest of this lesson is in Premium
Premium includes every lesson, with its notes, practice and progress.