Build an LLM From Scratch

Lesson 34

Step size and clipping

Build an LLM From Scratch, 53 lessons

A step moves the knobs against the gradient, but how far? Draw the step size over a whole run, see why the same step that helps early hurts late, and see what a cap on the push's length does to one bad batch.

The question for this lesson: how big should each step be, and what stops one bad step?

Last lesson showed what one step reads: a batch of windows, whose push on the knobs is a rough copy of the push the whole text would give. The step then moves every knob against that push. How far is the step size, the learning rate of lesson 5. Too small and a run crawls, and too big and it overshoots.

BREAF Premium

The rest of this lesson is in Premium

Premium includes every lesson, with its notes, practice and progress.