Lesson 36
What fills memory
Build an LLM From Scratch, 53 lessons
Lesson 36: What fills memory
Build an LLM From Scratch, Lesson 36 of 53

Training keeps far more in memory than the knobs. See why the way forward saves its numbers for the way back, what a real model's memory bill is made of, and why a 16-bit number can lose an update that a 32-bit number keeps.
The question for this lesson: what fills the GPU’s memory, and how do fewer bits help?
Last lesson ended with Adam, which keeps four numbers for each knob: the knob, its push, and two running blends of pushes. A real model has billions of knobs, and it has to be trained on a card with a fixed amount of memory. A run that needs one byte more than the card has simply stops with an out-of-memory error. So what is that memory being spent on?
The rest of this lesson is in Premium
Premium includes every lesson, with its notes, practice and progress.