Build an LLM From Scratch

Lesson 36

What fills memory

Build an LLM From Scratch, 53 lessons

Training keeps far more in memory than the knobs. See why the way forward saves its numbers for the way back, what a real model's memory bill is made of, and why a 16-bit number can lose an update that a 32-bit number keeps.

The question for this lesson: what fills the GPU’s memory, and how do fewer bits help?

Last lesson ended with Adam, which keeps four numbers for each knob: the knob, its push, and two running blends of pushes. A real model has billions of knobs, and it has to be trained on a card with a fixed amount of memory. A run that needs one byte more than the card has simply stops with an out-of-memory error. So what is that memory being spent on?

BREAF Premium

The rest of this lesson is in Premium

Premium includes every lesson, with its notes, practice and progress.