Lesson 39
Why generation stays fast
Build an LLM From Scratch, 53 lessons
Lesson 39: Why generation stays fast
Build an LLM From Scratch, Lesson 39 of 53

Writing a word needs the keys and values of every word before it, and they never change, so the model keeps them. That turns a growing pile of repeated work into one new word's work, and the price is memory, which sharing key-value sets shrinks.
The question for this lesson: why does generation stay fast, and what does that cost?
Last lesson ended on the loop. The model reads the text so far and gives every token a probability. A sampler picks one. The pick joins the text, and the model reads again. Choosing is cheap: a few lines of code. Reading is the expensive part, and the loop seems to make the model read everything written so far, every time. A long answer would mean reading a long text again for every new word. Why isn’t that unbearably slow?
The rest of this lesson is in Premium
Premium includes every lesson, with its notes, practice and progress.