Build an LLM From Scratch

Lesson 39

Why generation stays fast

Build an LLM From Scratch, 53 lessons

Writing a word needs the keys and values of every word before it, and they never change, so the model keeps them. That turns a growing pile of repeated work into one new word's work, and the price is memory, which sharing key-value sets shrinks.

The question for this lesson: why does generation stay fast, and what does that cost?

Last lesson ended on the loop. The model reads the text so far and gives every token a probability. A sampler picks one. The pick joins the text, and the model reads again. Choosing is cheap: a few lines of code. Reading is the expensive part, and the loop seems to make the model read everything written so far, every time. A long answer would mean reading a long text again for every new word. Why isn’t that unbearably slow?

BREAF Premium

The rest of this lesson is in Premium

Premium includes every lesson, with its notes, practice and progress.