Lesson 18
Memory and forgetting
Build an LLM From Scratch, 53 lessons
Lesson 18: Memory and forgetting
Build an LLM From Scratch, Lesson 18 of 53

Give a network a notepad it rewrites at every word, and watch what happens to a word read early. Old influences can fade, blame can shrink or grow on the way back, and gates help a fixed-size memory choose what to keep.
The question for this lesson: can a network remember by carrying a notepad, and why does that fade?
Lesson 17 gave every word a place on a map. But a word’s numbers come from a table, and the table doesn’t care what sits next to the word. “It” gets the same numbers in every sentence. The words don’t see each other. So when a network reaches “it” in “the cat sat on the mat because it was tired”, how can it know the cat is in play?
The rest of this lesson is in Premium
Premium includes every lesson, with its notes, practice and progress.