Build an LLM From Scratch

Lesson 18

Memory and forgetting

Build an LLM From Scratch, 53 lessons

Give a network a notepad it rewrites at every word, and watch what happens to a word read early. Old influences can fade, blame can shrink or grow on the way back, and gates help a fixed-size memory choose what to keep.

The question for this lesson: can a network remember by carrying a notepad, and why does that fade?

Lesson 17 gave every word a place on a map. But a word’s numbers come from a table, and the table doesn’t care what sits next to the word. “It” gets the same numbers in every sentence. The words don’t see each other. So when a network reaches “it” in “the cat sat on the mat because it was tired”, how can it know the cat is in play?

BREAF Premium

The rest of this lesson is in Premium

Premium includes every lesson, with its notes, practice and progress.