Lesson 15
The same model as a network
Build an LLM From Scratch, 53 lessons
Lesson 15: The same model as a network
Build an LLM From Scratch, Lesson 15 of 53

Rebuild the counting model as a network with every connection drawn. See why its scores become competing shares, why grading by minus ln rewards honest chances, and why walking downhill lands on the counting table.
The question for this lesson: how does a network turn its scores into next-token probabilities, and how do we grade them?
Lesson 14 built a language model out of a table of counts. It could write, and it ended on a worry: a table needs a cell for every context, and it knows nothing about a context it never saw. A network has a fixed number of knobs, and lesson 5 showed that knobs can be turned downhill by themselves.
The rest of this lesson is in Premium
Premium includes every lesson, with its notes, practice and progress.