Build an LLM From Scratch

Lesson 15

The same model as a network

Build an LLM From Scratch, 53 lessons

Rebuild the counting model as a network with every connection drawn. See why its scores become competing shares, why grading by minus ln rewards honest chances, and why walking downhill lands on the counting table.

The question for this lesson: how does a network turn its scores into next-token probabilities, and how do we grade them?

Lesson 14 built a language model out of a table of counts. It could write, and it ended on a worry: a table needs a cell for every context, and it knows nothing about a context it never saw. A network has a fixed number of knobs, and lesson 5 showed that knobs can be turned downhill by themselves.

BREAF Premium

The rest of this lesson is in Premium

Premium includes every lesson, with its notes, practice and progress.