Lesson 14
A language model by counting
Build an LLM From Scratch, 53 lessons
Lesson 14: A language model by counting
Build an LLM From Scratch, Lesson 14 of 53

Build a model that predicts the next token with nothing but a table of counts, make it write by drawing from the table, and see where counting runs out.
The question for this lesson: what is a language model, and can we build one just by counting?
Lesson 13 turned any text into tokens, each with a number. The numbers were only labels, so how would you predict the next one?
Your phone does it each time you type: it suggests the next word. A simple way to build that is to count what followed what in text people already wrote, and this lesson builds exactly that, with no network at all. We keep lesson 13’s cat and hen text, because we need its real tokens to count.
The rest of this lesson is in Premium
Premium includes every lesson, with its notes, practice and progress.