Build an LLM From Scratch

Lesson 14

A language model by counting

Build an LLM From Scratch, 53 lessons

Build a model that predicts the next token with nothing but a table of counts, make it write by drawing from the table, and see where counting runs out.

The question for this lesson: what is a language model, and can we build one just by counting?

Lesson 13 turned any text into tokens, each with a number. The numbers were only labels, so how would you predict the next one?

Your phone does it each time you type: it suggests the next word. A simple way to build that is to count what followed what in text people already wrote, and this lesson builds exactly that, with no network at all. We keep lesson 13’s cat and hen text, because we need its real tokens to count.

BREAF Premium

The rest of this lesson is in Premium

Premium includes every lesson, with its notes, practice and progress.