Build an LLM From Scratch

Lesson 13

Tokens

Build an LLM From Scratch, 53 lessons

A network only takes numbers, and words are not numbers. See how a tokeniser learns its pieces by counting, and why a number from a table is a label.

The question for this lesson: how does any piece of text become a list of whole numbers, with nothing lost?

Part 1 ended on a network that learns from numbers, and a test on new students to see whether it really learned. Everything it ever took in was a number: hours of study, hours of sleep, hours on the phone.

Words are not numbers. From here on we follow one sentence:

the cat sat on the mat because it was tired

BREAF Premium

The rest of this lesson is in Premium

Premium includes every lesson, with its notes, practice and progress.