Lesson 31
Where the text comes from
Build an LLM From Scratch, 53 lessons
Lesson 31: Where the text comes from
Build an LLM From Scratch, Lesson 31 of 53

A model knows only what it reads. Clean a small scrape yourself, find that every filter has a cost, and see why copies must go before the held-back pages are chosen.
The question for this lesson: where does training text come from, and how is it cleaned?
Last lesson, the diagram became a program. It has the right shape, but every number inside it is still a guess, so it knows nothing. In this course, it learns from the text we give it, represented as tokens.
A pile of text to train on is called a corpus. The word is Latin for body, and it means no more than that. We measure a corpus in tokens, as in lesson 13, and not in pages or megabytes, because tokens are what the model reads and what the computer pays for.
The rest of this lesson is in Premium
Premium includes every lesson, with its notes, practice and progress.