Build an LLM From Scratch

Lesson 31

Where the text comes from

Build an LLM From Scratch, 53 lessons

A model knows only what it reads. Clean a small scrape yourself, find that every filter has a cost, and see why copies must go before the held-back pages are chosen.

The question for this lesson: where does training text come from, and how is it cleaned?

Last lesson, the diagram became a program. It has the right shape, but every number inside it is still a guess, so it knows nothing. In this course, it learns from the text we give it, represented as tokens.

A pile of text to train on is called a corpus. The word is Latin for body, and it means no more than that. We measure a corpus in tokens, as in lesson 13, and not in pages or megabytes, because tokens are what the model reads and what the computer pays for.

BREAF Premium

The rest of this lesson is in Premium

Premium includes every lesson, with its notes, practice and progress.