Lesson 33
What one step reads
Build an LLM From Scratch, 53 lessons
Lesson 33: What one step reads
Build an LLM From Scratch, Lesson 33 of 53

A training step does not read a book. It reads a batch of short windows cut from the text, with the answers cut one place later. Cut windows yourself, fill the rows of a batch, and see how well a few windows stand in for the whole text.
The question for this lesson: what does one training step read, and why average many?
Last lesson priced a whole run: how big the model should be, and how much text it should read. That counted the run in one lump. A run is really a very long line of small steps, and each step reads only a little of the text.
GPT-3’s biggest model read 300 billion tokens, 3.2 million tokens at a time. That is at least 93,750 steps, because the paper starts with smaller batches. Each of them is the loop of lesson 11: predict, measure, blame, nudge. What we have not said is what a step predicts on.
The rest of this lesson is in Premium
Premium includes every lesson, with its notes, practice and progress.