Build an LLM From Scratch

Lesson 33

What one step reads

Build an LLM From Scratch, 53 lessons

A training step does not read a book. It reads a batch of short windows cut from the text, with the answers cut one place later. Cut windows yourself, fill the rows of a batch, and see how well a few windows stand in for the whole text.

The question for this lesson: what does one training step read, and why average many?

Last lesson priced a whole run: how big the model should be, and how much text it should read. That counted the run in one lump. A run is really a very long line of small steps, and each step reads only a little of the text.

GPT-3’s biggest model read 300 billion tokens, 3.2 million tokens at a time. That is at least 93,750 steps, because the paper starts with smaller batches. Each of them is the loop of lesson 11: predict, measure, blame, nudge. What we have not said is what a step predicts on.

BREAF Premium

The rest of this lesson is in Premium

Premium includes every lesson, with its notes, practice and progress.