Lesson 32
How big a model
Build an LLM From Scratch, 53 lessons
Lesson 32: How big a model
Build an LLM From Scratch, Lesson 32 of 53

Training costs size times text. Price real runs yourself, see where the models that were really trained sit, and find why, for a fixed budget, a bigger model is not automatically the better one.
The question for this lesson: how big a model should you train for the compute and text you have?
Last lesson cleaned a pile of text and counted it in tokens. We now have a pile of text and a computer to read it with, and both are limited. The model’s size is the one thing we still choose. Lesson 29 counted a model’s size in parameters, the numbers it learns.
It is tempting to think that bigger is simply better.
The rest of this lesson is in Premium
Premium includes every lesson, with its notes, practice and progress.