Build an LLM From Scratch

Lesson 32

How big a model

Build an LLM From Scratch, 53 lessons

Training costs size times text. Price real runs yourself, see where the models that were really trained sit, and find why, for a fixed budget, a bigger model is not automatically the better one.

The question for this lesson: how big a model should you train for the compute and text you have?

Last lesson cleaned a pile of text and counted it in tokens. We now have a pile of text and a computer to read it with, and both are limited. The model’s size is the one thing we still choose. Lesson 29 counted a model’s size in parameters, the numbers it learns.

It is tempting to think that bigger is simply better.

BREAF Premium

The rest of this lesson is in Premium

Premium includes every lesson, with its notes, practice and progress.