Build an LLM From Scratch

Lesson 29

The whole model

Build an LLM From Scratch, 53 lessons

The whole model is a token table and a position table at the bottom, a stack of blocks with the same design and separate weights with the stream running through them, a final norm, and an output head that scores every word against the token table's own rows. Its size comes from a few numbers, and width counts far more than heads.

The question for this lesson: what is the full model, and what decides its size?

Lesson 28 gave us one block: attention, then a feed-forward network, each reading a normed copy of the stream and adding its answer back. A block takes a list of numbers for every word in and hands a list of the same shape out. That is all it does.

A model has to do more. It has to turn words into those numbers, and turn the numbers at the top into a guess at the next word. And someone will ask the plain question: how many numbers is it?

BREAF Premium

The rest of this lesson is in Premium

Premium includes every lesson, with its notes, practice and progress.