Build an LLM From Scratch

Lesson 17

Meaning as position

Build an LLM From Scratch, 53 lessons

Train lesson 16's network on 81 real words and watch words used alike land on the same spot, then compare the two ways to say "near" and try the same ideas on real published vectors.

The question for this lesson: why do tokens used in similar ways end up near each other, and how do we measure that?

Lesson 16 gave every token its own small set of numbers, and training found them. Nobody told those numbers what a word means. Yet the cost fell, so training found something. But that network had only six tokens, and a token could hardly be like any other. So here is the question it left: do tokens used in the same way end up with similar numbers, and why?

BREAF Premium

The rest of this lesson is in Premium

Premium includes every lesson, with its notes, practice and progress.