Lesson 16
Embeddings
Build an LLM From Scratch, 53 lessons
Lesson 16: Embeddings
Build an LLM From Scratch, Lesson 16 of 53

Squeeze the counting network through a narrow middle. See that the weights leaving a token's input circle are its own small set of numbers, that training finds them, and what a narrower middle costs.
The question for this lesson: how can a network give each token its own small set of numbers?
Lesson 15 rebuilt lesson 14’s counting table as a network: six input circles, six output circles and one weight for every pair, 36 in all. Training found the same chances as counting, and nobody counted. It ended with a bill. One weight for every pair of tokens is as big as the table was, and with 50,257 tokens that is 2,525,766,049 weights.
The rest of this lesson is in Premium
Premium includes every lesson, with its notes, practice and progress.