Build an LLM From Scratch

Lesson 16

Embeddings

Build an LLM From Scratch, 53 lessons

Squeeze the counting network through a narrow middle. See that the weights leaving a token's input circle are its own small set of numbers, that training finds them, and what a narrower middle costs.

The question for this lesson: how can a network give each token its own small set of numbers?

Lesson 15 rebuilt lesson 14’s counting table as a network: six input circles, six output circles and one weight for every pair, 36 in all. Training found the same chances as counting, and nobody counted. It ended with a bill. One weight for every pair of tokens is as big as the table was, and with 50,257 tokens that is 2,525,766,049 weights.

BREAF Premium

The rest of this lesson is in Premium

Premium includes every lesson, with its notes, practice and progress.