Build an LLM From Scratch

Lesson 23

Several heads

Build an LLM From Scratch, 53 lessons

One attention pass gives a word one list of shares that adds up to 100%, so two things it needs have to split it. Several heads each get their own query, key and value networks and their own 100%, run side by side, and cost no extra knobs.

The question for this lesson: why run attention many times side by side?

Lesson 22 put the mask on every word, so each word reads itself and only earlier words. Attention is now complete, and it works. But look at what a word gets out of it. It gets one blend, made from one list of shares, and those shares add up to 100%.

BREAF Premium

The rest of this lesson is in Premium

Premium includes every lesson, with its notes, practice and progress.