Lesson 20
Queries, keys and values
Build an LLM From Scratch, 53 lessons
Lesson 20: Queries, keys and values
Build an LLM From Scratch, Lesson 20 of 53

Where attention's scores come from. Every word gets three small networks. One makes what it asks, one makes what it advertises, one makes what it hands over. Match the first against the second, and you have the score.
The question for this lesson: where do the relevance scores come from?
Last lesson’s attention(scores, values) turned scores into shares with softmax, then blended the words’ numbers by those shares. But we typed the scores in.
A real model can’t have someone type them. A stored table of word-pair scores would be huge, and would give a pair the same score in every context. Networks work from the words’ current numbers instead, so they can score pairs never met during training. Later layers will make those numbers reflect the surrounding words.
The rest of this lesson is in Premium
Premium includes every lesson, with its notes, practice and progress.