Build an LLM From Scratch

Lesson 20

Queries, keys and values

Build an LLM From Scratch, 53 lessons

Where attention's scores come from. Every word gets three small networks. One makes what it asks, one makes what it advertises, one makes what it hands over. Match the first against the second, and you have the score.

The question for this lesson: where do the relevance scores come from?

Last lesson’s attention(scores, values) turned scores into shares with softmax, then blended the words’ numbers by those shares. But we typed the scores in.

A real model can’t have someone type them. A stored table of word-pair scores would be huge, and would give a pair the same score in every context. Networks work from the words’ current numbers instead, so they can score pairs never met during training. Later layers will make those numbers reflect the surrounding words.

BREAF Premium

The rest of this lesson is in Premium

Premium includes every lesson, with its notes, practice and progress.