Lesson 38
Choosing the next token
Build an LLM From Scratch, 53 lessons
Lesson 38: Choosing the next token
Build an LLM From Scratch, Lesson 38 of 53

A model only gives probabilities. Choosing from them is our decision, and every choice trades repeating against rambling. Temperature is the dial, and cutting the tail first cuts the rambling at a mild temperature.
The question for this lesson: how does a model choose the next token?
Last lesson supplied an unrun TinyStories training loop and chose a checkpoint from a separate recorded character-level run. But the model never writes a word. Each time, it gives every token it knows a score, called a logit, and softmax turns the scores into probabilities: positive numbers that add up to 100%, as in lesson 14. Lesson 15 called them shares. That is where the model stops.
The rest of this lesson is in Premium
Premium includes every lesson, with its notes, practice and progress.