Build an LLM From Scratch

Lesson 38

Choosing the next token

Build an LLM From Scratch, 53 lessons

A model only gives probabilities. Choosing from them is our decision, and every choice trades repeating against rambling. Temperature is the dial, and cutting the tail first cuts the rambling at a mild temperature.

The question for this lesson: how does a model choose the next token?

Last lesson supplied an unrun TinyStories training loop and chose a checkpoint from a separate recorded character-level run. But the model never writes a word. Each time, it gives every token it knows a score, called a logit, and softmax turns the scores into probabilities: positive numbers that add up to 100%, as in lesson 14. Lesson 15 called them shares. That is where the model stops.

BREAF Premium

The rest of this lesson is in Premium

Premium includes every lesson, with its notes, practice and progress.