Build an LLM From Scratch

Lesson 42

Learning from preference

Build an LLM From Scratch, 53 lessons

When no reply is simply right, people can still say which of two is better. A reward model turns those choices into a score, training chases the score, and because the score is only a stand-in the chase needs a leash. DPO uses the model's own change in place of a separate reward model.

The question for this lesson: when no single answer is right, how can preference train a model?

Lessons 40 and 41 taught the toy to answer, and every example had one right row: the capital of France is Paris. A real request rarely works that way. Ask for “gravity, explained to a 10-year-old” and a one-liner, a short analogy and a long accurate reply can all be fine. Nothing in the data says which to write.

BREAF Premium

The rest of this lesson is in Premium

Premium includes every lesson, with its notes, practice and progress.