Build an LLM From Scratch

Lesson 28

The block

Build an LLM From Scratch, 53 lessons

A transformer block is attention then a feed-forward network, each reading a normed copy of the stream and adding its answer back. The 2017 figure had an encoder, cross-attention and the norm after each add, because it translated. GPT keeps one text, so it drops the first two and moves the norm into each branch.

The question for this lesson: how do the parts fit into one block, and how does that differ from 2017?

Lessons 19 to 27 gave us every part: attention, several heads, the mask, the feed-forward network, adding the input back, and a norm. They are separate tools on a bench. This lesson puts them in order, and the order has a history.

BREAF Premium

The rest of this lesson is in Premium

Premium includes every lesson, with its notes, practice and progress.