Build an LLM From Scratch

Lesson 7

Hidden neurons and the bend

Build an LLM From Scratch, 53 lessons

Slide through the bend of a hidden neuron, move it with your own hands, and build the peak that one neuron could never make. Then see why a network needs bends and smooth curves, and why ordinary downhill training gets stuck with a hard on/off switch.

The question for this lesson: what do hidden neurons and the bend add?

Last lesson found a wall. Students 1, 2 and 3 sleep 4, 7 and 10 hours and score 46, 58 and 46. One neuron with no quiet rule could do no better than saying 50 for all three, a loss of 32. A stack of neurons with the rule off is still one neuron.

So the thing that lets our network say “some sleep is best” has to be the quiet rule, working inside the hidden neurons. This lesson plays with exactly that.

BREAF Premium

The rest of this lesson is in Premium

Premium includes every lesson, with its notes, practice and progress.