Lesson 7
Hidden neurons and the bend
Build an LLM From Scratch, 53 lessons
Lesson 7: Hidden neurons and the bend
Build an LLM From Scratch, Lesson 7 of 53

Slide through the bend of a hidden neuron, move it with your own hands, and build the peak that one neuron could never make. Then see why a network needs bends and smooth curves, and why ordinary downhill training gets stuck with a hard on/off switch.
The question for this lesson: what do hidden neurons and the bend add?
Last lesson found a wall. Students 1, 2 and 3 sleep 4, 7 and 10 hours and score 46, 58 and 46. One neuron with no quiet rule could do no better than saying 50 for all three, a loss of 32. A stack of neurons with the rule off is still one neuron.
So the thing that lets our network say “some sleep is best” has to be the quiet rule, working inside the hidden neurons. This lesson plays with exactly that.
The rest of this lesson is in Premium
Premium includes every lesson, with its notes, practice and progress.