Lesson 9
Writing a whole layer
Build an LLM From Scratch, 53 lessons
Lesson 9: Writing a whole layer
Build an LLM From Scratch, Lesson 9 of 53

Write all the neurons of a layer as one grid, and see why a row is a neuron and a column is an input. Then count a network's knobs from the shapes of its grids, and see how width and depth change those counts.
The question for this lesson: how do we write a whole layer in one line, and count its knobs?
Lesson 8 found that more neurons in a layer give more straight cuts, and that more layers fold the map first, so the same cuts land in many places together. It ended on a big network: 3 inputs, 10 layers of 100 neurons and 1 output. That network has 90,400 connections and 1,001 biases.
A computer has to hold every one of those numbers, and run a sum for each of its 1,001 neurons. How do we write that down?
The rest of this lesson is in Premium
Premium includes every lesson, with its notes, practice and progress.