Build an LLM From Scratch

Lesson 9

Writing a whole layer

Build an LLM From Scratch, 53 lessons

Write all the neurons of a layer as one grid, and see why a row is a neuron and a column is an input. Then count a network's knobs from the shapes of its grids, and see how width and depth change those counts.

The question for this lesson: how do we write a whole layer in one line, and count its knobs?

Lesson 8 found that more neurons in a layer give more straight cuts, and that more layers fold the map first, so the same cuts land in many places together. It ended on a big network: 3 inputs, 10 layers of 100 neurons and 1 output. That network has 90,400 connections and 1,001 biases.

A computer has to hold every one of those numbers, and run a sum for each of its 1,001 neurons. How do we write that down?

BREAF Premium

The rest of this lesson is in Premium

Premium includes every lesson, with its notes, practice and progress.