Build an LLM From Scratch

Lesson 1

What a neural network is

Build an LLM From Scratch, 53 lessons

Course content1 of 53

What a neural network is

The question for this lesson: what is a neural network, and what does it do with numbers?

A problem rules struggle with

Say you want to guess a student’s marks from three numbers: hours of study, hours of sleep, and hours on the phone.

Here are twelve students. We made them up, so don’t read them as advice about sleep or phones. They stay with us for this whole part of the course.

StudentStudySleepPhoneMarks
144246
247258
3410246
416521
528424
635147
759352
866364
968081
1077183
1185280
1289183

You could try writing a rule. “More study, more marks” gets you a long way.

Now look at students 1, 2 and 3. They studied the same and used their phones the same. Only sleep differs: 4 hours, 7 hours, 10 hours. Their marks are 46, 58, 46. More sleep helped, and then it hurt.

So you add a rule about sleep, then an exception, then an exception to the exception. With real data, the list soon gets too long to write.

A neural network takes a different route. Instead of a list of rules about sleep or phones, it holds a set of adjustable numbers. Your numbers go in one side, pass through calculations using those settings, and an answer comes out the other. The aim is to find settings that give close answers, including for students who aren’t in the table. That’s the reason to build one. Learning means adjusting those settings using examples, rather than writing each rule yourself. A network can always give an answer; whether to trust it needs a separate test.

The parts

Here is a small network for our students: three circles on the left, two in the middle, one on the right, joined by lines. Picture 1 draws it. Tap each name there to see which part it means. We’ll call the two middle circles neuron 1, the top one, and neuron 2.

  • Inputs are the numbers going in, with one circle for each: study, sleep, phone. These circles don’t calculate anything. They only hold the numbers.
  • Neurons are the other circles. Each one is a small calculation: numbers arrive, and one number leaves. The name comes from brain cells, but the likeness is loose. Picture a small sum, not a brain.
  • Connections are the lines. Each carries a number from one circle to the next. An input’s number is copied along every line out of it, not split between them. Here every input connects to every neuron in the next column, and both of those connect to the last one. That makes 6 + 2 = 8 connections.
  • Layers are the columns. The last layer is the output, one neuron whose number is the answer.
  • The hidden layer is everything between inputs and output. It is called hidden because our data never shows it. The table gives hours and marks, and says nothing about what the middle should be. Here the middle numbers follow from weights we set by hand. Later, the network will find those weights itself. Why have a middle at all? Lesson 7 earns it.

People write this shape as 3→2→1: three inputs, two hidden neurons, one output.

Do the two tries in Picture 1 before reading on. They teach the names of the connections, which we use from here on.

Weights

So a connection carries a number. It also has a setting of its own, called a weight, which stays put while different numbers pass through. A weight says how much that input matters to that neuron.

  • A large weight means each hour counts for a lot in that neuron’s calculation. Study to neuron 1 is 7.
  • A negative weight means it pushes the other way. Phone to neuron 1 is −3, so more phone pushes neuron 1 down. Its number can stop at zero; lesson 2 explains why.
  • A weight of zero means it is ignored. Study to neuron 2 is 0. The connection is still there, but it counts for nothing, for now.

Think of each weight as a knob. Turning it can change the network’s answers. If nothing arrives on that connection, turning it has no effect for that student.

We set these eight by hand, so the numbers stay round. The next few lessons show how a network finds good settings by itself.

Follow one student

Student 5 studied 2 hours, slept 8, and spent 4 on the phone.

  1. The inputs take the values 2, 8 and 4.
  2. Those numbers travel along six connections into the hidden layer. Neuron 1 gives 22. Neuron 2 gives 4.
  3. Those two numbers travel along two connections into the output. It gives 26.

So the network’s answer for student 5 is 26. The table says 24. Close, not exact.

That trip, left to right, one layer at a time, is called a forward pass. One student goes in, the numbers make one pass, and one answer comes out.

Every student goes through the same circles and the same weights. Only the inputs change.

Picture 2 is the real network, running. Pick a student or drag the hours. Drag any connection sideways to turn its weight. The row of twelve shows every student’s answer at once, and counts how many are within 5 marks of the table. With our weights, all twelve start close.

Do the eight tries in Picture 2 before reading on. Each one asks you to guess or choose, then shows you why. Three of them close an idea with a new student, and a twin case if you miss.

Three things came out of the tries.

  • A connection can change only what it feeds, and what that feeds in turn. Turning the connection from neuron 1 to the output can’t change neuron 1.
  • One step on a knob adds one more lot of the number arriving on it to the next circle’s calculation. A hidden circle passes that change on while its number stays above 0. Its connection to the output then scales that change too. So a thin knob with big hours arriving can beat a fat knob with small ones. A knob into a circle at 0 may do nothing at all, and lesson 2 says why. Picture 2 says so if you find one.
  • Every student goes through the same weights. For the goal, you could turn study to neuron 1 down to 6, which makes student 5 exactly 24. But then all eleven of the other students move further from the table, and that’s with only eight connections to think about.

The code

This is Picture 2 as Python. The function network is the whole picture, and the eight weights in it are the ones you moved.

weights = [[7, 4, -3], [0, 2, 1]]
extra = [-12, -16]
to_answer = [1, -4]
start = 20

def dot(a, b):
    return sum(p * q for p, q in zip(a, b))

def network(study, sleep, phone):
    hours = [study, sleep, phone]
    neurons = []
    for w, e in zip(weights, extra):
        total = dot(w, hours)
        neurons.append(max(0, total + e))
    answer = start + dot(to_answer, neurons)
    return neurons, answer

print(network(2, 8, 4))
# ([22, 4], 26)
weights[0][0] = 6
print(network(2, 8, 4)[1])
# 24
print(network(8, 9, 1)[1])
# 77

Here is how to read it, even if you have never coded.

  • weights holds the six connections into the hidden layer, one list for each neuron: study, sleep and phone to neuron 1, then the same three to neuron 2. These are the knobs you dragged on the left of Picture 2. to_answer holds the two connections to the output.
  • extra and start are three more numbers we haven’t explained. Lesson 2 opens them, together with max(0, ...).
  • dot multiplies two lists pair by pair and adds the results. It is the arithmetic behind each circle’s number.
  • hours is the three input circles, the sliders in Picture 2.
  • The for line runs once for each hidden neuron. total is dot of the hours with that neuron’s weights. neurons collects the two hidden circles.
  • answer does the same on the two connections to the output. It is the “Network says” tile. return hands back both.
  • The first print is student 5: neurons 22 and 4, and the answer 26.
  • weights[0][0] = 6 turns the study to neuron 1 knob down to 6, which is the goal. Student 5 now says 24. The last print is student 12, who studied 8 hours: the network said 85 before, the table says 83, and now it says 77.

From eight connections to 175 billion

Our network has eight connections, and next lesson you’ll meet a few more knobs. GPT-3, a language model from 2020 (a network trained on text), has 175 billion adjustable numbers. It also has parts we haven’t met yet, and later parts of this course build them one at a time.

The idea is the same. Numbers go in, numbers come out, and adjustable numbers decide what happens in between. Later, the numbers going in will stand for pieces of text, and the numbers coming out will say what text is likely to come next.

What we simplified

  • The eight weights are not all the knobs. Three more numbers, extra and start in the code, are adjustable too, and a rule inside each hidden neuron decides when it speaks. Lesson 2 explains both.
  • We set the weights by hand, so the numbers stay round. They aren’t the best weights for these students. Lessons 3 to 5 measure how wrong a network is and walk it towards better settings.
  • The twelve students and their marks are made up, so don’t read advice about sleep or phones into them. A real network would be built from far more examples.
  • Giving an answer for unlisted hours does not show it is accurate. Lesson 12 tests whether a network learned something useful for new examples. Our output can also go outside the range of possible marks; we have not constrained it.
  • Every circle here is part of one small 3→2→1 shape. Lesson 7 shows why a middle layer is worth having.

What’s still missing

Look again at neuron 1 for student 5. The numbers arriving are 2, 8 and 4, and the weights are 7, 4 and −3. A fair guess is that a weight of 7 makes each hour count 7 times. Multiplying each pair and adding gives 14 + 32 − 12 = 34.

But neuron 1 said 22.

So something more is going on inside that circle. What happens inside one of those circles?