Lesson 2
Inside one neuron
Build an LLM From Scratch, 53 lessons
Lesson 2: Inside one neuron
Build an LLM From Scratch, Lesson 2 of 53
Course content2 of 53
Inside one neuron
The question for this lesson: what does a single neuron do with its inputs?
Where we left off
Last lesson, student 5 went through the network: 2 hours of study, 8 of sleep, 4 on the phone. Neuron 1 gave 22.
We tried multiplying each input by its weight and adding, and got 34. So 34 isn’t the whole story. Let’s open the circle.
Do the three tries in Picture 1 before reading on, the last of them a check. Here are the four steps you went through.
Step 1: multiply each input by its weight
A weight says how much an input matters to this neuron. So each hour of study counts for 7 here, and 2 hours count for 7 × 2 = 14.
| Input | Hours | Weight | Hours × weight |
|---|---|---|---|
| Study | 2 | 7 | 14 |
| Sleep | 8 | 4 | 32 |
| Phone | 4 | −3 | −12 |
Phone comes out negative because its weight is negative, so it pulls the neuron’s number down.
Step 2: add them up
To combine the weighted inputs into one number, this neuron adds them: 14 + 32 − 12 = 34.
Multiplying by weights and then adding is called a weighted sum. It’s the same idea as an exam where some tests count for more than others.
Step 3: add the bias
Each neuron also has one number of its own, called a bias. It belongs to the neuron, not to any connection, and it gets added to the sum. Neuron 1’s bias is −12, so 34 − 12 = 22. Call the result the total.
That’s the 22 we were looking for.
The word has nothing to do with unfairness. It’s only a name. We’ll see why a neuron needs one in a moment.
Step 4: the quiet rule
If the total is below zero, the neuron gives 0. Otherwise it gives the total. We’ll call this the quiet rule. The output neuron is the one exception, and we’ll come to why. Networks use other rules like this one too, and lesson 7 compares them.
Why zero? Because anything times 0 is 0. A neuron giving 0 adds nothing to the next neuron’s sum, so giving 0 means staying silent.
Student 5 has a total of 22, so neuron 1 gives 22. Student 1 has a total of −6 at neuron 2, so neuron 2 gives 0. That’s why, last lesson, turning a weight on neuron 2 did nothing for student 1: the total was below zero and it stayed there.
Why have a quiet rule at all?
It looks like a strange thing to add. Here’s a way to find out why it’s there.
Read neuron 2 as “too much sleep”. The network doesn’t name anything, so that’s only our reading, and a rough one. Phone pushes this neuron up too. Its weight into the output is −4, so whatever it gives takes marks away.
Picture 2 runs one student through every amount of sleep from 4 to 10 hours, with study and phone held still.
Do the four tries in Picture 2 before reading on, one of them a check.
Now take students 1, 2 and 3, who differ only in sleep.
- Student 3 slept 10 hours. Neuron 2’s total is 6, and −4 × 6 = −24. Marks come off. Reasonable.
- Student 2 slept 7 hours. The total is 0. Nothing comes off.
- Student 1 slept 4 hours. The total is −6. Without the quiet rule, −4 × −6 = +24, so too little sleep would earn 24 extra marks.
That’s the problem. A penalty for too much sleep should do nothing at all when there isn’t too much sleep. The quiet rule makes that happen. Student 1’s −6 becomes 0, and no marks change.
With the rule on, the answers are 46, 58, 46, which rises and then falls. With it off, they are 70, 58, 46, which only ever falls.
Here’s why. Each extra hour of sleep adds 4 × 1 = 4 through neuron 1. Past 7 hours, neuron 2 is awake, and each extra hour also adds 2 × −4 = −8 through it. So marks climb by 4 an hour up to 7 hours, then fall by 4 an hour. That change of direction is called a bend. With the rule off, neuron 2 is at work at every hour, so there is no bend and marks only ever fall.
So without the rule, this network can’t say “some sleep is best”, whatever you set its knobs to. Lesson 6 proves that.
Why have a bias at all?
The goal in Picture 2 showed one reason: with the weights held still, neuron 2’s bias moves where it wakes up. Moving it from −16 to −18 moved the bend from 7 hours to 8, and no weight changed. Weights can move it too, as lesson 7 shows.
Look at it from the other side. Neuron 2’s weights are 0 for study, 2 for sleep and 1 for phone. For all twelve students, that sum comes out positive. Without a bias, this neuron would react to every student, only more or less strongly. Across these students, there would be no change from quiet to awake.
A bias is where the neuron’s number starts, before any input arrives. Neuron 2 starts at −16, so its inputs must add 16 just to reach zero. So zero works as a bar to clear. Take student 2, who slept 7 hours with 2 on the phone. The sum is 14 + 2 = 16, and 16 − 16 = 0. Exactly at the bar.
The output neuron starts at +20, so its answer is 20 before anything arrives.
Picture 3 lets you feel the difference, on one neuron and all twelve students.
Do the four tries in Picture 3 before reading on, the last of them a check.
So a weight says how much each input counts. A bias says where the neuron starts.
Changing a bias moves every student’s total by the same amount. Raising a weight by one moves each total by that student’s hours on that input. Together they let a neuron pick out one kind of student. That’s why a neuron is often described as a detector. Our students’ hours are whole numbers. One way to make that detector for “studied 6 hours or more” is a weight of 1 on study, 0 on sleep and phone, and a bias of −5. It isn’t only on or off: student 8, on 6 hours, gets 1, and student 12, on 8 hours, gets 3.
The whole network is this recipe, repeated
Here is student 5 through all three neurons.
| Neuron | Weighted sum | Bias | Total | Gives |
|---|---|---|---|---|
| Neuron 1 | 14 + 32 − 12 = 34 | −12 | 22 | 22 |
| Neuron 2 | 0 + 16 + 4 = 20 | −16 | 4 | 4 |
| Output | 22 − 16 = 6 | +20 | 26 | 26 |
The output neuron’s inputs are what the two hidden neurons gave, 22 and 4. Multiplied by their weights, they are 1 × 22 and −4 × 4.
It does the same sum, with one difference: it skips the quiet rule. We chose its total as our prediction of marks, so we leave it as it is, even below zero. That’s our design choice for this network; having no next neuron does not force that choice. Lesson 7 looks at other choices.
Nothing else happens anywhere in this network. Eight weights and three biases make eleven knobs.
The code
This is the whole recipe as Python. Each line is something you did in the pictures.
def total(weights, bias, inputs):
t = bias
for w, x in zip(weights, inputs):
t += w * x
return t
def quiet(t):
return max(0, t)
student_5 = [2, 8, 4]
w1 = [7, 4, -3]
w2 = [0, 2, 1]
h1 = quiet(total(w1, -12, student_5))
h2 = quiet(total(w2, -16, student_5))
answer = total([1, -4], 20, [h1, h2])
print(h1, h2, answer) # 22 4 26
student_1 = [4, 4, 2]
n1 = quiet(total(w1, -12, student_1))
t = total(w2, -16, student_1)
print(t, quiet(t)) # -6 0
print(total([1, -4], 20, [n1, t])) # 70
print(total([1, -4], 20, [n1, quiet(t)])) # 46Here is how to read it, even if you have never coded.
def total(weights, bias, inputs):starts a recipe calledtotal. It takes a neuron’s weights, its bias and the numbers arriving.t = biasstarts the total at the bias, the number the neuron begins with. The loop then goes through each weight and input together, andt += w * xadds each product in. That is the multiply and the add of Picture 1, with the bias in too.quietis the quiet rule.max(0, t)picks the bigger of 0 and the total, so a total below zero becomes 0.h1andh2are the two hidden neurons for student 5, using neuron 1’s weights and bias and neuron 2’s. Theanswerline runs the output neuron on what they gave. Its comment is the table you saw: 22, 4 and 26.- The last five lines are Picture 2.
n1is neuron 1’s answer for student 1, which is 26 and is not student 5’s answer. Student 1 has a total of −6 at neuron 2. With the rule,quiet(t)is 0 and the answer is 46. Without it, −6 goes straight into the output, and the answer is 70.
In maths books the total is written total = w₁x₁ + w₂x₂ + w₃x₃ + b, where each x is an input and each w is its weight. With one input it shrinks to the y = wx + b of school.
What we simplified
- The weights and biases here were set by hand. Lesson 5 shows a network finding its own.
- “Too much sleep” for neuron 2 is our own rough reading, because phone pushes it up too. The network has no names for its neurons.
- We showed that this network can’t say “some sleep is best” with the rule off, on one student. Lesson 6 proves it for any weights.
- The quiet rule is one rule among several, and the output skipping it is our design choice. Lesson 7 compares the others.
What’s next
Now you can work out the network’s answer for any student by hand. It said 26 for student 5, who scored 24. That’s off by 2 for one student.
How wrong is the whole network?
Answer them all, then check. Retry as often as you like.
1.A neuron has inputs 2 and 3, weights 4 and −1, and a bias of −2. What does it give?
2.Same neuron, but now the bias is −9. What does it give?
3.Student 6 studied 3 hours, slept 5 and spent 1 on the phone. What does our network say for them?
4.Suppose a student's total at the output neuron came out as −3. What would the network answer?
5.A neuron has weights 1, 0, 0 and a bias of −5. For our whole-hour students, it only gives something to those who studied 6 hours or more. Change the bias to −6. Which students now get 0 from it?