Lesson 43
Running it cheaply
Build an LLM From Scratch, 53 lessons
Lesson 43: Running it cheaply
Build an LLM From Scratch, Lesson 43 of 53

Round stored weights, make room for separate chat caches, and let waiting conversations join the next decode pass.
Your model can now produce answers people prefer. The next problem is practical: will it fit on your machine, and what happens when someone else asks a question while it is still answering you?
A local chat server makes both problems concrete. It keeps a model loaded and accepts requests from a chat app. Restarting and loading the model for each message would repeat work that the server can keep ready.
BREAF Premium
The rest of this lesson is in Premium
Premium includes every lesson, with its notes, practice and progress.