Build an LLM From Scratch

Lesson 43

Running it cheaply

Build an LLM From Scratch, 53 lessons

Round stored weights, make room for separate chat caches, and let waiting conversations join the next decode pass.

Your model can now produce answers people prefer. The next problem is practical: will it fit on your machine, and what happens when someone else asks a question while it is still answering you?

A local chat server makes both problems concrete. It keeps a model loaded and accepts requests from a chat app. Restarting and loading the model for each message would repeat work that the server can keep ready.

BREAF Premium

The rest of this lesson is in Premium

Premium includes every lesson, with its notes, practice and progress.