From Scratch to Agents — Build a Real LLM, Fine-Tune It, Add RAG, Make It an Agent (Ep:00.00)

Views: 3

A hands-on, code-only course: build a real GPT-style LLM from scratch on Google Colab’s free tier, pretrain it on real text, fine-tune it with LoRA, add RAG, and turn it into a tool-using agent. Every step in raw Python, PyTorch, and production libraries.

By the end of this series, you will have personally built, trained, and run:

• A real GPT-style language model, written from scratch, trained on real text, that generates actual coherent-ish text — not a toy that memorizes four sentences.
• A LoRA fine-tuning pipeline that adapts that model to a specific task or style.
• A working RAG system that lets your model answer questions using documents it was never trained on.
• An agent — built on top of everything above — that can reason, call tools, and complete multi-step tasks.

And you will have done every single piece of it three ways: hand-written in raw Python so you understand the actual mechanism, rebuilt in PyTorch so you know how it’s really done, and finally matched against the production library the field actually uses day to day — Hugging Face transformers/PEFT, LangChain or LlamaIndex, and a real agent framework. Raw Python teaches you why. PyTorch teaches you how it scales. The production library teaches you what to reach for on the job.

Every episode runs entirely on Google Colab’s free tier. No paid compute, no local GPU required. If you have a browser, you can follow every line of this course and reproduce every result yourself.

1. Why this course is different from the last one

If you followed From Zero to Agents, this is its practical sequel — but it’s built on a different philosophy on purpose.

That course started from zero and built up every mathematical foundation by hand, using small toy problems specifically because they made the underlying mechanism undeniable — you could see a gradient vanish, watch a matrix multiply, verify a claim against a hand-computed number. Theory first, small and cheap, so nothing was hidden.

This course assumes you already have that foundation (or are comfortable picking up what you need as it comes up) and flips the emphasis: almost no theory, almost all code. The model we train here isn’t a toy proving a mechanism — it’s a small-but-real language model, trained on real text, that actually writes something worth reading. You’re not here to prove attention works. You’re here to build the thing.

2. The three-layer build, every time

Every major component in this course gets implemented three times, back to back:

1. Raw Python. No PyTorch, no autograd, no shortcuts. If it’s a matrix multiply, you write the loops. This is where you actually understand what’s happening, because there’s nowhere for the mechanism to hide.
2. PyTorch. The same component, rebuilt using tensors, autograd, and nn.Module — the way it’s actually written in practice, at a scale raw Python loops can’t reach.
3. The production library. Where one exists for the topic at hand — Hugging Face transformers and PEFT for the model and fine-tuning, LangChain or LlamaIndex for RAG, a real agent framework for the agent module — we rebuild it one more time using the tool the industry actually uses, and compare it directly against what we just built by hand.

You’ll see, repeatedly, that the production library isn’t magic — it’s the same mechanism you just wrote, wrapped in better engineering.

3. Visuals — because a training run is genuinely hard to picture

Practical, code-heavy content has a specific failure mode: it’s easy to run a cell, see a number change, and not actually picture what just happened inside the model. This course fights that two ways:

• Result visuals, generated by the code itself — loss curves, attention heatmaps, embedding projections, generated-text samples over training time. If the code produces a number, we plot it.
• Concept visuals — separately produced animated diagrams for the architectural and data-flow ideas that are genuinely hard to hold in your head from code alone (how a batch actually moves through a transformer block, what a LoRA update looks like sitting next to a frozen weight matrix, how a RAG pipeline routes a query).

Both are woven directly into each episode, not bolted on as an afterthought.

4. What you need to follow along

• A Google account (Colab free tier — that’s it, no paid compute at any point in this course)
• Comfort reading Python; you don’t need to be an expert, but you shouldn’t be seeing a for loop for the first time
• Whatever foundation you have in the underlying math — if you want the from-first-principles version of that foundation, that’s what From Zero to Agents is for, and this course will point back to it where it’s genuinely useful, without requiring it

5. The road ahead

• Module 00 — Setup & Real Data: Colab environment, choosing and preparing a real training dataset, building a tokenizer from scratch and comparing it against production tokenizers.
• Module 01 — Building the LLM From Scratch: embeddings through a complete GPT architecture, three ways.
• Module 02 — Pretraining on Real Data: the real training loop, checkpointing, and watching the model actually learn to write.
• Module 03 — Fine-Tuning: instruction tuning and LoRA, three ways.
• Module 04 — RAG: retrieval-augmented generation, three ways.
• Module 05 — Agentic AI: tool use, reasoning loops, and a working agent built on everything before it.

6. Where this leaves us

Next episode: getting your Colab environment set up exactly the way this course needs it, picking the real dataset we’ll train on for the rest of the series, and understanding why that choice matters more than most people expect.

Next: Episode 00.01 — Colab Setup and Choosing Our Training Data