-
A Complete LoRA Fine-Tuning Pipeline, End to End: Every Module 5 Concept in One Run (Ep:05.08)
Pretrain a base model on one task, freeze it, LoRA-adapt it to a second task with proper loss masking, then measure new-task accuracy, old-task interference, and exact recoverability — every…
-
Perplexity Can Lie: Why Which Tokens You Evaluate On Changes the Answer (Ep:05.07)
Full-sequence perplexity makes an unmasked-trained model look better than a masked-trained one — the opposite of the truth. Module 5 of From Zero to Agents derives perplexity from cross-entropy, then…
-
Loss Masking, Properly Tested: Why the First Experiment Failed and the Second One Worked (Ep:05.06)
A properly designed experiment — more data, a task requiring genuine rule-learning, a real held-out test set — finally shows loss masking’s real benefit clearly, and reveals a precise mechanistic…
-
Loss Masking Explained: Why Fine-Tuning Shouldn’t Train on the Prompt (Ep:05.05)
Instruction fine-tuning data is a (prompt, response) pair, but training loss should only ever be computed on the response tokens. Module 5 of From Zero to Agents derives loss masking…
-
Does LoRA Actually Prevent Catastrophic Forgetting? The Real Answer, Measured (Ep:05.04)
The common claim is that LoRA reduces catastrophic forgetting. Measured directly, an attached LoRA adapter can forget just as much as full fine-tuning — the real benefit is something more…
-
QLoRA Explained: Why Quantizing the Frozen Base Isn’t Quite Free (Ep:05.03)
QLoRA shrinks a frozen model’s memory footprint by storing it in 4 bits instead of 16 — but quantization noise isn’t low-rank, so a LoRA adapter sized for the task…
-
Where to Apply LoRA in a Transformer: Allocating a Fixed Rank Budget Optimally (Ep:05.02)
For a fixed total parameter budget, splitting LoRA rank equally across weight matrices is usually wrong — some matrices need far more adaptation capacity than others. Module 5 of From…
-
Choosing LoRA’s Rank: The Eckart-Young Theorem Predicts Your Exact Loss Floor (Ep:05.01)
A theorem deferred all the way back in Module 0 turns out to predict, exactly, how much loss LoRA training will plateau at for any chosen rank. Module 5 of…
-
LoRA — Adapting a Model Without Touching Most of Its Weights (Ep:05.00)
LoRA fine-tunes a 4096×4096 layer with 256x fewer trainable parameters by freezing the original weights and learning a low-rank update instead. Module 5 of From Zero to Agents derives why…
-
The One-Line Difference Between an Encoder and a GPT-Style Decoder
BERT and GPT are often described as fundamentally different architectures. The actual difference in the attention computation itself is a single additive term. Here’s what that term does, and why…
-
How Attention Replaced Recurrence: The Full Story, From RNNs to a Working GPT
RNNs, LSTMs, attention, transformer blocks, positional encoding, causal masking — six ideas that each fixed a specific, provable failure in the one before it, ending in a complete working language…
-
Building a Tiny GPT From Scratch: Causal Masking and a Complete Working Model (Ep:04.05)
The only architectural difference between an encoder and a GPT-style decoder is one additive mask before softmax. Module 4 of From Zero to Agents derives causal masking, verifies it against…
