-
Loss Masking Explained: Why Fine-Tuning Shouldn’t Train on the Prompt (Ep:05.05)
Instruction fine-tuning data is a (prompt, response) pair, but training loss should only ever be computed on the response tokens. Module 5 of From Zero to Agents derives loss masking…
-
Does LoRA Actually Prevent Catastrophic Forgetting? The Real Answer, Measured (Ep:05.04)
The common claim is that LoRA reduces catastrophic forgetting. Measured directly, an attached LoRA adapter can forget just as much as full fine-tuning — the real benefit is something more…
-
QLoRA Explained: Why Quantizing the Frozen Base Isn’t Quite Free (Ep:05.03)
QLoRA shrinks a frozen model’s memory footprint by storing it in 4 bits instead of 16 — but quantization noise isn’t low-rank, so a LoRA adapter sized for the task…
-
Where to Apply LoRA in a Transformer: Allocating a Fixed Rank Budget Optimally (Ep:05.02)
For a fixed total parameter budget, splitting LoRA rank equally across weight matrices is usually wrong — some matrices need far more adaptation capacity than others. Module 5 of From…
-
Choosing LoRA’s Rank: The Eckart-Young Theorem Predicts Your Exact Loss Floor (Ep:05.01)
A theorem deferred all the way back in Module 0 turns out to predict, exactly, how much loss LoRA training will plateau at for any chosen rank. Module 5 of…
-
LoRA — Adapting a Model Without Touching Most of Its Weights (Ep:05.00)
LoRA fine-tunes a 4096×4096 layer with 256x fewer trainable parameters by freezing the original weights and learning a low-rank update instead. Module 5 of From Zero to Agents derives why…
-
Building a Tiny GPT From Scratch: Causal Masking and a Complete Working Model (Ep:04.05)
The only architectural difference between an encoder and a GPT-style decoder is one additive mask before softmax. Module 4 of From Zero to Agents derives causal masking, verifies it against…
-
Positional Encoding Explained: How Transformers Learn Word Order (Ep:04.04)
Attention is mathematically blind to sequence order — swap two tokens and every output is just the same values, permuted. Module 4 of From Zero to Agents derives sinusoidal positional…
-
The Transformer Block Explained: Multi-Head Attention, Residuals, and Layer Norm (Ep:04.03)
Residual connections fix vanishing gradients across depth the same way LSTM gating fixed them across time — both give the gradient a near-identity path. Module 4 of From Zero to…
-
Learned Q, K, V Explained: What Makes Attention Actually Trainable (Ep:04.02)
Attention only becomes powerful once the query, key, and value projections are learned rather than reused from static embeddings. Module 4 of From Zero to Agents trains a real attention…
-
How LSTMs Fix the Vanishing Gradient Problem: The Gating Math, Derived and Measured (Ep:04.01)
An LSTM’s cell state has a gradient path that doesn’t pass through a matrix multiply or a squashing nonlinearity at every timestep — that’s the actual mechanism that fixes vanishing…
-
RNNs and the Vanishing Gradient Problem: Why Attention Had to Be Invented (Ep:04.00)
An RNN reuses the same weight matrix at every timestep, which means the vanishing gradient problem from Module 3 comes back — across time instead of depth, and often much…
