The Math Behind Every LLM, in Five Steps: Vectors to Cross-Entropy Loss

Views: 2

One continuous derivation connects vectors, matrices, gradients, and probability into the exact training objective behind every modern language model. Here’s the full arc, with the reasoning that forces each step.

Most explanations of “the math behind transformers” hand you a wall of notation and hope familiarity substitutes for understanding. It’s more useful to build the notation in the order it becomes necessary — each concept forced into existence by a specific, concrete gap the previous one left open. Follow that order once, and equations that look intimidating in a paper stop being a wall and start being a checklist you already know.

Here’s the full arc, five steps, each one forced by the last.

Step 1: Vectors — the objects everything else operates on

An embedding, a set of attention scores, a gradient — every one of them is, underneath, just a list of numbers with two things attached: a magnitude and a direction. The dot product — multiply corresponding entries, sum the results — turns out to have a precise geometric meaning: it equals the product of both vectors’ lengths times the cosine of the angle between them. That single identity is the reason cosine similarity has meant anything at all, every time it’s been used to compare embeddings. And one-hot vectors, it turns out, aren’t just a convenient encoding trick — they’re literally the standard basis vectors of the vector space, which is exactly why they’re forced into perfect mutual orthogonality: standard basis vectors are the maximally spread-apart directions a space has to offer, by definition. Read More

Step 2: Matrices — organized, simultaneous vector operations

A single dot product answers one similarity question. Real systems need thousands of similarity questions answered at once — every word in a sentence, compared against every other word. Matrix multiplication is exactly that operation, formalized: the $i$-th output entry of a matrix-vector product is the dot product of the matrix’s $i$-th row with the vector. Stack that up to matrix-matrix multiplication, and the entire scaled dot-product attention formula from “Attention Is All You Need” — $\text{softmax}(QK^T/\sqrt{d_k})V$ — becomes fully readable: $Q$ and $K$ are matrices of per-word vectors, $QK^T$ computes every pairwise similarity score in one operation, and the softmax-then-multiply-by-$V$ sequence turns those scores into a weighted blend of every word’s information, for every word, simultaneously. One clean mathematical object replaces what would otherwise be a nested loop of individual comparisons. Read More

Step 3: Derivatives and gradients — the mechanism of improvement

None of this is useful if it can’t get better at a task. The derivative answers a narrow, specific question — if I nudge this one number slightly, how does the output change? — and the gradient collects that answer across every adjustable number in a system at once, into a single vector that points in the direction of steepest increase. Flip its sign, and you have a direction that reliably decreases whatever you’re trying to minimize. That’s the entire idea behind “training”: compute the gradient of how wrong the system currently is, and nudge every parameter slightly against it. Read More

Step 4: Iterating that process — and discovering it doesn’t always work

Repeating that nudge thousands of times is what training actually is — but repetition alone doesn’t guarantee success. For a simple enough loss shape, gradient descent’s convergence condition turns out to be an exact, derivable number, not a rule of thumb: too small a step size converges painfully slowly; too large a step size doesn’t just train worse, it provably diverges to infinity, oscillating further from the answer with every single step. This is the real mathematical reason a learning rate that’s set too high can make a model’s loss explode rather than improve — not a mysterious instability, a predictable consequence of a specific, calculable threshold being crossed. Read More

Step 5: Probability — turning scores into a distribution, and measuring wrongness

The last piece: a network’s raw output scores are just numbers — they don’t sum to anything meaningful, and some can be negative. Softmax fixes that, converting any vector of real numbers into a genuine probability distribution (non-negative, sums to exactly 1) while preserving which score was largest. Once a system outputs a real probability distribution, cross-entropy loss can measure precisely how wrong that distribution is relative to the true answer — and it collapses, in the language-modeling case, into something almost embarrassingly simple: just the negative log of the probability the model assigned to the token that was actually correct. Confident and right costs almost nothing; confident and wrong costs severely. Read More

Where the five steps land

Put together in one sentence, this is what happens at every single position, every single training step, inside every modern language model: matrices compute attention scores across the sequence, softmax turns those scores into a probability distribution over the vocabulary, cross-entropy measures how wrong that distribution was against the token that actually came next, and the gradient of that loss — computed via the calculus from step 3 — nudges every weight in the network slightly toward being less wrong. Repeated billions of times, across billions of parameters, that process is the entire mechanism behind how a language model learns to write.

None of these five ideas were introduced because a textbook says they belong in a syllabus. Each one exists because the idea before it left a specific, concrete gap — vectors needed an “operate on many at once” upgrade; that needed a “how do I improve this” mechanism; that needed a “does this process actually converge” guarantee; and the whole system needed a way to turn raw output into something a loss function can actually score. Seen in that order, the notation in a research paper stops being a wall of Greek letters and starts being a sequence of moves you already recognize.

Read more in our blog series: From Zero to Agents.