Views: 1
RNNs, LSTMs, attention, transformer blocks, positional encoding, causal masking — six ideas that each fixed a specific, provable failure in the one before it, ending in a complete working language model.
Every “transformers explained” piece eventually shows the same diagram — boxes, arrows, the words “attention is all you need.” Almost none of them explain what came before, or why each specific piece exists. The actual story is a chain of six ideas, each one a direct, provable fix for a specific failure in the idea before it — and following that chain once turns the transformer from a diagram to memorize into something derivable.
The problem: language doesn’t come in fixed sizes
Every network up to this point in a from-scratch AI curriculum takes a fixed-size input and produces a fixed-size output. A sentence might be three words or three hundred. The first serious attempt at bridging this gap was the recurrent neural network: process one token at a time, carry a “hidden state” forward as a summary of everything seen so far, and reuse the same small set of weights at every step.
The problem RNNs ran into
That hidden state, in principle, could carry information arbitrarily far — token 1 influencing how token 200 gets processed. In practice, it almost never does. Because the same weight matrix gets applied once per timestep, the gradient connecting an early token to the final loss has to pass through a long chain of repeated matrix multiplications and squashing nonlinearities — and that chain shrinks geometrically with sequence length. Measured directly: across just 50 timesteps, the gradient reaching the earliest token can be roughly a quadrillion times smaller than the gradient at the most recent one — computationally zero. A vanilla RNN, in practice, cannot learn dependencies spanning more than a few dozen steps.
The gated fix, and its honest limits
LSTMs (1997) addressed this with a second, separate memory path — a “cell state” updated mostly through addition rather than full rewriting, controlled by learned gates. The gradient flowing through that specific path doesn’t have to pass through a matrix multiply and a squashing function at every step; it’s multiplied by something the network can learn to keep close to 1. Measured directly against a vanilla RNN at the same sequence length: roughly five orders of magnitude better gradient preservation. Not a complete fix — the LSTM’s gradient still shrinks with length, just far more slowly — but enough to make LSTMs the dominant architecture for sequence modeling for nearly two decades.
The mechanism that sidesteps the problem entirely
Attention takes a different approach: instead of a chain of timesteps, connect any two positions with one direct computation — a similarity score between a query and a key, computed for every pair simultaneously in a single matrix operation. No sequential chain means no chain-length gradient decay at all. But the earliest version of this idea, built from raw embeddings with no learned components, had a real limitation: it could only detect semantic similarity, because that’s what the embeddings underneath it were trained for. Making the query, key, and value projections genuinely learnable — trained via ordinary backpropagation like any other layer — lets a network discover whatever notion of “relevant” actually matters for its task. Demonstrated directly: a task specifically designed so raw embedding similarity provides zero useful signal, solved perfectly after training, with attention weights shifting from near-uniform before training to laser-focused on exactly the right position afterward.
Two more pieces, and a genuinely unifying discovery
Multiple attention heads, each with independent learned projections, let a network track several different kinds of relationships in parallel rather than compromising between them in one shared computation. And stacking many of these blocks deep introduces the vanishing-gradient problem all over again — this time across network depth rather than sequence length. The fix, residual connections (add a layer’s original input back onto its output), turns out to be mathematically the identical trick LSTM gating used: a guaranteed additive path for the gradient that doesn’t depend on how well any individual layer’s own gradient behaves. Measured at 40 layers deep: a stack without residual connections has its gradient shrink to roughly a trillionth of its original size; the identical stack with residual connections stays healthy and stable. Two problems — across time, across depth — solved by the same underlying idea, discovered independently, 18 years apart.
The two pieces that make it actually usable
Attention computes relationships from content alone — swap the order of two input tokens, and every output is provably just the same values, permuted. A model built this way genuinely cannot distinguish “the cat sat” from “sat the cat.” The fix isn’t architectural — it’s informational: inject a unique, mathematically well-behaved signal for each position directly into the input itself, using sine and cosine waves at different frequencies, before attention ever runs.
And for a model that generates text one token at a time, there’s one more constraint: a position predicting the next token can’t be allowed to see tokens that haven’t been generated yet. The fix is almost absurdly small — one additive term, forcing every “attend to the future” score to negative infinity before the softmax step, which guarantees exactly zero probability on those positions after normalization. This single addition is the entire architectural difference between a model built to understand a complete sequence and one built to generate a sequence — not two different mechanisms, one formula with one extra term.
Where it lands
Put all six pieces together — learned attention, multi-head parallelism, residual connections, layer normalization, positional encoding, causal masking — and the result is a complete, trainable, generative architecture. Built at genuinely tiny scale and trained on nothing more than a repeating three-character pattern, a model assembled from exactly these pieces trains to near-zero loss and, seeded with a single character, correctly generates the entire pattern on its own, one token predicted and fed back in at a time — the same generation process real language models use, mechanism for mechanism, just at a scale small enough to fully see and verify every step.
None of these six ideas were arranged from a top-down blueprint. Each is a direct, derivable patch for a specific, measured failure in what came immediately before it — which is exactly why understanding the failures, in order, turns out to be a faster route to real understanding than memorizing the final diagram.
Leave a Reply