From Words to Attention: The Full Story of How AI Learned to Represent Language (Summary)

Views: 6

A single throughline connects one-hot encoding, word2vec, fastText, and the attention mechanism inside every modern LLM — each one solving the exact problem the last one left behind. Here’s the whole arc, in order.

Every explanation of “how LLMs work” eventually says some version of: “the model turns words into vectors, and attention lets them interact.” True, and almost useless as an explanation, because it skips the part that actually matters — why each piece exists. Every major idea in this stack was invented to fix a specific, demonstrable failure in the idea before it. Seen in that order, the whole thing stops being a list of buzzwords and starts being one continuous argument, each step forced by the last.

Here’s the whole arc, start to finish.

Step 1: Text isn’t numbers, and the obvious fix doesn’t work

A neural network only ever does arithmetic. Text has to become numbers before anything else is possible. The first obvious move — assign each word a unique index, represent it as a vector of zeros with a single 1 in its slot (one-hot encoding) — is numerically valid and completely useless for meaning. It’s not an approximation problem; it’s a hard mathematical fact: two distinct one-hot vectors are always exactly, provably unrelated to each other, by construction. “King” and “queen” are precisely as dissimilar, numerically, as “king” and “banana.” No amount of clever engineering fixes this from inside the one-hot scheme — the flaw is structural.

Step 2: Meaning from company kept

The fix is a linguistics idea older than modern AI: J.R. Firth’s 1957 observation that “you shall know a word by the company it keeps.” Count which words tend to appear near which other words across a large body of text, and words used in similar contexts turn out to have measurably similar usage patterns — computable, gradable similarity, something one-hot encoding could never produce even in principle. This is the seed every subsequent technique grows from.

Step 3: Compressing statistics into something a network can actually use

Co-occurrence counts solve similarity but create a new problem: the vectors are as large as the vocabulary and mostly zeros. Two families of fix emerged. One compresses the existing statistics mathematically (matrix factorization via SVD) into a small, dense vector. The other — word2vec (Mikolov et al., 2013) — skips explicit counting altogether and trains a small network to predict nearby words, letting dense embeddings fall out as a side effect of getting good at that prediction task. Neither approach is ever told anything about grammar or meaning directly — and yet word2vec embeddings famously support arithmetic like king − man + woman ≈ queen, geometric structure that emerged purely because it was the most efficient way to solve the prediction problem the network was actually given.

Step 4: What about words the model has never seen?

Every technique so far still assigns exactly one vector per whole word — a genuinely new word, a typo, a rare inflection, gets nothing. fastText (Bojanowski et al., 2017) fixed this by representing a word as the sum of its character-fragment embeddings rather than one atomic vector. A word the model has never seen can still get a sensible vector at inference time, assembled from pieces it recognizes — real compositional generalization, not a special-cased fallback. Tokenization schemes like Byte-Pair Encoding solve the identical problem one level down, at the level of raw text-to-token conversion, guaranteeing no input text is ever fundamentally unrepresentable, no matter how unusual.

Step 5: The problem composition can’t solve — ambiguity

Every method up through fastText still shares one deep assumption: a word’s vector is fixed once and reused everywhere, regardless of the sentence it’s actually in. “Bank” gets an identical vector next to “river” as it does next to “loan” — necessarily a blurry compromise between every sense the word has ever been used in. This is a different failure than the ones before it: it’s not that the word is unrepresented, it’s that it’s represented wrong, confidently, with no error signal anywhere.

Step 6: Making the vector a function of the sentence, not the word

The structural fix — refined into modern form by ELMo (2018) and then decisively by the attention mechanism — is to stop treating “get a word’s vector” as a table lookup at all. Instead, compute each word’s vector fresh, by letting it mix in information from the specific words actually surrounding it in that specific sentence. Even the simplest possible version of this — blending a word’s own vector with a plain average of its neighbors, no training involved — already produces measurably different vectors for “bank” depending on whether the sentence is about finance or rivers. Attention is this same idea with one refinement: instead of weighting every neighboring word equally, it learns to weight each one by how relevant it actually is — computed as a similarity score between the word being contextualized and every candidate neighbor, turned into proper weights. Uniform averaging turns out to be attention’s own special case, for exactly the situations where nothing distinguishes the candidates.

The throughline

Look at the sequence as a whole and a single pattern repeats at every step: build the simplest thing that could possibly work, find the specific way it breaks, then fix exactly that break — no more. One-hot broke on similarity; co-occurrence fixed it and broke on size; dense embeddings fixed size and broke on unseen words; composition fixed unseen words and broke on ambiguity; attention is what fixes ambiguity. Nothing in this stack was designed top-down from a grand theory of language. Every piece is a patch for a specific, demonstrable, testable failure of the piece before it — which is exactly why understanding the failures is more valuable than memorizing the architectures. The architectures are just what the failures forced.

That’s also exactly where this arc is headed next: not a single attention calculation bolted onto fixed embeddings, but full transformer layers where every word contextualizes against every other word simultaneously, stacked, with everything — including the query/key/value weighting itself — learned from data rather than hand-built. The mechanism is already fully visible in miniature. What’s left is scale, and the machinery (gradients, backpropagation, learned parameters) to let it teach itself.