Views: 18
Episode 00.01 landed on a working definition of intelligence built around selecting good actions under uncertainty, and generalizing efficiently. Every example we used to demonstrate that — the lookup table, the nearest-neighbor classifier — quietly only ever operated on numbers: coordinates, labels, distances.
A neural network, underneath everything, is a machine that does arithmetic on numbers. Every AI system that reasons about language — from a spam filter to an LLM-powered agent — has to solve one unglamorous problem first: turn words into numbers without losing what makes them meaningful. Most explanations skip straight to “embeddings” as if that were obvious. It isn’t, and seeing why the obvious approaches fail is what makes the real solution click.
Attempt one: just encode the characters
Text is already numeric at the hardware level — ASCII assigns each character a number. But this encodes symbols, not meaning: there’s no reason two similar words should have similar character codes, and no reason two numerically adjacent codes should mean anything alike. Character encoding solves storage. It doesn’t solve understanding.
Attempt two: one word, one slot
The next instinct is one-hot encoding: build a vocabulary, give each word a unique index, and represent it as a vector of zeros with a single 1 at that word’s position. It’s numeric, well-defined, and usable — and it has a specific, provable flaw: every pair of distinct words ends up exactly equally dissimilar. “King” and “queen” are numerically no more related than “king” and “banana.” This isn’t a rough edge — it’s a direct consequence of how the vectors are built. No vocabulary design fixes it.
Attempt three: let context define meaning
The idea that breaks the deadlock predates modern AI by decades. Linguist J.R. Firth put it in 1957: “you shall know a word by the company it keeps.” This is the distributional hypothesis — a word’s meaning can be approximated by the contexts it tends to appear in. “King” and “queen” show up near “rules” and “kingdom.” “Apple” and “orange” show up near “eats” and “fruit.” Words that share contexts probably share meaning, even to a machine with no concept of meaning at all.
This single idea is the seed of every word embedding technique that follows — co-occurrence matrices, word2vec, GloVe, and eventually the learned representations inside a transformer.
Proving it, not just claiming it
Cosine similarity is the standard way to measure how aligned two vectors are. For any two distinct one-hot vectors, the math forces their similarity to exactly zero, every time — not approximately, exactly, because their nonzero positions never overlap by construction. It’s a clean, provable limitation, not a fuzzy intuition.
Co-occurrence vectors break that constraint. Build one small vector per word, counting how often other words appear nearby across a corpus, and words used in similar contexts start sharing nonzero positions — which means their cosine similarity becomes a real, graded number. In a tiny eight-sentence test corpus, “king” and “queen” came out 75% similar by this measure, while “king” and “apple” came out 41% similar. Nobody told the system that kings and queens are both royalty — it inferred a meaningful distinction purely from surrounding words.
What’s still missing
Real vocabularies run to hundreds of thousands of words, and a co-occurrence vector is exactly that long per word — mostly zeros, hugely wasteful. We’ve solved the similarity problem but not the size problem. That compression — from huge, sparse co-occurrence vectors down to small, dense embeddings — is exactly where word2vec and its successors come in, and it’s the subject of the next episode.
The full worked example, with all the runnable code, is on GitHub.
Leave a Reply