Views: 1
A single artificial neuron can’t solve XOR. Here’s the full, provable chain of fixes — nonlinearity, activation functions, initialization, backpropagation, batching, and the right loss function — that turns that failure into a real trainable network.
“Neural networks learn from data” is true and explains almost nothing. The actual mechanism is a specific, derivable chain of fixes, each one addressing a concrete failure in the idea before it — the same pattern that shaped word embeddings and attention, now playing out at the level of the network itself. Follow the chain once and “how does a neural network learn” stops being a black box.
The failure that started it all
In 1958, Frank Rosenblatt’s perceptron — the simplest possible artificial neuron, a weighted sum passed through a threshold — genuinely worked for a range of problems. In 1969, Minsky and Papert proved, algebraically, that it could never solve XOR: output 1 if two inputs differ, 0 if they’re the same. Not a training difficulty — a hard mathematical impossibility, provable by contradiction from the function’s own definition. A single neuron draws exactly one straight decision boundary; XOR’s positive cases sit on opposite corners of a square, unreachable by any single line. That result contributed directly to a decade-plus collapse in neural network research funding, now known as the first AI winter.
The fix, and the theorem hiding inside it
The fix looks almost too simple: combine multiple perceptrons in layers. One hidden layer computing OR and NAND, feeding a final layer computing AND of those two results, reproduces XOR exactly — three simple, individually-solvable pieces combining into something none of them could do alone. But there’s a subtlety easy to miss: stack layers without a genuine nonlinearity between them, and the whole stack collapses algebraically into a single linear layer, no matter how many layers deep — provable directly from matrix algebra, and confirmed by generating random weights and checking that two “layers” without activation functions produce output identical to one combined layer. The nonlinear activation function isn’t an implementation detail. It’s the entire reason depth adds any expressive power at all.
The activation function that almost broke training anyway
Early networks used sigmoid — smooth, differentiable, seemingly a reasonable choice. It has a specific, quantifiable flaw: its derivative maxes out at exactly 0.25, and shrinks rapidly for any input far from zero. Backpropagation multiplies these derivatives together across every layer on the way back to early weights — and multiplying many numbers under 0.25 together shrinks the product geometrically with depth. Measured directly in a real 15-layer network: sigmoid’s gradient reaching the layer nearest the input is roughly 5×10⁻¹⁴ times smaller than at the output — computationally zero, meaning that layer receives no usable training signal at all. Swapping to ReLU — whose derivative is a clean 1 for any positive input, no saturation — preserves the gradient roughly nine orders of magnitude better in the identical architecture. This single substitution is a large part of why ReLU-family functions became the default for hidden layers industry-wide.
The starting point matters just as much as the mechanism
Even with the right activation function, a network’s initial weights turn out to matter enormously. Identical starting weights across a layer create a symmetry problem: every neuron computes the same thing, receives the same gradient, and stays identical forever — a layer with 100 neurons behaving exactly like a layer with 1, no matter how long training runs. And the scale of random initial weights matters just as much: deriving the variance of a weighted sum shows that keeping a signal’s scale stable across layers requires the initial weight variance to shrink in proportion to how many inputs feed into each neuron. Get this wrong in either direction, and a 15-layer network’s activations either collapse to numerically zero by layer five, or explode past 200 million by layer fifteen — measured directly, using nothing but a different starting scale on an otherwise identical network.
The actual learning algorithm
Backpropagation itself is four equations, not a black box: compute how wrong the final output is, then propagate that “blame” backward one layer at a time, using each layer’s own weights — transposed, since blame flows in the reverse direction of the original computation — combined with each layer’s local activation derivative. Implementing this from raw arithmetic, with no automatic differentiation library involved anywhere, and comparing every single computed gradient against PyTorch’s autograd on identical weights, produces an exact match — not approximately, exactly, to eight decimal places. Autograd isn’t doing anything more sophisticated than this; it’s automating exactly this, at scale.
Processing many examples at once requires no new math
Real training never processes one example at a time — it batches many examples together. It turns out this needs no new mathematical machinery at all: replace every vector in the backpropagation equations with a matrix (one column per example), and ordinary matrix multiplication automatically sums the right quantities across the batch, as a direct, provable consequence of how matrix multiplication is defined. Verified directly: computing gradients in a single batched matrix operation produces results identical, to the decimal, to computing each example’s gradient in a loop and averaging by hand.
The loss function that turned out to matter more than expected
The final piece is easy to underestimate: does the specific loss function used to measure “how wrong” actually matter, beyond convention? Comparing squared-error loss against cross-entropy loss in the exact scenario where it counts — a network that’s confidently, badly wrong — reveals a real, measured pathology: squared error’s gradient, paired with a saturating activation, shrinks to nearly nothing exactly when the correction needed is largest. Cross-entropy’s gradient, in the identical scenario, comes out over 400 times stronger — because a specific cancellation in the calculus removes the saturating term entirely. This is a provable, quantified reason cross-entropy became the standard for classification and language modeling, not an arbitrary convention.
The throughline
Every fix in this chain was forced by a specific, demonstrable failure in what came before it — a single neuron’s ceiling, a saturating derivative, a badly-scaled starting point, an unautomated gradient computation, an unbatched training loop, a loss function that goes quiet exactly when it should be loudest. None of it was designed top-down. Each piece is a patch for something that provably broke, verified against a working implementation at every step — which is exactly why a network built this way, piece by piece, from nothing but arithmetic, ends up doing something that looks, from the outside, like learning.
Leave a Reply