Views: 4
A classification model that’s confidently wrong early in training can get an almost nonexistent correction signal if you’re using the wrong loss function. Here’s the exact mechanism, measured, and why cross-entropy fixes it.
A classification model sits stubbornly at high loss for the first stretch of training, barely improving epoch over epoch, before eventually — sometimes much later than expected — starting to learn properly. The usual response is patience, or a learning-rate tweak. Worth checking first: what loss function is actually being used, and whether it’s quietly sabotaging the model at the exact moment it needs help most.
The mismatch hiding in a very common setup
Using a sigmoid output activation together with mean-squared-error loss is a completely reasonable-looking choice for a classification task — treat the 0/1 label as a target to hit exactly, same as any regression problem. It’s also, for classification specifically, a setup with a real, provable flaw.
Sigmoid saturates: for inputs far from zero in either direction, its output flattens and its derivative shrinks toward zero. That’s a known property. The problem is when it shows up. A model that’s confidently, badly wrong — predicting close to 0 for something that should be 1 — has exactly the kind of extreme pre-activation value that triggers this saturation. And the gradient that trains the output layer under MSE is the raw error multiplied directly by that same shrinking derivative. The result: the correction signal gets weaker exactly when the model is most wrong and needs the strongest possible push in the right direction.
How large the gap actually is
Measured directly, in a scenario where a model predicts only a 0.25% probability for the correct class (confidently, badly wrong): the MSE+sigmoid correction signal comes out to roughly 0.0025 — barely distinguishable from zero, as if the model were only slightly off. Cross-entropy loss paired with a softmax output, in the identical scenario, produces a correction signal of roughly 0.998 — over 400 times stronger.
That gap isn’t a rounding difference. It’s the direct, provable consequence of a specific cancellation in the calculus: when softmax feeds directly into cross-entropy, the saturating derivative term cancels out of the gradient entirely, leaving a correction signal that’s simply, linearly proportional to how wrong the prediction is — no matter how extreme. MSE paired with sigmoid has no such cancellation; the saturation stays baked into every gradient.
Why this shows up specifically as “slow to start”
Early in training, weights are close to their random initialization, and it’s common for a model to be confidently wrong about at least some training examples purely by chance. Under MSE+sigmoid, exactly those examples — the ones that should be driving the largest, most useful updates — contribute almost nothing to the gradient. The model effectively can’t hear its own biggest mistakes. As training continues and predictions drift toward less extreme values, the saturation eases and normal-looking learning eventually kicks in — which is exactly the “stubbornly flat, then suddenly improving” pattern this failure mode tends to produce.
The practical takeaway
For a classification task, this isn’t a stylistic preference between loss functions — it’s a measurable, several-hundred-fold difference in how strongly a model can correct its worst mistakes. If a classifier is training suspiciously slowly at the start, particularly one using a sigmoid output activation with squared-error loss, switching to a softmax output layer with cross-entropy loss addresses the actual mechanism, not just the symptom.
Leave a Reply