Why Your Training Loss Exploded to NaN: The Math of Learning Rate, Not Guesswork

Views: 1

A loss curve that spikes to NaN mid-training isn’t random instability — it’s a provable consequence of crossing an exact convergence threshold. Here’s the derivation, and what it means for setting a learning rate.

Every engineer who’s trained a model has seen it: the loss curve is dropping nicely, and then — a spike, a NaN, a run that has to be killed and restarted with a smaller learning rate. The usual explanation is “instability,” treated as something to route around with trial and error rather than something with an actual cause. It has an actual cause, and it’s provable, not mysterious.

The single number that decides convergence or divergence

Gradient descent updates every trainable parameter by the same basic rule: take the current value, subtract the learning rate times the gradient. For the simplest possible loss shape — a smooth single-minimum bowl, the same basic shape a squared-error loss has near its optimum — that update rule reduces to a geometric sequence: each step multiplies the current distance from the minimum by a fixed ratio determined by the learning rate.

Geometric sequences have an exact, well-known rule: they shrink toward zero if that ratio’s absolute value is less than 1, and they grow without bound if it’s greater than 1. Applied to gradient descent, this produces an exact threshold — not a rule of thumb, a specific derivable number — separating “this will converge” from “this will diverge.” Below the threshold, training settles toward the minimum, however slowly. At or above it, the loss doesn’t just fail to improve — it oscillates in sign and grows every single step, exploding outward from whatever value it started at.

What this looks like in practice

Verified directly: a learning rate comfortably below the threshold converges steadily. One just above the midpoint of the safe range still converges, but with a visible oscillation — bouncing from one side of the minimum to the other while gradually shrinking, which is exactly the “noisy but improving” loss curve pattern many practitioners have seen and shrugged off as normal variance. A learning rate past the threshold does something qualitatively different: the tracked value doesn’t decay at all — it grows, unboundedly, past the threshold. In a real network, where the loss surface is far more complex than a single bowl but the underlying mechanism is identical, this is exactly the signature of a loss curve that spikes toward NaN: not random bad luck, a direct consequence of the update step overshooting far enough, repeatedly, that each correction makes the error larger rather than smaller.

The practical takeaway

A learning rate that’s “too high” isn’t a fuzzy, relative judgment — it’s crossing a specific, calculable line past which the mathematics of the update rule guarantees things get worse, not better, with every step. This is exactly why learning-rate warmup (starting small and ramping up) and learning-rate scheduling (shrinking it over time) are standard practice rather than superstition: early training, when gradients tend to be largest and least reliable, is precisely when the risk of crossing that threshold is highest. When a training run explodes, the fix that actually addresses the cause is checking whether the effective step size crossed the convergence boundary for the loss landscape currently being explored — not just picking a smaller number and hoping.