Gradient descent in one line → θ ← θ − η · ∇L(θ). The gradient ∇L points uphill; we step the opposite way, scaled by learning rate η. Every optimizer below is a smarter version of this update — adding memory (momentum) or per-parameter step sizes (adaptive).
// 01
The race · same surface, same start, six optimizers
SURFACE
A simple convex bowl — every optimizer reaches the bottom.
SHOW
CONTROLS
BASE LR η
0.050
SPEED
3
LOSS SURFACE · trajectories iter 0
LOSS vs ITER
STANDINGS
// 02
What each one adds
| OPTIMIZER | IDEA | UPDATE | WATCH OUT |
|---|---|---|---|
| SGD | step opposite the gradient | θ −= η·g | zig-zags in ravines, slow |
| Momentum | accumulate a velocity | v = μv − η·g; θ += v | can overshoot the minimum |
| NAG | momentum, but look ahead first | grad at θ+μv | slightly more stable than plain momentum |
| AdaGrad | per-param LR ∝ 1/√(past grad²) | G += g²; θ −= η·g/√G | LR decays to 0, stalls late |
| RMSProp | AdaGrad with a forgetting factor | E = ρE+(1−ρ)g² | needs LR tuning |
| Adam | momentum + RMSProp, bias-corrected | m, v moments → θ −= η·m̂/√v̂ | the default; sometimes generalizes worse than SGD |
Try this → Switch to RAVINE and run. Watch SGD bounce across the valley walls while Adam and Momentum glide down the floor. Then try ROSENBROCK — the classic hard case — and crank the LR until SGD diverges. This is exactly why Adam became the go-to default for deep nets.