The whole loop in one line → predict → measure loss → compute ∂loss/∂weights → step weights downhill. Everything in the Optimizers and MLP modules is in service of minimizing this number. The loss is the only thing the network actually optimizes.
// 01
Regression losses · "how far off is my number?"
LOSS vs RESIDUAL (e = ŷ − y) drag to set the error
SHOW
RESIDUAL e
2.00
HUBER δ
1.0
The outlier story → Push the residual past ±3. MSE (squared) explodes — one bad outlier dominates the whole gradient, so MSE chases outliers hard. MAE (absolute) grows linearly — robust to outliers but has a non-smooth kink at 0. Huber is the compromise: quadratic near 0 (smooth), linear far out (robust). That δ slider is the switchover point.
// 02
Classification losses · "how confident & how right?"
LOSS vs PREDICTED PROBABILITY of the TRUE class drag p
SHOW
PROB p
0.60
FOCAL γ
2.0
Why cross-entropy dominates → Drag p toward 0 (model confidently wrong about the true class). Cross-entropy = −log(p) shoots to infinity — a brutal, unbounded punishment for confident mistakes, with a steep gradient that fixes them fast. Focal loss adds a (1−p)γ factor that down-weights easy, already-correct examples (high p) so training focuses on the hard ones — the trick that made dense object detection work.
// 03
Reference · which loss for which job
| LOSS | TASK | FORMULA | CHARACTER |
|---|---|---|---|
| MSE / L2 | regression | (ŷ−y)² | smooth, outlier-sensitive |
| MAE / L1 | regression | |ŷ−y| | robust, kink at 0 |
| Huber | regression | quad near 0, linear far | best of both |
| Log-Cosh | regression | ln(cosh(ŷ−y)) | smooth Huber, twice-diff |
| Cross-Entropy | multi-class | −Σ y·log(p) | pairs with softmax |
| Binary CE | binary / multi-label | −[y log p + (1−y) log(1−p)] | pairs with sigmoid |
| Hinge | SVM / margins | max(0, 1 − y·ŷ) | margin-based, sparse |
| Focal | imbalanced detect | −(1−p)γ log(p) | ignores easy examples |
Mental model → The loss is a landscape the optimizer descends. Its steepness where you're wrong = how strong the correction. MSE: gentle near the answer, violent at outliers. Cross-entropy: infinite cliff for confident errors. Choosing a loss is choosing what the model fears most.