The core tension → Underfit (too simple) = high bias, misses the pattern. Overfit (too complex) = high variance, chases noise. The sweet spot minimizes error on unseen data. Regularization pulls an over-powered model back toward simplicity.
// 01
Watch it overfit · polynomial regression with ridge (L2)
FIT — true — model ● train ○ test
CONTROLS
DEGREE
11
λ (L2)
1e-6
NOISE σ
0.16
SAMPLES
16
train error · —
test error · —
‖w‖² · —
verdict · —
Do this → Set degree to 11 and λ to 1e-6: the curve wiggles wildly through every point — tiny train error, huge test error. That's overfitting. Now drag λ up: the penalty +λ‖w‖² shrinks the weights, the curve smooths, and test error drops even as train error rises slightly. Too much λ and it underfits into a flat line.
// 02
L1 vs L2 · why one zeros weights and the other just shrinks them
CONSTRAINT GEOMETRY loss ellipses meet the budget region
L2 · ‖w‖² ≤ t (circle)
L1 · ‖w‖₁ ≤ t (diamond)
BUDGET t
0.80
THE INTUITION
The optimal weights sit at the center of the loss ellipses. Regularization forces the solution to live inside a budget region. Where the ellipse first touches that region is the regularized answer.
L2 (circle): smooth boundary → the contact point usually has both weights small but non-zero. Shrinks everything toward zero. This is "weight decay" — the most common regularizer.
L1 (diamond): sharp corners on the axes → the ellipse tends to touch at a corner, where one weight is exactly 0. L1 produces sparse models — automatic feature selection.
// 03
Dropout · randomly silence neurons during training
EACH STEP, A RANDOM SUBSET IS DROPPED TRAINING
CONTROLS
DROP RATE p
0.40
MODE
Training: each forward pass drops neurons with probability p — the network can't rely on any single unit, so it learns redundant, robust features.
Inference: all neurons active, outputs scaled by (1−p) so the expected signal matches. No randomness at test time.
Inference: all neurons active, outputs scaled by (1−p) so the expected signal matches. No randomness at test time.
// 04
The full toolbox
| METHOD | HOW | EFFECT |
|---|---|---|
| L2 / weight decay | add λ‖w‖² to loss | shrinks all weights; smoother function |
| L1 / Lasso | add λ‖w‖₁ to loss | drives weights to exactly 0 → sparse |
| Dropout | randomly zero activations | prevents co-adaptation; ensemble effect |
| Batch / Layer Norm | normalize activations per batch/layer | stabilizes training; mild regularizer |
| Early stopping | halt when val loss stops improving | free; stops before memorization |
| Data augmentation | flip/crop/jitter inputs | more effective data → less overfit (see Aug Lab) |
| Label smoothing | soften one-hot targets | less over-confidence; better calibration |
In practice → you rarely pick one. A typical recipe stacks weight decay + dropout + batch-norm + data augmentation + early stopping together. Each attacks overfitting from a different angle; the combination is what makes big models generalize.