M04 / FOUNDATIONS

Activation functions. The kink
that makes networks non-linear.

Stack a hundred linear layers and you still have… one linear layer. The activation is the small non-linear bend applied after each layer that lets a network represent curves, corners, and concepts. Different bends have very different consequences for training.

EXPERIMENT 04.01 · ACTIVATION.PLOT FUNCTIONS · 9 STATUS · LIVE
Why we need them → A layer computes y = Wx + b — a straight line. Compose two: W₂(W₁x+b₁)+b₂ = W'x+b' — still a straight line. The activation σ(·) inserts a non-linearity between layers, so σ(W₂·σ(W₁x)) can bend. That single bend, repeated, is what gives deep nets their expressive power.
// 01

The interactive plotter · function & its gradient

f(x) solid  ·  f′(x) dashed drag the input marker
SHOW
INPUT x 1.00
GRADIENT
READOUT @ x = 1.00
// 02

Two failure modes to recognize

VANISHING GRADIENT

Look at sigmoid / tanh in the plotter. When |x| is large, the curve flattens and its gradient (dashed) → 0. In a deep net, gradients are multiplied layer by layer during backprop. Many near-zero gradients multiply to nothing, and early layers stop learning.

This is exactly why ReLU (gradient = 1 for x>0) replaced sigmoid as the default — its gradient doesn't shrink for positive inputs.

DYING ReLU

ReLU's gradient is 0 for x < 0. If a neuron's weights push it permanently negative, its gradient is always zero — it never updates again. It's dead.

Leaky ReLU / ELU / GELU fix this by giving a small non-zero slope (or smooth curve) for negative inputs, keeping a trickle of gradient alive.

// 03

The full gallery

// 04

Softmax · the output-layer special case

LOGITS → PROBABILITIES
TEMPERATURE 1.0
WHY SOFTMAX IS DIFFERENT

Every other activation works on one number at a time. Softmax works on a whole vector: it exponentiates each logit and normalizes so the outputs are positive and sum to 1 — a probability distribution over classes.

softmax(z)i = ezi/τ / Σj ezj/τ

Temperature τ controls confidence: low τ → sharp (winner takes all), high τ → flat (uncertain). It pairs with cross-entropy loss (see the Loss Functions module) to train classifiers.

Rule of thumb → Hidden layers: start with ReLU; upgrade to GELU/SiLU for transformers and big models. Avoid sigmoid/tanh in deep hidden stacks (vanishing gradient). Output layer: softmax for multi-class, sigmoid for binary/multi-label, none (linear) for regression.