// 01
The self-distillation loop
STUDENT LEARNS TO MATCH TEACHER · TEACHER = SLOW COPY OF STUDENT
one image,
two crops
two crops
STUDENT gθs
gradient ↓ updates this
TEACHER gθt
EMA, no gradient
1 · both see crops of same image
2 · teacher output → center + sharpen
3 · student matches it
min −Σ Pt log Ps
min −Σ Pt log Ps
4 · θt ← λθt + (1−λ)θs
Why doesn't it cheat? If student and teacher were identical and free, both could output a constant and the loss would be zero — total collapse. DINO blocks this two ways: the teacher is a momentum average (it changes slowly, giving a stable target) and its outputs are centered (subtract a running mean) and sharpened (low temperature). Try breaking it in §3.
EMA · TEACHER MOMENTUM λ
MOMENTUM λ
0.9960
Student weights (yellow) jump around each step; teacher (magenta) is their slow trailing average. Higher λ → smoother, more stable target, slower to adapt.
// 02
Multi-crop · global views for the teacher, local for the student
CROP STRATEGY 2 global + 6 local
THE CROPS EACH NETWORK RECEIVES
Global crops (large, 224²) go to teacher. Local crops (small, 96²) go to student. The student must predict the teacher's "big picture" view from a tiny fragment — forcing it to learn part-to-whole relationships.
The intuition → A local crop of a cat's ear must produce the same representation as the global crop of the whole cat. To pull that off, the student learns "this ear belongs to this kind of object" — semantic structure, from geometry alone.
// 03
Avoiding collapse · centering & sharpening
TEACHER OUTPUT CONTROLS
CENTERING
TEACHER TEMP
0.04
Low temp = sharp (one prototype dominates). High temp = flat (no signal). Centering subtracts a running mean so no single prototype can dominate every image.
state · HEALTHY
output entropy · 2.1
TEACHER OUTPUT DISTRIBUTION · over K=16 prototypes, for 4 different images
Healthy: each image peaks on a different prototype. Collapsed: every image peaks on the same one — the model learned nothing.
// 04
The payoff · emergent attention segmentation
DINO was never told where objects are. Yet if you look at the attention from the [CLS] token of a DINO-trained ViT, it lights up exactly on the foreground object — clean enough to use as an unsupervised segmentation mask. This is the famous, surprising result. Pick an image and inspect the heads.
INPUT
OVERLAY
70%
[CLS] ATTENTION · per head no labels were used
MEAN ATTENTION (SEGMENTATION)
INDIVIDUAL HEADS · each finds a different part
// 05
Three ways to pretrain · pick your supervision
| METHOD | SUPERVISION | SIGNAL | EMERGES |
|---|---|---|---|
| Supervised (ResNet/ViT) | class labels | "this is class k" | class-discriminative features |
| CLIP | image–text pairs | "this image ↔ this caption" | open-vocabulary, zero-shot |
| DINO | none (self) | "my crops should agree" | object segmentation, strong kNN |
Where this leads → DINOv2 scaled this recipe to produce general-purpose visual features that work — frozen, no fine-tuning — for depth, segmentation, retrieval, and classification. Self-supervision is now the default way to pretrain a vision backbone when labels are scarce.