Background · why attention exists
Before 2017, sequence models were recurrent (RNNs, LSTMs): they read one token at a time, carrying a hidden state forward. Two problems. (1) Distance: relating word 1 to word 100 means threading information through 99 steps — signal decays. (2) Sequential: you can't parallelize; step t needs step t−1.
The transformer threw recurrence away. Its claim: you don't need to walk the sequence — just let every element directly query every other in one parallel operation. The mechanism that does this is self-attention. Distance becomes irrelevant (token 1 and token 100 are one hop apart) and the whole thing is a few big matrix multiplies, perfect for GPUs.
Vision borrowed it in 2020 (the Vision Transformer): chop an image into patches, treat each patch like a "word", and run the exact same machinery. No convolutions required.
| receptive field | local (k×k) | global (all) |
| weights | fixed kernel | data-dependent |
| long-range | needs depth | 1 layer |
| cost | O(n) | O(n²) |
Anatomy · queries, keys & values
The whole mechanism is a soft, learnable dictionary lookup. Every token emits three vectors via three learned projection matrices: a Query ("what am I looking for?"), a Key ("what do I offer?"), and a Value ("what I'll actually pass on"). A token's output is a weighted blend of everyone's Values, where the weights come from how well its Query matches each Key.
Vision Transformer · turning an image into tokens
224²×3
16×16 grid
embed
→ 768-D
embedding
"where"
[CLS]
summary
blocks
classifier
Patchify + embed: a 224² image becomes a 14×14 grid of 16×16 patches → 196 patches. Each patch is flattened and linearly projected to a 768-D vector. Now it's a sequence of 196 "tokens", just like words.
Positional embedding: attention is order-blind — shuffle the patches and it can't tell. So we add a learned position vector to each token, encoding where it sat in the grid.
[CLS] token: a special extra token is prepended. It attends to all patches across every layer and, at the end, its vector is the image's summary — fed to a tiny classifier head. (This is also the token whose attention map DINO visualizes.)
N blocks: stack the transformer block 12–32 times. Early blocks mix local patches; deep blocks relate far-apart regions into object-level meaning.
Interactive · hover a patch, watch it attend
Now make it concrete. Below is a patchified image. Hover any patch — that patch becomes the query. Lines shoot to the patches it attends to most; the heatmap shows the full distribution; the big matrix is every patch's attention to every other (one row per query).
Multi-head · several attentions in parallel
One attention can only express one notion of "relevant". So transformers run several heads in parallel, each with its own Q/K/V projections, then concatenate. One head might track colour, another position, another contrast. Hover the image above; each head's response is shown below.