M11 / ARCHITECTURES

CLIP. Teach a model to read
and to see — in the same space.

CLIP doesn't classify into fixed labels. It learns one shared space where an image of a cat and the words "a photo of a cat" land in the same place. Once you have that, classification becomes a similarity lookup — and you can invent new classes just by typing them.

EXPERIMENT 06.01 · CONTRASTIVE.ALIGN PARADIGM · DUAL-ENCODER · CONTRASTIVE CLASSES · 6 STATUS · LIVE
// 01

The idea · two encoders, one space

DUAL-ENCODER ARCHITECTURE
IMAGE ENCODER
ViT-B/32 or ResNet
image vector
I ∈ ℝ⁵¹²
SHARED
EMBEDDING SPACE
cos θ → 1
pull matched
pairs together
"a photo
of a cat"
TEXT ENCODER
Transformer
text vector
T ∈ ℝ⁵¹²
Why this is powerful → A normal classifier has a fixed output head — 1000 ImageNet classes, no more. CLIP has no output head. To classify, you embed the image once and embed any set of label strings you like, then pick the closest. New class? Just write a new sentence. This is zero-shot classification.
HOW TO READ THIS MODULE
§2
Contrastive pretraining
Watch the N×N similarity matrix snap from noise to a glowing diagonal as training proceeds.
§3
Zero-shot classification
Click an image; CLIP scores it against six text prompts and predicts the winner.
§4
Prompt engineering
The exact wording of the label changes the score. See why "a photo of a {}" wins.
§5
CLIP vs OpenCLIP
What the open reproduction changed — data, scale, licensing.
// 02

Contrastive pretraining · the N×N matrix

Take a batch of N image–caption pairs. Embed all N images and all N captions. Compute every image-to-text cosine similarity — an N×N grid. The diagonal (matched pairs) should be high; everything off-diagonal (mismatched) should be low. The loss (InfoNCE) is just a softmax cross-entropy that says "for each row, the correct column is the diagonal."

IMAGE × TEXT SIMILARITY cosine(Iᵢ, Tⱼ)
TRAINING
TRAIN STEP 0%
TEMPERATURE τ 0.10
InfoNCE loss · 2.48
diagonal mean · 0.18
off-diag mean · 0.16
batch accuracy · 17%
L = ½(Limg→txt + Ltxt→img)
Li→t = −log exp(sᵢᵢ/τ) / Σⱼ exp(sᵢⱼ/τ)
// 03

Zero-shot classification · the payoff

PICK A TEST IMAGE unseen samples
PROMPT TEMPLATE
prompt set: "a photo of a cat" · "a photo of a dog" · …
SOFTMAX OVER PROMPT SIMILARITIES prediction → cat
What just happened → The image was embedded once. Each label string was embedded into the same space. We took cosine similarity, divided by temperature, and softmaxed. No retraining, no labels at train time for these specific classes — that's why it's called zero-shot.
// 04

CLIP vs OpenCLIP · same recipe, open kitchen

ASPECT CLIP (OpenAI '21) OpenCLIP (LAION)
Training data400M private pairs (WIT)LAION-400M / 2B / DataComp — public
Weightsreleased; data notweights + data + code all open
BackbonesResNet-50…, ViT-B/32…L/14adds ViT-G/14, ConvNeXt, more
Scaling lawsimplicitexplicitly studied & reproducible
Top zero-shot (IN-1k)~76% (ViT-L/14)~80%+ (ViT-G/14, bigger data)
Why it mattersproved the recipemade it auditable & extendable
THE TAKEAWAY

OpenCLIP is not a different model — it's the same contrastive recipe re-implemented in the open by LAION, trained on publicly downloadable image-text pairs. Because everything is public, researchers can study exactly how zero-shot accuracy scales with data size and model size.

Practically: if you want a CLIP model you can fine-tune, audit, or ship commercially without licensing worry, you reach for OpenCLIP. The open_clip library is the de-facto standard interface, and most newer vision-language models bootstrap from an OpenCLIP backbone.

The mental model → CLIP turns "classification" into "search". Build the catalogue of text vectors once; embed any image; nearest neighbour wins. That's also why CLIP powers text-to-image search, image dedup, and the text guidance inside diffusion models.