← All papers

Learning Transferable Visual Models From Natural Language Supervision

Radford, Kim, Hallacy et al. · 2021 · ICML 2021

MultimodalRead on arXiv

Introduced CLIP, training image and text encoders jointly on 400M image-text pairs from the internet. CLIP enabled powerful zero-shot visual understanding by connecting vision and language in a shared embedding space.

Key Idea

CLIP (Contrastive Language-Image Pre-training) learns visual representations from natural language supervision rather than fixed label sets. It trains an image encoder and a text encoder jointly to predict which (image, text) pairs go together in a batch, using a contrastive objective.

Architecture & Training

  • Dual encoder: separate image encoder (ViT or ResNet) and text encoder (Transformer)
  • Contrastive learning: for a batch of N (image, text) pairs, maximize similarity of matching pairs and minimize similarity of non-matching pairs
  • Trained on 400M image-text pairs scraped from the internet (WebImageText dataset)
  • No manual labels required, captions provide natural language supervision

Zero-Shot Classification

  • To classify an image, embed it and compare against text embeddings of candidate labels (e.g., "a photo of a dog", "a photo of a cat")
  • The label whose text embedding is most similar to the image embedding is the prediction
  • Achieves competitive performance with fully supervised models on many benchmarks, without seeing a single labeled example

Why It Matters

  • Zero-shot transfer: a single model works across hundreds of visual tasks without task-specific training
  • Connected vision and language: enabled text-driven image search, generation, and editing
  • Foundation for DALL-E, Stable Diffusion (text conditioning), and modern multimodal models
  • Showed that web-scale data + contrastive learning can replace curated datasets

Key Takeaways for Interviews

  • CLIP uses contrastive learning to align image and text embeddings in a shared space
  • Zero-shot classification via text-image similarity comparison
  • Robust to distribution shift compared to supervised models (natural language provides richer supervision)
  • Limitations: struggles with fine-grained tasks, systematic biases from web data