← All papers

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Dosovitskiy, Beyer, Kolesnikov et al. · 2020 · ICLR 2021

ArchitectureRead on arXiv

Proved that a pure Transformer applied directly to sequences of image patches can match or exceed the best convolutional networks. ViT unified the vision and language architectures, enabling the modern multimodal AI stack.

Key Idea

Vision Transformer (ViT) applies the standard Transformer encoder (from "Attention Is All You Need") directly to images by splitting them into fixed-size patches, linearly embedding each patch, and processing the sequence with self-attention, no convolutions needed.

How It Works

  1. Split the image into non-overlapping patches (typically 16×16 pixels)
  2. Flatten each patch into a vector and project it through a linear embedding
  3. Prepend a learnable [CLS] token (classification token)
  4. Add learnable positional embeddings to each patch embedding
  5. Process the sequence through a standard Transformer encoder
  6. Classify using the [CLS] token's output through an MLP head

Key Findings

  • ViT underperforms CNNs when trained on small datasets (ImageNet-1K alone), Transformers lack the inductive biases (translation invariance, locality) that convolutions provide
  • But when pretrained on large datasets (ImageNet-21K, JFT-300M), ViT matches or exceeds the best CNNs
  • The inductive biases of CNNs are helpful for small data but become less important when data is abundant
  • ViT scales better than CNNs, performance continues to improve with more data and compute

Why It Matters

  • Unified vision and language architectures: the same Transformer works for both modalities
  • Enabled multimodal models (CLIP, DALL-E, GPT-4V) by using a shared architecture for images and text
  • Simpler architecture than CNNs with fewer domain-specific design choices
  • Spawned a family: DeiT, Swin Transformer, BEiT, MAE, DINO

Key Takeaways for Interviews

  • ViT treats images as sequences of patch embeddings, no convolutions
  • Data scale is critical, ViT needs large-scale pretraining to match CNNs
  • The [CLS] token aggregates global information for classification
  • Positional embeddings encode spatial relationships (since self-attention is permutation-invariant)