← All papers

Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture

Assran, Duval, Misra, Bojanowski, Vincent, Rabbat, LeCun, Ballas · 2023 · ICCV 2023

ArchitectureRead on arXiv

Introduces I-JEPA, a non-generative self-supervised learning method that predicts representations of target image blocks in latent space rather than pixel space, without requiring hand-crafted data augmentations like masking or cropping.

Key Idea

Most self-supervised vision methods either rely on invariance-based approaches (contrastive learning with heavy augmentations) or generative approaches (reconstructing pixels like MAE). Both have drawbacks: contrastive methods need carefully designed augmentations that bake in human priors, while generative methods waste capacity predicting low-level pixel details.

I-JEPA takes a different path: predict the representations of target blocks, not the pixels. Given a context block from an image, the model learns to predict the embeddings of other blocks in the same image, entirely in latent space.

How It Works

  • Context encoder processes a single visible block of the image
  • Predictor takes the context representation and positional tokens for target blocks, then predicts what those target representations should be
  • Target encoder (EMA-updated, like BYOL/DINO) produces the ground-truth representations to predict against
  • Multi-block masking strategy: large target blocks (semantic-scale) with spatially distributed context ensure the model captures high-level features, not local textures

Why It Matters

  • Eliminates dependence on hand-crafted augmentations, no random crops, color jitter, or multi-crop needed
  • Learns semantic representations rather than pixel-level details, leading to strong transfer across diverse tasks (classification, counting, depth estimation)
  • Extremely efficient to train, ViT-Huge on ImageNet in under 72 hours on standard hardware
  • Represents Yann LeCun's vision for a non-generative self-supervised learning paradigm, influencing subsequent work like V-JEPA (video) and broader JEPA-family architectures

Key Takeaways for Interviews

  • I-JEPA = predict target representations from context representations, not pixels
  • Uses EMA target encoder (no contrastive negatives needed) + masking in representation space
  • The masking strategy matters: large semantic-scale target blocks force the model to learn high-level features
  • Compare with MAE (reconstructs pixels, good for dense tasks but wastes capacity on low-level details) and DINO/contrastive (needs augmentation pipelines, which bake in human priors)
  • Key insight: predicting in latent space lets the model focus on semantic content and ignore pixel-level noise