High-Resolution Image Synthesis with Latent Diffusion Models

Key Idea

Standard diffusion models operate in pixel space, which is computationally expensive for high-resolution images. Latent Diffusion Models (LDMs) first compress images into a lower-dimensional latent space using a pretrained autoencoder, then apply the diffusion process in this compact representation.

Architecture

• Autoencoder (VAE): an encoder compresses images to a latent representation (e.g., 512×512 → 64×64×4), and a decoder reconstructs images from latents
• U-Net with cross-attention: performs the denoising diffusion process in latent space, with cross-attention layers for text/image conditioning
• Conditioning: CLIP text embeddings are injected via cross-attention, enabling text-to-image generation

The Diffusion Process

1. Forward process: gradually add Gaussian noise to the latent representation over T timesteps
2. Reverse process: train a U-Net to predict and remove the noise at each step
3. Sampling: start from pure noise, iteratively denoise to generate a clean latent, decode to an image

Why It Matters

• 10-50x more efficient than pixel-space diffusion (DDPM) while maintaining quality
• Enabled high-resolution image generation (512×512, 1024×1024) on consumer GPUs
• Foundation of Stable Diffusion, the most widely used open-source image generation model
• Architecture extended to video generation (Stable Video Diffusion) and 3D generation

Key Takeaways for Interviews

• The key insight: separate perceptual compression (autoencoder) from generative learning (diffusion)
• Operating in latent space reduces compute by orders of magnitude vs pixel-space diffusion
• Cross-attention enables flexible conditioning (text, images, segmentation maps)
• Classifier-free guidance balances fidelity (following the prompt) vs diversity