🎨 GANs, Diffusion & Flow Matching

Generative Image Models: 2014 → 2026

Generative image modeling has gone through three distinct eras in just over a decade. Each era was defined by a single idea that solved the failure mode of the previous one, and each shifted the dominant training objective for frontier visual models.

Era 1: Generative Adversarial Networks (2014 to ~2021)

Goodfellow et al. (2014) introduced GANs as a two-player minimax game: a generator G maps random noise z ~ p(z) to a fake image G(z), and a discriminator D tries to distinguish real images x ~ p_data from fakes. The objective is:

min_G max_D E[log D(x)] + E[log(1 - D(G(z)))]

At Nash equilibrium, G produces samples indistinguishable from p_data and D outputs 0.5 everywhere. In practice, equilibrium is elusive: gradients to G vanish when D becomes too confident, and G can collapse onto a small set of modes that fool D without covering p_data.

DCGAN (Radford et al., 2015) was the first stable convolutional GAN: strided convolutions, batch norm, no fully-connected layers, ReLU in G and LeakyReLU in D. It made GANs reproducible and unlocked the wave of image GAN work that followed.

WGAN (Arjovsky et al., 2017) replaced the Jensen-Shannon divergence implicit in the original objective with the Wasserstein-1 distance (Earth Mover's Distance). The Wasserstein loss provides meaningful gradients even when supports of p_data and p_G don't overlap, the case where the original GAN loss saturates. WGAN-GP added a gradient penalty to enforce the 1-Lipschitz constraint on the critic, removing the brittle weight-clipping of the original WGAN.

StyleGAN (Karras et al., NVIDIA, 2018) was the high-water mark of GAN image quality. It introduced a mapping network from z to a disentangled latent space W, style mixing at each resolution, and adaptive instance normalization (AdaIN) for style injection. StyleGAN2 fixed characteristic artifacts (water-droplet textures) by replacing AdaIN with weight demodulation; StyleGAN3 (2021) addressed aliasing by making the generator equivariant to translation and rotation, a frequency-aware design rooted in signal processing.

The chronic GAN pains were mode collapse (G outputs few distinct samples), training instability (loss curves uninformative, requires careful hyperparameter tuning), and lack of likelihood (no clean way to score sample quality or do density estimation). By ~2022, diffusion had overtaken GANs for text-to-image generation, though GANs persist in narrow contexts where their fast single-step inference matters: super-resolution (ESRGAN, Real-ESRGAN), face editing, and on-device generation.

Era 2: Denoising Diffusion (2020 to 2023)

Diffusion models (Sohl-Dickstein 2015; Ho et al., DDPM, 2020) flipped the problem on its head. Instead of learning to map noise → image in one shot, they learn to reverse a gradual noising process.

The forward process is a fixed Markov chain that adds small amounts of Gaussian noise over T timesteps (typically T=1000), turning a clean image x_0 into pure noise x_T ~ N(0, I):

q(x_t | x_{t-1}) = N(x_t; √(1-β_t) · x_{t-1}, β_t · I)

Because each step is Gaussian, you can sample x_t directly from x_0 in closed form: x_t = √(ᾱ_t) · x_0 + √(1-ᾱ_t) · ε where ε ~ N(0, I) and ᾱ_t = ∏(1 - β_s).

The reverse process is the learned part: a neural network (typically a UNet, more recently a Diffusion Transformer / DiT) predicts the noise ε added at timestep t given the noisy x_t and t. The training objective is simply MSE between predicted and true noise:

L = E_{x_0, ε, t}[ ‖ε - ε_θ(x_t, t)‖² ]

This is dramatically more stable than GAN training: there is no adversary, no minimax, no equilibrium to chase. Samples are generated by running the reverse process: start from noise x_T, iteratively denoise to x_0 over T steps.

DDIM (Song et al., 2021) reformulated sampling as a deterministic non-Markovian process that admits much shorter trajectories. The same trained model can be sampled in 20-50 steps instead of 1000, with only minor quality loss, making diffusion practical for interactive use.

Latent Diffusion (Rombach et al., 2022) was the unlock that gave us Stable Diffusion. Train an autoencoder once to map 512×512 images down to a 64×64×4 latent. Run the diffusion process in latent space, not pixel space. The compute savings are ~48× per forward pass; the autoencoder handles the perceptually irrelevant high-frequency detail. This dropped inference from a server-side concern to something that runs on a consumer GPU.

Classifier-Free Guidance (Ho & Salimans, 2022) is the trick that made text-to-image diffusion actually controllable. Train a single conditional model ε_θ(x_t, t, c) but drop the condition c at random (~10%) so the same network also learns the unconditional score ε_θ(x_t, t, ∅). At inference, sample with:

ε̂ = ε_θ(x_t, t, ∅) + w · (ε_θ(x_t, t, c) - ε_θ(x_t, t, ∅))

Guidance scale w > 1 pushes samples toward the conditioning at the cost of diversity. This is why "increase the CFG scale" dials prompt adherence in every modern diffusion UI.

Score-based view (Song et al., 2020-2021) showed that diffusion is equivalent to learning the score function ∇_x log p_t(x) of a family of noise-perturbed distributions, and that the forward/reverse processes are time-reversals of a stochastic differential equation (SDE). The probability-flow ODE corresponding to that SDE produces deterministic samples and connects diffusion to continuous normalizing flows, the conceptual bridge to Era 3.

Why diffusion beat GANs by 2022: (1) stable training, with an MSE loss, no adversary, and monotonic scaling with compute and data; (2) mode coverage, since the objective is implicitly maximum likelihood, so the model is forced to cover the data distribution; (3) controllability, because CFG, ControlNet, and inpainting masks compose cleanly when the model is a denoiser, not a black-box sampler; (4) scaling, where DALL-E 2, Imagen, and Stable Diffusion all rode the same training recipe to dramatically better quality just by adding data and parameters.

Era 3: Flow Matching & Continuous-Time Generation (2023 to 2026)

Diffusion's weakness is sampling cost. Even with DDIM, you need 20-50 network evaluations per image. The 2023+ wave of work attacks this directly by learning straighter probability paths.

Flow Matching (Lipman et al., 2023) reframes generation as learning a time-dependent vector field v_θ(x, t) such that following the ODE dx/dt = v_θ(x, t) from t=0 (noise) to t=1 (data) transports the source distribution to the target. The training objective is a simple regression on a conditional vector field that's chosen analytically (e.g., the constant velocity along straight lines between paired noise/data samples):

L = E_{t, x_0, x_1}[ ‖v_θ(x_t, t) - (x_1 - x_0)‖² ]

where x_t = (1-t)·x_0 + t·x_1 is a linear interpolant. This is mathematically equivalent to diffusion in the continuous-time limit, but with two big practical wins: (1) you can pick the path (not just learn it) and choose straight lines, which means fewer ODE-integration steps; (2) the training objective is just an MSE on a velocity, no noise schedule to design.

Rectified Flow (Liu et al., 2022) introduces the "reflow" procedure: train a flow, then re-couple noise/data pairs by simulating the trained flow and re-training on the new pairs. Each reflow step makes trajectories straighter; after one reflow, 1-2 step generation becomes possible.

Consistency Models (Song et al., 2023) distill a diffusion or flow model into a one-step (or few-step) sampler by enforcing that points along the same probability-flow trajectory all map to the same endpoint. This is how modern "real-time" image and video tools (Latent Consistency Models, SDXL Turbo, Flux Schnell) generate in 1-4 steps instead of 50.

Flow matching is now the default training objective for frontier image, video, and audio models. It scales better than score-matching diffusion, produces straighter paths so distillation works, and integrates cleanly with the Diffusion Transformer (DiT) architecture.

2026 State of the Art

The frontier image generation landscape in 2026:

• Flux.1 (Black Forest Labs, the ex-Stable Diffusion team): rectified flow + 12B-parameter DiT. Schnell variant generates in 1-4 steps.
• Stable Diffusion 3 / 3.5: flow matching, multimodal DiT (MMDiT) that processes text and image tokens through joint attention rather than cross-attention.
• DALL-E 3 (OpenAI): diffusion with heavy prompt-rewriting via GPT-4 for caption quality.
• Imagen 3 / Gemini Image (DeepMind / Google): cascaded diffusion with strong text rendering.
• Ideogram, Midjourney v7: proprietary architectures, almost certainly DiT + flow matching.

Video generation followed the same trajectory: Sora (OpenAI), Veo 3 (DeepMind), Mochi-1 (Genmo), Hunyuan-Video (Tencent), and Runway Gen-4 are all diffusion-transformer architectures with temporal attention or 3D attention, trained with flow matching or velocity-prediction objectives.

Beyond vision, diffusion has spread into domains where the structured-noise prior fits naturally: protein structure (AlphaFold 3 uses a diffusion head for atomic coordinates), discrete diffusion for text (MAR, MDLM, Mercury, all non-autoregressive language modeling), robotic policies (Diffusion Policy, π0, with action chunks as denoising targets), and 3D / mesh generation (Trellis, latent diffusion on Gaussian-splat or mesh representations).