Adam: A Method for Stochastic Optimization

Key Idea

Adam (Adaptive Moment Estimation) maintains per-parameter learning rates by combining two ideas: momentum (exponential moving average of gradients) and adaptive learning rates (exponential moving average of squared gradients), with bias correction to handle initialization.

The Algorithm

1. Compute gradient g at timestep t
2. Update first moment estimate: m = β₁·m + (1-β₁)·g (momentum / mean of gradients)
3. Update second moment estimate: v = β₂·v + (1-β₂)·g² (RMSProp / variance of gradients)
4. Bias correction: m̂ = m/(1-β₁ᵗ), v̂ = v/(1-β₂ᵗ)
5. Update parameters: θ = θ - α · m̂ / (√v̂ + ε)

Default hyperparameters: α=0.001, β₁=0.9, β₂=0.999, ε=1e-8

Why Each Component Matters

• First moment (m): provides momentum, accelerating convergence in consistent gradient directions and damping oscillations
• Second moment (v): adapts learning rate per parameter: parameters with large gradients get smaller updates, sparse gradients get larger updates
• Bias correction: critical in early training steps when m and v are biased toward zero (initialized at 0)

Why It Matters

• Became the de facto standard optimizer for deep learning, the default choice when you don't know what else to use
• Works well across a wide range of architectures (CNNs, RNNs, Transformers) with minimal tuning
• Robust to noisy gradients, sparse gradients, and non-stationary objectives
• Variants: AdamW (decoupled weight decay, preferred for Transformers), RAdam, AdaFactor

Key Takeaways for Interviews

• Adam = Momentum + RMSProp + Bias Correction
• AdamW is preferred for Transformers, it decouples weight decay from the adaptive learning rate
• Default hyperparameters (β₁=0.9, β₂=0.999) work surprisingly well in most cases
• Adam can converge to sharp minima, SGD with momentum sometimes generalizes better for CNNs, but Adam dominates for Transformers