← All papers

Attention Is All You Need

Vaswani, Shazeer, Parmar et al. · 2017 · NeurIPS 2017

ArchitectureRead on arXiv

Introduced the Transformer architecture, replacing recurrence and convolutions entirely with self-attention mechanisms. This paper became the foundation for virtually all modern large language models.

Key Idea

The paper proposes the Transformer, a sequence-to-sequence model built entirely on attention mechanisms, eliminating the need for recurrent (RNN/LSTM) or convolutional layers. The core innovation is multi-head self-attention, which allows every position in a sequence to attend to every other position in a single step.

Architecture

  • Encoder-Decoder structure with stacked layers of multi-head self-attention and position-wise feed-forward networks
  • Scaled dot-product attention: Attention(Q, K, V) = softmax(QKᵀ / √dₖ) V
  • Multi-head attention runs multiple attention functions in parallel, then concatenates and projects results
  • Positional encodings (sinusoidal) inject sequence order information since the model has no inherent notion of position

Why It Matters

  • Parallelization: unlike RNNs, all positions are processed simultaneously, enabling massive GPU utilization
  • Long-range dependencies: self-attention connects any two positions in O(1) operations vs O(n) for RNNs
  • Scalability: the architecture scales to billions of parameters, enabling GPT, BERT, and all modern LLMs
  • Achieved state-of-the-art on machine translation (English-German, English-French) at the time of publication

Key Takeaways for Interviews

  • Understand Q, K, V matrices and how attention scores are computed
  • Know why we scale by √dₖ (to prevent softmax saturation with large dimensions)
  • Multi-head attention = multiple representation subspaces learned in parallel
  • Positional encodings are necessary because self-attention is permutation-invariant