🧠 Transformers & Attention

A transformer is a neural architecture built on self-attention, which lets every token attend to every other token in parallel, replacing the sequential recurrence of RNNs. It is the foundation of modern LLMs because it scales to long contexts and huge parameter counts while staying trainable.

The Transformer Architecture

The Transformer, introduced in "Attention Is All You Need" (Vaswani et al., 2017), replaced recurrent architectures with a fully attention-based mechanism. It processes entire sequences in parallel rather than step-by-step, enabling massive speedups on modern hardware.

Core Components

Self-Attention computes relationships between all positions in a sequence simultaneously. For each token, it produces three vectors (Query (Q), Key (K), and Value (V)) by multiplying the input embedding with learned weight matrices. The attention score between two positions is the dot product of Q and K, scaled by √d_k to prevent gradient issues, then passed through softmax to get weights over the Values.

Attention(Q, K, V) = softmax(QK^T / √d_k) · V

Want to poke at it directly? The self-attention playground lets you pick a query token, slide the √d_k scaling to watch softmax sharpen or flatten, and toggle the causal mask.

Multi-Head Attention runs multiple self-attention operations in parallel, each with different learned projections. This allows the model to attend to information from different representation subspaces; one head might capture syntactic relationships while another captures semantic ones. Outputs are concatenated and linearly projected.

Positional Encoding is necessary because self-attention is permutation-invariant: it has no inherent notion of sequence order. Shuffle the input tokens and the raw attention math produces the same output, so order has to be injected explicitly. The original paper added sinusoidal vectors (sines and cosines of different frequencies) to the token embeddings, giving each absolute position a unique fingerprint. Early LLMs (BERT, GPT-2) instead used learned absolute position embeddings, one trainable vector per slot.

Both are absolute schemes with a shared weakness: they bake in a fixed maximum length and generalize poorly beyond it, and what usually matters for language is relative position ("the adjective two words back"), not the absolute index.

Rotary Position Embeddings (RoPE) — used by LLaMA, GPT-NeoX, Qwen, and most modern LLMs — solve this elegantly. Instead of adding a position vector, RoPE rotates each query and key vector by an angle proportional to its position: the vector at position m is rotated by m·θ (applied to pairs of dimensions, each pair spinning at its own frequency). The magic is what happens in the attention dot product: when a query at position m meets a key at position n, the rotations combine so the score depends only on their relative offset (m − n), not their absolute positions. This gives three big wins: (1) relative-position awareness falls out for free, (2) nothing is added to the residual stream so it composes cleanly at every layer, and (3) it extrapolates — a model trained at 4K context can be stretched to longer sequences (with tricks like NTK-aware scaling or YaRN) because the rotation is a smooth function of position, not a lookup table. RoPE is why long-context models became practical.

The RoPE playground makes the relative-position trick tangible: drag the query and key positions to watch their vectors rotate, then move both together and see the attention score stay perfectly fixed — because it depends only on the gap between them.

Encoder-Decoder vs Decoder-Only

The original Transformer has an encoder (bidirectional attention) and decoder (causal/masked attention with cross-attention to the encoder). BERT uses only the encoder; GPT uses only the decoder; T5 uses both. Modern LLMs are almost exclusively decoder-only with causal masking.

Computational Complexity

Self-attention has O(n²·d) time and O(n²) memory complexity, where n is sequence length. This quadratic scaling is the main bottleneck for long sequences. Approaches to mitigate this include sparse attention (Longformer), linear attention (Performers), and sliding-window attention (Mistral).

The attention-complexity playground makes the wall concrete: sweep the sequence length and watch full attention's cost quadruple each time you double n, then switch to windowed or sparse-global patterns and see the work drop back toward linear.

From CNN/RNN to Attention

Before 2017, sequence modeling was dominated by recurrent networks: LSTMs (Hochreiter & Schmidhuber, 1997) and GRUs, which processed tokens one at a time, propagating a hidden state forward through the sequence. This sequential dependency had two fatal flaws at scale: it prevented parallelization across the time dimension (each step waits for the previous), and the hidden state became a brutal information bottleneck, struggling to preserve long-range dependencies despite gating mechanisms. Gradients still vanished or exploded across hundreds of timesteps, and training a 100M-parameter LSTM on a multi-billion-token corpus was practically impossible.

Bahdanau et al. (2014) introduced attention as an additive mechanism layered on top of an encoder-decoder RNN for machine translation, letting the decoder softly look back at all encoder states instead of relying on a single fixed-length vector. This solved the bottleneck problem but still inherited the RNN's sequential cost. The Vaswani et al. (2017) "Attention Is All You Need" paper made the bolder leap: drop recurrence entirely. Once every position can attend to every other position via a single matrix multiply, the model parallelizes perfectly across the sequence, training throughput jumps by an order of magnitude on GPU/TPU hardware, and the path length between any two tokens collapses from O(n) to O(1), making it dramatically easier for gradients to flow between distant positions.

Convolutional sequence models (ConvS2S, ByteNet, WaveNet) tried a different escape from recurrence (stacking dilated convolutions to expand the receptive field) but they still had a fixed local prior and required deep stacks to model long-range structure. Attention sidestepped both limits in one design.

ViT: When Transformers Came for Vision

Computer vision was the last holdout. CNNs had reigned since AlexNet (2012), and their inductive biases (translation equivariance, local receptive fields, hierarchical feature composition) seemed irreplaceable for pixel data. Dosovitskiy et al. (2020) showed otherwise with the Vision Transformer (ViT): split an image into 16×16 patches, flatten each patch into a vector, add learned positional embeddings, and feed the sequence into a standard Transformer encoder. No convolutions, no pooling, no hand-engineered visual priors.

On small datasets (ImageNet-1k from scratch), ViT underperformed ResNets; without the convolutional inductive bias, it needed more data to learn that nearby pixels are related. But pre-trained on JFT-300M (300M images) and fine-tuned downstream, ViT matched or beat state-of-the-art CNNs while using fewer FLOPs. The lesson generalized: with enough data and compute, the Transformer's lack of priors becomes a feature, not a bug; it learns the right inductive biases from data rather than having them imposed. This unlocked the modern multimodal era (CLIP, DALL-E, Flamingo, Gemini), where the same architecture processes text, images, audio, and video tokens through one attention stack.

Layer Normalization and Residual Connections

Every sub-layer (attention, feed-forward) is wrapped with two mechanisms that together are what actually let Transformers stack to 100+ layers without collapsing.

Residual connections add a sub-layer's input back to its output: x + Sublayer(x). This creates a "gradient highway" — an uninterrupted path from the loss all the way back to the earliest layers, so gradients can flow without vanishing through dozens of blocks. It also reframes each layer's job: instead of computing a whole new representation from scratch, a layer only has to learn a delta — a small refinement to the running representation. The sum of all these deltas as you go up the stack is called the residual stream, and interpretability research treats it as the model's shared "workspace" that every layer reads from and writes to.

Layer normalization stabilizes the scale of activations. For each token, it standardizes that token's feature vector to zero mean and unit variance, then applies a learned gain and bias. Without it, activations drift and explode as they accumulate through the residual stream, and training becomes hyperparameter-fragile. (Crucially, it normalizes per token across features, not across the batch — so it's independent of batch size and works fine at batch size 1, unlike batch normalization.)

Pre-norm vs post-norm is where the placement matters. The original Transformer used post-norm: LayerNorm(x + Sublayer(x)) — normalization after the residual add. This puts a normalization layer directly on the residual highway, which repeatedly rescales the accumulated signal and destabilizes gradients in deep stacks, forcing careful learning-rate warmup and often diverging past a few dozen layers. Pre-norm — x + Sublayer(LayerNorm(x)) — normalizes the input to each sub-layer instead, leaving the residual stream itself a clean, un-normalized additive path. This is dramatically more stable at depth and scale, so essentially every modern LLM uses pre-norm.

RMSNorm is the modern simplification (LLaMA, most 2023+ models): it drops the mean-subtraction and the bias, normalizing only by the root-mean-square of the features. It's cheaper to compute and empirically matches LayerNorm's quality, so it has largely replaced it in frontier models.

Feed-Forward Network

Each transformer block includes a position-wise FFN, typically two linear layers with a non-linearity (ReLU or SwiGLU in modern models). The hidden dimension is usually 4× the model dimension.