← All papers

LoRA: Low-Rank Adaptation of Large Language Models

Hu, Shen, Wallis et al. · 2021 · ICLR 2022

EfficiencyRead on arXiv

Introduced a parameter-efficient way to fine-tune large models by injecting trainable low-rank decomposition matrices into each layer. LoRA made it practical to customize billion-parameter models on consumer hardware.

Key Idea

Instead of fine-tuning all parameters of a large model, LoRA freezes the pretrained weights and injects small trainable rank-decomposition matrices into each layer. For a weight matrix W ∈ ℝ^{d×k}, LoRA learns ΔW = BA where B ∈ ℝ^{d×r} and A ∈ ℝ^{r×k} with rank r << min(d, k).

How It Works

  • Freeze all pretrained model weights (no gradient updates)
  • Inject two small matrices A and B into attention layers: W' = W + BA
  • Train only A and B, typically r = 8 or 16, so the trainable parameter count is tiny
  • At inference, merge BA back into W, zero additional latency

Advantages

  • Memory efficient: only stores and updates a small fraction of parameters (0.01 to 1% of full model)
  • No inference overhead: adapted weights merge back into the base model
  • Composable: multiple LoRA adapters can be swapped or combined for different tasks
  • Hardware accessible: fine-tune a 7B model on a single consumer GPU

Why It Matters

  • Made LLM fine-tuning accessible to researchers and companies without massive GPU clusters
  • Enabled the open-source fine-tuning revolution (Alpaca, Vicuna, etc.)
  • Spawned a family: QLoRA (quantized base + LoRA), DoRA, AdaLoRA, LoRA+
  • Standard technique in production systems for task-specific model customization

Key Takeaways for Interviews

  • LoRA exploits the hypothesis that weight updates during fine-tuning have low intrinsic rank
  • The rank r is a key hyperparameter: higher = more expressive but more parameters
  • LoRA adapters are small files (~10-100MB) that can be swapped at serving time
  • QLoRA combines 4-bit quantization of the base model with LoRA for even greater efficiency