← All papers

QLoRA: Efficient Finetuning of Quantized LLMs

Dettmers, Pagnoni, Holtzman, Zettlemoyer · 2023 · NeurIPS 2023

EfficiencyRead on arXiv

Enables finetuning a 65B-parameter model on a single 48GB GPU by combining 4-bit quantization with LoRA adapters. Introduces NormalFloat4 data type and double quantization, achieving full 16-bit finetuning quality at a fraction of the memory cost.

Key Idea

LoRA made finetuning efficient by training small low-rank adapters instead of full weight matrices. But the base model still needs to fit in GPU memory at 16-bit precision, a 65B model needs ~130GB just for weights. QLoRA solves this by quantizing the frozen base model to 4-bit precision and backpropagating gradients through the quantized weights into the LoRA adapters, which remain in 16-bit.

How It Works

Three innovations work together:

  1. 4-bit NormalFloat (NF4): A new data type optimized for normally distributed neural network weights. Quantization levels are placed at quantiles of the normal distribution, making it information-theoretically optimal, each quantization bin holds equal probability mass
  2. Double Quantization: The quantization constants (scale factors) themselves are quantized from 32-bit to 8-bit, saving ~0.37 bits per parameter. Small savings per parameter, but significant at 65B scale
  3. Paged Optimizers: Uses NVIDIA unified memory to automatically page optimizer states between GPU and CPU memory during gradient checkpointing spikes, preventing OOM errors

Training Flow

  • Base model weights: frozen, stored in NF4 (4-bit)
  • LoRA adapter weights: trainable, stored in BFloat16 (16-bit)
  • Forward pass: dequantize base weights to BF16 on the fly, compute with LoRA
  • Backward pass: gradients flow through dequantized weights, update only LoRA adapters
  • Memory: ~48GB for a 65B model (vs ~130GB at 16-bit)

Why It Matters

  • Democratized LLM finetuning: 65B model finetuning on a single GPU, previously required multi-GPU setups costing $100K+
  • No quality loss: Matches full 16-bit finetuning performance despite 4-bit base model, the key insight is that LoRA adapters compensate for quantization error during training
  • Guanaco: The QLoRA-finetuned model achieved 99.3% of ChatGPT performance with just 24 GPU hours on a single GPU
  • Combined with LoRA (already widely adopted), QLoRA became the default finetuning method for practitioners

Key Takeaways for Interviews

  • QLoRA = 4-bit quantized frozen base model + 16-bit LoRA adapters, backprop through quantized weights
  • NF4 is optimal for normally distributed weights (most neural net weights are approximately normal)
  • Double quantization: quantize the quantization constants, marginal per-parameter but significant at scale
  • Compare with LoRA (16-bit base, ~2× memory of full model) vs QLoRA (4-bit base, ~4× less memory than LoRA)
  • In system design: QLoRA is for training/finetuning efficiency; for inference, use GPTQ/AWQ-style post-training quantization instead