⚡ Model Quantization
Reducing model size and latency with INT8/INT4 quantization techniques
What is Quantization?
Quantization reduces the numerical precision of model weights and/or activations, typically from 32-bit floating point (FP32) or 16-bit (FP16/BF16) to 8-bit integers (INT8) or even 4-bit (INT4). This reduces memory footprint, speeds up inference, and enables deployment on smaller hardware.
Why Quantize?
A 7B parameter model in FP16 needs ~14 GB of VRAM. In INT4, the same model fits in ~3.5 GB, runnable on a consumer GPU or even a laptop. Quantization also improves throughput since memory bandwidth is typically the bottleneck for LLM inference (the model is memory-bound, not compute-bound during autoregressive decoding).
Post-Training Quantization (PTQ)
PTQ quantizes a pre-trained model without additional training. It's fast and convenient but can degrade quality, especially at low bit-widths.
Weight-only quantization quantizes weights to INT8/INT4 but keeps activations in FP16. This is the most common approach for LLMs because weights dominate memory and activations are harder to quantize (they have outliers).
Weight + activation quantization (W8A8) quantizes both, enabling INT8 matrix multiplication on hardware with INT8 support. More speedup but harder to maintain quality.
Quantization-Aware Training (QAT)
QAT simulates quantization during training, allowing the model to adapt to reduced precision. It produces higher quality results than PTQ at the same bit-width but requires full training infrastructure and significant compute. Used when PTQ quality is insufficient.
Modern Quantization Methods
GPTQ: A one-shot weight quantization method that uses approximate second-order information (Hessian) to minimize quantization error layer by layer. Fast to apply, widely supported, works well at 4-bit.
AWQ (Activation-Aware Weight Quantization): Observes that a small fraction of weights are critical (those corresponding to large activations). It scales those channels before quantization to protect important weights. Often slightly better quality than GPTQ.
GGUF (llama.cpp): Quantization format for CPU inference. Supports mixed-precision (important layers at higher precision). Popular for local/edge deployment.
SmoothQuant: Addresses the activation outlier problem by mathematically migrating the quantization difficulty from activations to weights (which are easier to quantize). Enables W8A8 for LLMs.
Quantization Granularity
- Per-tensor: One scale factor per entire tensor (fastest, least accurate)
- Per-channel: One scale factor per output channel (good balance)
- Per-group: One scale factor per group of weights (e.g., group of 128), used by GPTQ/AWQ for 4-bit