LLM EngineeringMedium

🔧 Fine-tuning Strategies

Adapting pre-trained models to specific tasks with full fine-tuning, LoRA, QLoRA, and other PEFT methods

Fine-tuning adapts a pretrained model to a specific task or domain by training further on targeted data, often parameter-efficiently with LoRA or QLoRA rather than updating all weights. Interviews test when to fine-tune versus prompt or use RAG, and how to avoid catastrophic forgetting and overfitting.

Why Fine-tune?

Pre-trained foundation models are trained on broad internet data. They know a lot but they don't know your domain, your tone, or your task format. Fine-tuning specializes the model: a generic chat model becomes a medical-coding assistant, a coding assistant learns your internal APIs, or a base model becomes an instruction follower.

Fine-tuning is the right answer when:

  • You need a consistent output format the model keeps drifting from with prompting
  • Latency or cost forbids long few-shot prompts
  • You need style/voice transfer that prompts can't reliably capture
  • The task requires implicit knowledge that doesn't fit in context (large vocabulary, specialized syntax)

It is the wrong answer when the underlying need is "the model should know X"; that's retrieval (RAG). And it's overkill when a well-crafted prompt already works.

Full Fine-tuning vs Parameter-Efficient Methods

Full fine-tuning (FFT) updates every parameter. For a 7B model in BF16 with AdamW, you need roughly:

  • 14 GB for weights
  • 14 GB for gradients
  • 28 GB for optimizer state (two moments)
  • Activations on top of that

Total: ~70-80 GB of VRAM, before activations. That's why FFT for 7B+ models needs A100/H100-class hardware or model parallelism.

Parameter-Efficient Fine-Tuning (PEFT) freezes the base model and trains a small number of new parameters (often <1% of the total). This collapses memory requirements by 10-100× and produces a tiny adapter you can swap in/out of the frozen base model.

LoRA: Low-Rank Adaptation

LoRA is the dominant PEFT method. The insight: weight updates during fine-tuning have low intrinsic rank, so you don't need a full d × d matrix to capture them.

For a frozen weight matrix W ∈ ℝ^{d×k}, LoRA represents the update as ΔW = B·A where:

  • A ∈ ℝ^{r×k} (initialized with Gaussian noise)
  • B ∈ ℝ^{d×r} (initialized to zero, so ΔW = 0 at start)
  • r is the rank, typically 4-64 (often 8 or 16)

The forward pass becomes: h = W·x + (α/r)·B·A·x. The scalar α (alpha) controls how strongly the adapter influences the output; a common heuristic is α = 2r, giving an effective scaling of 2.

Parameter count: For a d=4096 projection with r=8, LoRA adds 8·4096 + 4096·8 = 65,536 params vs 4096·4096 = 16.7M for FFT, about 0.4%.

LoRA is typically applied to attention projections (q_proj, k_proj, v_proj, o_proj) and sometimes MLP projections. At inference, you can either keep adapters separate (allowing hot-swapping different tasks) or merge them: W' = W + (α/r)·B·A, getting zero inference overhead.

QLoRA: Quantized Base + LoRA Adapters

QLoRA combines two ideas:

  1. Quantize the frozen base model to 4-bit NF4 (NormalFloat 4-bit, optimized for normally-distributed weights)
  2. Train LoRA adapters in BF16 on top

Plus double quantization (quantizing the quantization constants themselves) and paged optimizers (offloading optimizer state to CPU during spikes).

Result: a 65B model fine-tunable on a single 48 GB GPU. A 7B model fits on a 12 GB consumer card. The trade-off is slower training (~30% overhead from dequantizing weights during forward pass) and slightly lower quality than full-precision LoRA, but the cost reduction is dramatic.

Other PEFT Methods

Prefix Tuning prepends trainable continuous vectors (a "soft prompt") to the keys and values at every attention layer. ~0.1% of parameters. Works but is harder to train than LoRA.

Prompt Tuning is even simpler: only prepend trainable tokens to the input embedding (not at every layer). Only competitive at very large model scales (10B+).

Adapter Layers (Houlsby/Pfeiffer adapters) insert small bottleneck MLPs (down-project → nonlinearity → up-project) between transformer layers. An older approach that adds inference latency unlike merged LoRA.

IA³ (Infused Adapter by Inhibiting and Amplifying Inner Activations) learns three scaling vectors per layer that multiply keys, values, and the MLP intermediate activation. Smallest adapter size (~0.01%), modest quality.

In practice for 2026: LoRA/QLoRA dominate, with DoRA (Weight-Decomposed LoRA) gaining traction for slightly better quality at the same parameter budget.

Practical Fine-tuning

Data preparation matters more than the method. A few thousand high-quality examples often beat hundreds of thousands of noisy ones. Curate ruthlessly. For chat/instruction data, ensure consistent formatting with the model's chat template.

Learning rate: LoRA tolerates higher learning rates than FFT because you're training fewer parameters. Typical ranges:

  • Full fine-tuning: 1e-5 to 5e-5
  • LoRA/QLoRA: 1e-4 to 5e-4 (often 2e-4)

Use cosine decay with a short warmup (3-10% of steps).

Epochs: Usually 1-3. More than 3 epochs on small datasets risks memorization. Watch validation loss; if it diverges from training loss, stop.

Catastrophic forgetting is the risk that fine-tuning on a narrow task destroys general capability. Mitigations:

  • Lower learning rate
  • LoRA at low rank (less expressive change)
  • Mix general data with task data (10-20% replay)
  • Keep epochs low

Instruction Tuning and Chat Fine-tuning (SFT)

Supervised Fine-Tuning (SFT) trains a base model to follow instructions or hold conversations. The data is (prompt, response) pairs, often formatted with a chat template like:

<|user|>What's the capital of France?<|assistant|>Paris.<|end|>

Loss is computed only on the assistant tokens (masking the user portion); otherwise the model wastes capacity learning to generate user queries.

SFT is typically the first stage of post-training, followed by preference optimization (DPO/RLHF) to refine helpfulness, harmlessness, and tone.

Fine-tune vs RAG vs Prompt Engineer: Decision Guide

Need Best Tool
Inject up-to-date or large factual knowledge RAG
Change format, style, or behavior Fine-tune
Improve a specific narrow skill Fine-tune
Quick experiment or one-off task Prompt
Cite sources / show provenance RAG
Reduce prompt length / cost at scale Fine-tune

In real systems, these compose: fine-tune for format/behavior, RAG for facts, prompts to glue them together.

Training Harnesses

You almost never write the training loop from scratch. A training harness wraps the model with data loading, optimizer state, gradient accumulation, checkpointing, distributed strategy (DDP/FSDP/ZeRO), mixed precision, and logging, turning fine-tuning into a config-driven workflow. The current landscape: Axolotl (YAML-config-first, broad model coverage, popular for community LoRA/QLoRA work), TRL (HuggingFace's official trainer, the canonical entry point for SFT/DPO and the reference for new methods), LLaMA-Factory (UI + CLI, broad Chinese-language model support), and for RL-style training OpenRLHF and veRL (full PPO/GRPO/DPO pipelines with vLLM-backed rollouts). Pick on three axes: model coverage, method coverage (SFT only, or also DPO / GRPO / PPO?), and how much config-vs-code you want to write. See Harness Engineering for how this fits alongside eval, serving, and agent harnesses.