Fine-tuning Strategies

Fine-tuning adapts a pretrained model to a specific task or domain by training further on targeted data, often parameter-efficiently with LoRA or QLoRA rather than updating all weights. Interviews test when to fine-tune versus prompt or use RAG, and how to avoid catastrophic forgetting and overfitting.

Why Fine-tune?

Pre-trained foundation models are trained on broad internet data. They know a lot but they don't know your domain, your tone, or your task format. Fine-tuning specializes the model: a generic chat model becomes a medical-coding assistant, a coding assistant learns your internal APIs, or a base model becomes an instruction follower.

Fine-tuning is the right answer when:

You need a consistent output format the model keeps drifting from with prompting
Latency or cost forbids long few-shot prompts
You need style/voice transfer that prompts can't reliably capture
The task requires implicit knowledge that doesn't fit in context (large vocabulary, specialized syntax)

It is the wrong answer when the underlying need is "the model should know X"; that's retrieval (RAG). And it's overkill when a well-crafted prompt already works.

Full Fine-tuning vs Parameter-Efficient Methods

Full fine-tuning (FFT) updates every parameter. For a 7B model in BF16 with AdamW, you need roughly:

14 GB for weights
14 GB for gradients
28 GB for optimizer state (two moments)
Activations on top of that

Total: ~70-80 GB of VRAM, before activations. That's why FFT for 7B+ models needs A100/H100-class hardware or model parallelism.

Parameter-Efficient Fine-Tuning (PEFT) freezes the base model and trains a small number of new parameters (often <1% of the total). This collapses memory requirements by 10-100× and produces a tiny adapter you can swap in/out of the frozen base model.

LoRA: Low-Rank Adaptation

LoRA is the dominant PEFT method. The insight: weight updates during fine-tuning have low intrinsic rank, so you don't need a full d × d matrix to capture them.

For a frozen weight matrix W ∈ ℝ^{d×k}, LoRA represents the update as ΔW = B·A where:

A ∈ ℝ^{r×k} (initialized with Gaussian noise)
B ∈ ℝ^{d×r} (initialized to zero, so ΔW = 0 at start)
r is the rank, typically 4-64 (often 8 or 16)

The forward pass becomes: h = W·x + (α/r)·B·A·x. The scalar α (alpha) controls how strongly the adapter influences the output; a common heuristic is α = 2r, giving an effective scaling of 2.

Parameter count: For a d=4096 projection with r=8, LoRA adds 8·4096 + 4096·8 = 65,536 params vs 4096·4096 = 16.7M for FFT, about 0.4%.

LoRA is typically applied to attention projections (q_proj, k_proj, v_proj, o_proj) and sometimes MLP projections. At inference, you can either keep adapters separate (allowing hot-swapping different tasks) or merge them: W' = W + (α/r)·B·A, getting zero inference overhead.

QLoRA: Quantized Base + LoRA Adapters

QLoRA combines two ideas:

Quantize the frozen base model to 4-bit NF4 (NormalFloat 4-bit, optimized for normally-distributed weights)
Train LoRA adapters in BF16 on top

Plus double quantization (quantizing the quantization constants themselves) and paged optimizers (offloading optimizer state to CPU during spikes).

Result: a 65B model fine-tunable on a single 48 GB GPU. A 7B model fits on a 12 GB consumer card. The trade-off is slower training (~30% overhead from dequantizing weights during forward pass) and slightly lower quality than full-precision LoRA, but the cost reduction is dramatic.

Other PEFT Methods

Prefix Tuning prepends trainable continuous vectors (a "soft prompt") to the keys and values at every attention layer. ~0.1% of parameters. Works but is harder to train than LoRA.

Prompt Tuning is even simpler: only prepend trainable tokens to the input embedding (not at every layer). Only competitive at very large model scales (10B+).

Adapter Layers (Houlsby/Pfeiffer adapters) insert small bottleneck MLPs (down-project → nonlinearity → up-project) between transformer layers. An older approach that adds inference latency unlike merged LoRA.

IA³ (Infused Adapter by Inhibiting and Amplifying Inner Activations) learns three scaling vectors per layer that multiply keys, values, and the MLP intermediate activation. Smallest adapter size (~0.01%), modest quality.

In practice for 2026: LoRA/QLoRA dominate, with DoRA (Weight-Decomposed LoRA) gaining traction for slightly better quality at the same parameter budget.

Practical Fine-tuning

Data preparation matters more than the method. A few thousand high-quality examples often beat hundreds of thousands of noisy ones. Curate ruthlessly. For chat/instruction data, ensure consistent formatting with the model's chat template.

Learning rate: LoRA tolerates higher learning rates than FFT because you're training fewer parameters. Typical ranges:

Full fine-tuning: 1e-5 to 5e-5
LoRA/QLoRA: 1e-4 to 5e-4 (often 2e-4)

Use cosine decay with a short warmup (3-10% of steps).

Epochs: Usually 1-3. More than 3 epochs on small datasets risks memorization. Watch validation loss; if it diverges from training loss, stop.

Catastrophic forgetting is the risk that fine-tuning on a narrow task destroys general capability. Mitigations:

Lower learning rate
LoRA at low rank (less expressive change)
Mix general data with task data (10-20% replay)
Keep epochs low

Instruction Tuning and Chat Fine-tuning (SFT)

Supervised Fine-Tuning (SFT) trains a base model to follow instructions or hold conversations. The data is (prompt, response) pairs, often formatted with a chat template like:

<|user|>What's the capital of France?<|assistant|>Paris.<|end|>

Loss is computed only on the assistant tokens (masking the user portion); otherwise the model wastes capacity learning to generate user queries.

SFT is typically the first stage of post-training, followed by preference optimization (DPO/RLHF) to refine helpfulness, harmlessness, and tone.

Fine-tune vs RAG vs Prompt Engineer: Decision Guide

Need	Best Tool
Inject up-to-date or large factual knowledge	RAG
Change format, style, or behavior	Fine-tune
Improve a specific narrow skill	Fine-tune
Quick experiment or one-off task	Prompt
Cite sources / show provenance	RAG
Reduce prompt length / cost at scale	Fine-tune

In real systems, these compose: fine-tune for format/behavior, RAG for facts, prompts to glue them together.

Training Harnesses

You almost never write the training loop from scratch. A training harness wraps the model with data loading, optimizer state, gradient accumulation, checkpointing, distributed strategy (DDP/FSDP/ZeRO), mixed precision, and logging, turning fine-tuning into a config-driven workflow. The current landscape: Axolotl (YAML-config-first, broad model coverage, popular for community LoRA/QLoRA work), TRL (HuggingFace's official trainer, the canonical entry point for SFT/DPO and the reference for new methods), LLaMA-Factory (UI + CLI, broad Chinese-language model support), and for RL-style training OpenRLHF and veRL (full PPO/GRPO/DPO pipelines with vLLM-backed rollouts). Pick on three axes: model coverage, method coverage (SFT only, or also DPO / GRPO / PPO?), and how much config-vs-code you want to write. See Harness Engineering for how this fits alongside eval, serving, and agent harnesses.