← All papers

Visual Instruction Tuning

Liu, Li, Wu, Lee (Microsoft Research) · 2023 · NeurIPS 2023

MultimodalRead on arXiv

LLaVA wires a frozen CLIP vision encoder to an LLM via a small projection layer and trains it on **GPT-4 generated visual-instruction data**. It became the open-source recipe everyone copied for building multimodal LLMs cheaply.

Key Idea

Multimodal models had been heavy: large joint vision-language pretraining (Flamingo, BLIP-2 with Q-Former). LLaVA shows you can get strong instruction-following VLM behavior by freezing CLIP, freezing the LLM at first, training only a projection, then doing a short instruction-tuning pass, all on a single 8-GPU node, using GPT-4 to bootstrap the training data.

How It Works

  1. Architecture: three pieces:
    • Vision encoder: pretrained CLIP ViT-L/14, outputs patch features for the image
    • Projection: a single linear layer (LLaVA-1.0) or a 2-layer MLP (LLaVA-1.5) that maps CLIP features into the LLM's word-embedding space
    • LLM: Vicuna (LLaMA fine-tuned for chat), receives projected image tokens + text tokens in the standard autoregressive context
  2. Instruction data via GPT-4: given COCO image captions and bounding boxes (text-only, no image), GPT-4 generates ~158K (image, instruction, response) examples covering conversation, detailed description, and complex reasoning. This is the key data trick.
  3. Two-stage training:
    • Stage 1, feature alignment: freeze both CLIP and the LLM, train only the projection on 595K image-caption pairs from CC3M. Teaches the projection to map vision features into "word-like" embeddings.
    • Stage 2, visual instruction tuning: keep CLIP frozen, train projection + LLM on the 158K GPT-4 instruction data. Optionally add ScienceQA for reasoning.

Why It Matters

  • Cheap to train: LLaVA-1.5 13B trains in ~1 day on 8× A100s, a viable academic-budget VLM
  • Strong results: LLaVA-1.5 hits 85.9 on VQAv2, 66.1 on MM-Vet, and is competitive with much heavier closed models on multimodal chat
  • Simple architecture wins: the MLP projector + better data beat more complex cross-attention designs (Q-Former, Perceiver) at comparable scale
  • Open recipe: code, weights, and the GPT-4 instruction data are public, became the template copied by LLaVA-NeXT (1.6), InternVL, MiniGPT-4, ShareGPT4V, Qwen-VL, Idefics, Bunny, Cambrian and most open-source VLMs since
  • Established visual instruction tuning as the standard final stage for multimodal models (analog of SFT for text LLMs)

Key Takeaways for Interviews

  • LLaVA = CLIP + projection + Vicuna, two-stage training (alignment → instruction tuning), the canonical open VLM recipe
  • The projection layer is the only new module; CLIP is frozen throughout, LLM is frozen in stage 1 then tuned in stage 2
  • Data > architecture: GPT-4-distilled visual instruction data did more for quality than architectural complexity, a recurring lesson
  • LLaVA-1.5 upgraded the projector from linear → 2-layer MLP and added academic VQA data; LLaVA-NeXT added high-resolution tiling for OCR-heavy tasks
  • Tradeoff vs cross-attention designs (Flamingo, BLIP-2): simpler and cheaper, but image tokens consume LLM context window, matters for high-res or multi-image inputs
  • In system design: cite LLaVA as the default reference for "how do I bolt vision onto an LLM cheaply"; mention Q-Former / cross-attention as alternatives when context budget is tight