👁️ Multimodal Models & VLMs
Vision-language models, multi-modal architectures, and cross-modal understanding
What Are Multimodal Models?
Multimodal models process and reason across multiple data types (text, images, audio, video) within a single architecture. Vision-Language Models (VLMs) are the most prominent subclass, combining visual perception with language understanding. GPT-4o, Gemini, Claude's vision, and LLaVA are all VLMs.
Why Multimodal?
The real world is inherently multimodal. Humans don't process text in isolation. We see, hear, read, and reason across modalities simultaneously. Multimodal AI enables:
- Understanding images and answering questions about them (VQA)
- Generating images from text descriptions (text-to-image)
- Analyzing charts, diagrams, and documents with visual layout
- Video understanding and temporal reasoning
- Robotic perception (vision + language instructions)
VLM Architecture Patterns
Pattern 1: Vision Encoder + LLM (Decoder-only)
The dominant architecture. A pre-trained vision encoder (ViT, SigLIP, InternViT) extracts visual features from images, a projection layer maps them into the LLM's token embedding space, and the LLM processes visual tokens alongside text tokens.
Examples: LLaVA, InternVL, Qwen-VL, Phi-3-Vision
Image → Vision Encoder → Projection → [visual tokens] + [text tokens] → LLM → Response
Key design decisions:
- Vision encoder choice: ViT-L/14 (CLIP) vs SigLIP vs InternViT. SigLIP replaces contrastive loss with sigmoid, enabling better per-image understanding. Larger encoders (ViT-G, InternViT-6B) improve spatial understanding.
- Projection layer: Linear projection (LLaVA-1.0), 2-layer MLP (LLaVA-1.5), cross-attention (Flamingo), or perceiver resampler (reducing visual tokens from 576 to 64-128).
- Resolution handling: Higher resolution improves OCR and fine-grained understanding. Approaches: dynamic resolution with image tiling (InternVL, LLaVA-NeXT), multi-scale encoding, or native high-res encoders.
Pattern 2: Early Fusion
Visual and text tokens are mixed from the earliest layers. The model learns cross-modal representations from the ground up rather than bolting a vision encoder onto a text model.
Examples: Fuyu (no vision encoder, raw image patches → transformer), GPT-4o (rumored)
Pattern 3: Cross-Attention
Visual features are injected into the LLM via cross-attention layers interspersed with self-attention, rather than prepending visual tokens to the sequence.
Examples: Flamingo, IDEFICS
Training Pipeline
Stage 1: Pre-training Alignment (Feature Alignment)
Train only the projection layer on large-scale image-caption pairs (e.g., CC3M, LAION). The goal is to align the vision encoder's feature space with the LLM's embedding space. The vision encoder and LLM are frozen.
Stage 2: Instruction Tuning (Visual Instruction Following)
Fine-tune the full model (or LLM + projection, keeping vision encoder frozen) on high-quality visual instruction-following data. This teaches the model to follow complex multi-turn instructions involving images.
Data sources: LLaVA-Instruct (GPT-4-generated), ShareGPT-4V, ALLaVA, domain-specific datasets.
Stage 3: Preference Optimization (Optional)
Apply DPO/RLHF on visual preference data to improve response quality, reduce hallucination, and align with human preferences for visual description tasks.
Visual Grounding & Spatial Understanding
A key challenge: can the model localize objects within an image?
Referring expression comprehension: Given "the red cup on the left," output bounding box coordinates.
Visual grounding: Output <box>x1, y1, x2, y2</box> tokens alongside text. Models like Qwen-VL and CogVLM support coordinate outputs.
Spatial reasoning: Understanding relative positions, counting objects, reading text in images (OCR). Current models are improving but still struggle with precise counting and spatial relationships.
Video Understanding
Extending VLMs to video requires handling temporal information:
- Frame sampling: Select N frames uniformly or with motion-adaptive sampling, encode each with the vision encoder, concatenate visual tokens across frames
- Temporal modeling: Some models add temporal attention layers; others rely on the LLM's sequence modeling to capture temporal relationships
- Efficiency: Video generates massive numbers of visual tokens. Token compression (pooling, perceiver) is essential. A 1-minute video at 1fps with 576 tokens per frame = 34K visual tokens.
Examples: Video-LLaVA, LLaVA-Video, Gemini (native video)
Key Evaluation Benchmarks
| Benchmark | What it Tests |
|---|---|
| MMBench | General multi-modal understanding |
| MMMU | Multi-discipline reasoning (college-level) |
| MathVista | Mathematical reasoning with visual context |
| OCRBench | Text recognition in images |
| RealWorldQA | Real-world spatial/scene understanding |
| HallusionBench | Visual hallucination detection |
| ChartQA | Chart and graph understanding |
| DocVQA | Document visual question answering |
Multimodal Hallucination
VLMs hallucinate visual content, describing objects that aren't in the image or misidentifying spatial relationships. Causes:
- Language prior dominance: the LLM "guesses" based on text patterns rather than attending to visual features
- Low-resolution encoding: fine details are lost during image tokenization
- Training data bias: certain object co-occurrences in training data create spurious correlations
Mitigations: visual DPO (penalizing hallucinated descriptions), RLHF with human visual preference data, contrastive decoding, and improved vision encoders with higher resolution.