🌍 World Models
Generative spatial intelligence: how AI builds and reasons about navigable 3D worlds
What Is a World Model?
A world model is a learned system that captures the structure, dynamics, and visual appearance of an environment well enough that an agent (or a human) can query it, navigate it, or plan inside it. Unlike an LLM, which models the distribution of text, a world model must model space, geometry, physics, and time. It must answer questions like: if the camera moves left, what should the next frame look like? If I push this cup, will it fall? If I open this door, what room is behind it?
The 2025 to 2026 wave (Google DeepMind's Genie 3, World Labs' Marble, NVIDIA's Cosmos, and Meta AI's V-JEPA 2) turned world models from a research curiosity (Ha & Schmidhuber, 2018) into deployable infrastructure for robotics, autonomous driving, gaming, and embodied AI.
Fei-Fei Li's Functional Taxonomy (June 2026)
In her June 2026 essay "A Functional Taxonomy of World Models", Fei-Fei Li argued that the field's confusion comes from collapsing three distinct functions into one label. She decomposes any world model into three roles, each plugged into the classic POMDP agent loop (agent → action → state → observation → agent):
1. Renderer. Generates observations (pixels, sensor readings) conditioned on state. The quality bar is visual fidelity for human consumption. Video-generation models and Genie 3 are renderers: they can produce convincing frames of a forest or a city even when the underlying geometry is structurally implausible (a building may look perfect from the street but collapse if you walked around it).
2. Simulator. Outputs state with geometric, physical, or dynamical accuracy. The audience is dual: humans who need correctness (architects, AV engineers) and programs that compute on the output (RL agents, robot controllers, physics engines). Simulators are the under-served middle of the taxonomy and, Li argues, the most consequential for embodied AI. NVIDIA Cosmos and World Labs' Marble (with its collision meshes) target this role.
3. Planner. Outputs actions that move the agent toward a goal in the world model. A planner is "the inverse of a renderer": given an observation and a goal, produce the next action. Vision-Language-Action (VLA) models and World Action Models live here.
Li's deeper claim is that these three are projections of a single underlying understanding of the world. Future foundation models will likely fuse them into a single backbone that can render, simulate, and plan, but today's systems specialize in one role and approximate the others.
The 2026 Landscape
Genie 3 (DeepMind, Aug 2025). The first real-time, general, interactive world model. Generates 720p navigable environments at 24 fps from a text prompt, with visual memory of roughly one minute. Promptable events ("now it starts raining") let users steer the simulation. SIMA agents were trained inside Genie 3 to learn longer action horizons. In February 2026, Waymo specialized this lineage into a Waymo World Model for AV sim: a closed-loop simulator that can replay or counterfactually edit driving scenes for policy training. Genie is overwhelmingly a renderer in Li's taxonomy.
Marble (World Labs, Nov 2025). Fei-Fei Li's own company shipped Marble as a multimodal world model that takes text, images, video, or sketches and outputs explorable 3D Gaussian splat scenes plus collision meshes that a physics engine can ingest. Unlike Genie, Marble's output is persistent geometry, not streamed frames: you can load it into Unity, ingest it into a robot simulator, or share it as a 3D file. Marble straddles renderer and simulator.
NVIDIA Cosmos (CES 2025). Open infrastructure (model zoo + tokenizers + data pipeline) for "physical AI." 2M+ downloads by mid-2026; widely used as a foundation by AV and robotics teams who fine-tune Cosmos checkpoints on their own data rather than training a world model from scratch.
V-JEPA / V-JEPA 2 (Meta AI, LeCun's AMI Labs, 2026). Yann LeCun's bet: world models should predict in a latent space, not in pixel space. JEPA (Joint-Embedding Predictive Architecture) trains a context encoder, a target encoder, and a predictor that maps from context embedding + action to predicted target embedding. The loss lives entirely in latent space; you never reconstruct pixels. LeCun's argument: pixel-level generation wastes capacity modeling visually irrelevant detail (every leaf, every texture variation) when the only thing that matters for planning is whether the next abstract state is predicted correctly. V-JEPA 2 reports strong action-conditioned prediction on egocentric video and is the basis for the AMI Labs research agenda LeCun launched after departing Meta's main FAIR org.
Why World Models Are Not LLMs
LLMs model token sequences in a discrete vocabulary. World models must handle:
- Spatial reasoning: an object seen from the front and from above is the same object (view-consistency).
- Geometric coherence: parallel lines, occlusion, depth, scale.
- Physical plausibility: gravity, contact, friction, rigid-body dynamics.
- Persistent state: an object behind you should still exist when you turn back.
- Long-horizon consistency: the room you left ten minutes ago should not have rearranged itself.
- Action conditioning: the next observation depends not just on the past but on what you (or another agent) chose to do.
Transformers help, but raw next-token prediction on video frames is brutally inefficient. Most 2026 systems combine a tokenizer (VQ-VAE, Gaussian-splat encoder, or latent-diffusion VAE), an autoregressive or diffusion-based dynamics model, and some form of action conditioning.
Output Representations
| Format | Used by | Strengths | Weaknesses |
|---|---|---|---|
| Gaussian splats | World Labs Marble | Persistent, editable, fast to render, web-deployable | Hard to animate, large file sizes |
| Video frames | Genie 3, Cosmos | Maximum visual fidelity, generic | No persistent geometry, expensive at long horizons |
| Latent embeddings | V-JEPA | Compact, fast to predict, planning-friendly | Not human-interpretable, requires a decoder for any visual output |
| Volumetric grids / NeRFs | Older (2021-2023) systems | Continuous geometry | Slow rendering, hard to edit |
Applications by 2026
- Robot learning: sim-to-real transfer. Train a VLA in Marble-generated kitchens, deploy on a real Stretch robot.
- Autonomous driving: Waymo World Model lets a policy collide with a counterfactual pedestrian without anyone getting hurt.
- Interactive entertainment: Genie 3 prompts as a content pipeline for prototype games.
- VLM-grounded planning: an LLM proposes a plan, the world model rolls it out, the LLM revises.
- Training data for VLAs: synthetic trajectories generated in a world model are cheaper than real-robot data and have ground-truth state labels.
Open Challenges
Physical plausibility. Renderers regularly hallucinate non-physical motion (objects passing through walls, water flowing uphill). Even Genie 3's promptable events sometimes violate conservation laws.
Long-horizon consistency. Genie 3 holds visual memory for about a minute; beyond that, scenes drift. Persistent-geometry approaches (Marble) sidestep this but pay in editability.
Evaluation. There is no "BLEU for worlds." Common metrics (FVD for video, PSNR for reconstructions, success rate for downstream policies) each miss something. The community is still inventing benchmarks (WorldSim, GenieEval, JEPA-Probe).
Compute. Real-time inference at 720p / 24 fps is barely tractable today; 4K stereo for XR is not. Distillation and caching are active research.
Action conditioning. Most published world models accept low-dimensional actions (joystick, joint torques). Conditioning on rich natural-language actions or multi-agent action distributions is still open.