🤖 Vision-Language-Action Models
Foundation models that map camera + instruction directly to robot actions: the bridge from LLMs to physical agency
What is a VLA Model
A Vision-Language-Action (VLA) model is a foundation model that takes a stream of RGB camera images plus a natural-language instruction ("fold the towel", "put the cup on the saucer") and outputs a sequence of robot actions (joint commands, end-effector deltas, or gripper states) that execute the instruction on real hardware. VLAs are the missing third leg of the modality triad: a vision-language model (VLM) sees and describes, an LLM reads and writes, and a VLA acts. The paradigm was crystallized by Google DeepMind's RT-2 (mid-2023), which showed that you could co-fine-tune a 55B vision-language model on web data and robot trajectories, treating actions as just another tokenized output stream. RT-2 inherited the VLM's chain-of-thought, semantic grounding, and zero-shot generalization to objects the robot had never picked up before, a sharp break from prior policies that needed thousands of teleoperated episodes per new object.
The bet underlying the VLA wave of 2024-2026 is simple: the same recipe that built ChatGPT (large pre-trained transformer + diverse internet-scale data + instruction following) will work for robotics, if you can solve the data problem. Robotics has no Common Crawl, but Open X-Embodiment (a 21-institution collaboration released in late 2023) aggregated 1M+ episodes across 22 robot embodiments into a shared format, providing the first credible "ImageNet moment" for manipulation.
Architecture Choices
Single-model VLAs (RT-2, OpenVLA, π0) use one transformer to handle perception, language understanding, and action generation. The vision encoder (typically a SigLIP or CLIP ViT) tokenizes images into patches; the language tokenizer handles the instruction; the model autoregresses over a unified token stream that includes action tokens at the end. This is architecturally clean (one model, one optimizer, end-to-end gradient flow) and inherits everything good about a pretrained VLM (compositionality, instruction following, common-sense grounding). The downside is inference latency: a 7B-parameter VLA on a single GPU runs at maybe 5-20 Hz, which is fine for slow manipulation but marginal for reactive control.
Dual-model "System A / System B" VLAs (Figure's Helix, Gemini Robotics on-device) split the workload: a slow VLM running at 5-10 Hz does high-level scene understanding and sub-goal selection, while a small fast policy (often a few hundred million parameters) runs at 100-500 Hz to produce low-level joint commands conditioned on the VLM's latent. This mirrors Kahneman's "System 1 / System 2" (deliberative perception over here, reactive motor control over there) and lets the model meet real-time deadlines on humanoids and bimanual platforms where 50+ joints need updating every few milliseconds.
Action Representation
The deepest architectural decision in a VLA is how to represent actions.
Discrete action tokens (RT-2, OpenVLA) quantize each action dimension into ~256 bins and treat them as additional tokens in the language vocabulary. Generation is then identical to text generation: autoregressive next-token prediction with cross-entropy loss. This is the simplest path: it reuses the LLM training stack, KV cache, and inference kernels verbatim. But it has two costs. First, the discretization grid limits precision: fine in-hand manipulation that needs sub-millimeter accuracy gets clipped. Second, generating an 8-DOF action one token at a time is slow at high control frequencies.
Continuous actions via flow matching or diffusion (π0, π0-FAST, Diffusion Policy) instead predict a continuous-valued action trajectory directly. π0 (Physical Intelligence, late 2024) uses conditional flow matching: train a network to predict the velocity field that transports samples from a Gaussian prior to the data distribution conditioned on the current image + instruction + robot state. At inference, you integrate the ODE with a few Euler steps (typically 10) to produce a chunk of 50 future actions in one forward pass. The result: π0 outputs continuous joint trajectories at up to 50 Hz on real hardware and can do genuinely dexterous tasks (folding laundry from a basket, bussing a table, bagging groceries, assembling cardboard boxes) that discrete-action VLAs struggle with. π0-FAST added a frequency-domain action tokenizer that compresses the trajectory into fewer tokens, recovering some of the simplicity of token-based decoding while preserving continuous precision.
The empirical story by 2026: discrete tokens win on simplicity and infrastructure reuse; continuous flow-matching wins on dexterity. Most new models being trained at scale (Gemini Robotics, π0 successors, internal Tesla/Figure stacks) use continuous heads.
Training Data
The data bottleneck is the defining constraint of the VLA era. Open X-Embodiment (OXE) pooled 1M+ teleoperated episodes from 22 robot embodiments (Franka arms, ALOHA bimanual, Google's mobile manipulators, xArm, UR5) into a single RLDS-formatted dataset. OpenVLA, π0, and most academic VLAs train on OXE plus task-specific fine-tuning data. π0 layered on a large proprietary corpus of human teleoperation across 7 robot platforms and 68 task families. Gemini Robotics uses DeepMind-internal data plus simulation. Synthetic data from neural world models is just starting to matter; see the connection to world models below.
Cross-Embodiment Transfer
The holy-grail problem: train one VLA on data from many different robots and have it control a new robot zero- or few-shot. OXE was designed for exactly this: its 22 embodiments span 7-DOF arms, 14-DOF bimanuals, mobile bases, and humanoids. OpenVLA showed meaningful cross-embodiment transfer (a model trained on Franka + xArm + ALOHA generalizes partially to a held-out platform), but performance still degrades substantially. The hard question, solved for language but unsolved for action, is what the right embodiment-invariant representation is. End-effector pose? Joint targets? A learned latent? Different bodies have different action spaces, kinematic limits, and contact dynamics; sharing weights across them is more like cross-lingual transfer than cross-domain text transfer.
Real-World Tasks Already Working in 2026
What VLAs can reliably do today (with task-specific fine-tuning on a few hundred to a few thousand demonstrations): pick-and-place across novel objects, opening drawers and doors, wiping a table, folding shirts and towels (π0's signature demo), bagging groceries, simple cardboard-box assembly, basic kitchen prep like pouring and stirring, plug insertion, and bimanual handoffs. What's still hard: dexterous in-hand manipulation (rotating a screwdriver, threading a needle), tasks that need recovery from a failed sub-step, long-horizon assembly with intermediate verification, deformable-object manipulation beyond folding (e.g., tying knots), and any task that requires reasoning about physics the VLM never saw in pretraining.
Open Challenges
Five problems dominate the 2026 research agenda. (1) Generalization to novel objects and environments: VLAs still degrade sharply outside the visual distribution they saw during training. (2) Recovery from failure: current policies execute open-loop chunks of actions and have weak online error correction. (3) Dexterous in-hand manipulation, where multi-finger coordination at high frequency exposes both the data scarcity and the architectural limits. (4) Sim-to-real: simulation is cheap, but transferring policies from MuJoCo / Isaac / Genesis to real hardware still loses substantial performance. (5) Evaluation: there is no MMLU for robots. Each lab evaluates on its own task suite, often on its own hardware, with no shared leaderboard. LIBERO, SIMPLER, and RoboCasa are early attempts; none is canonical.
Connection to World Models
VLAs and world models compose naturally. A neural world model (Genie, V-JEPA, Cosmos) learns to predict future video frames conditioned on actions, effectively a learned simulator. World models can (a) generate synthetic training data by rolling out diverse trajectories in latent space, sidestepping the teleoperation bottleneck; (b) serve as a planner that the VLA queries ("imagine the next 5 frames if I take action a; is that closer to the goal?"), turning a reactive policy into a model-predictive controller. The synthesis of large VLAs with large world models is the most likely path to the kind of long-horizon, recoverable, generalizable behavior that current single-model VLAs lack.