💻 computer science

Robots Need More than VLA and World Models

This position paper argues that achieving generalist robot intelligence requires moving beyond simple policy scaling to address the critical bottleneck of converting unstructured behavioral data into grounded robot supervision through four key interfaces: autolabelling, motion retargeting, physics-grounded reasoning, and reward inference.

Original authors: Elis Karcini, Faisal Mehrban, Quang Nguyen, Mac Schwager, Arash Ajoudani, Cesar Cadena, Jan Peters, Marco Hutter, Haitham Bou-Ammar

Published 2026-06-08

📖 6 min read🧠 Deep dive

CC0 1.0

Original authors: Elis Karcini, Faisal Mehrban, Quang Nguyen, Mac Schwager, Arash Ajoudani, Cesar Cadena, Jan Peters, Marco Hutter, Haitham Bou-Ammar

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: Robots Are Stuck in a "Textbook" World

Imagine you are trying to teach a child how to cook. Currently, the best way to do this is to have a master chef stand next to the child, show them exactly how to chop an onion, and have the child copy the hand movements. In robotics, this is called Robot-Native Supervision. We collect data where a robot actually performs the task, and we teach the AI to copy those specific movements.

The paper argues that while this method has made robots smarter, we have hit a wall. We can't just keep collecting more "chef demonstrations" from robots because:

It's incredibly expensive and slow to get robots to do everything.
The real world is full of other ways to learn. There are billions of videos of humans cooking, factories running, and people fixing things on the internet.

The Analogy: Imagine you are trying to learn a new language.

Current Approach (Robot-Native): You only learn from a teacher who speaks only to you in a classroom, using a specific textbook. You never hear the language spoken on the street, in movies, or by friends.
The Paper's Argument: The world is full of people speaking that language (videos, human motion, simulations). But right now, robots can't understand these "street conversations" because they are missing the dictionary and the grammar rules that translate raw noise into something a robot can use.

The authors say: "We don't just need bigger brains (bigger AI models); we need better translators to turn the messy real world into a language robots can learn from."

The Missing "Translation Kit"

The paper proposes that to unlock the next generation of robots, we need four specific tools (or "interfaces") to translate the real world into robot instructions. Think of these as the missing parts of a translation kit.

1. The "Autolabelling Engine" (The Detective)

The Problem: If you watch a video of a human opening a jar, the video shows pixels moving. It doesn't tell the robot: "The hand touched the lid," "The force was 5 Newtons," or "The task is now 'unscrewing'."
The Solution: We need a system that acts like a super-detective. It watches messy videos, human motion sensors, or robot failures and automatically writes down the important notes.

What it does: It turns a blurry video of a person dropping a cup into a structured report: "Event: Contact lost. Object: Cup. State: Broken. Task Phase: Failure."
Why it matters: It turns raw, unorganized footage into a "textbook" the robot can actually study.

2. The "Retargeting Interface" (The Translator)

The Problem: Humans have two arms and five fingers. A robot might have a single claw or a different number of joints. If you just tell a robot to "copy the human's arm angle," it might break its own arm or fail to pick up the object.
The Solution: We need a translator that doesn't copy movements, but copies results.

The Analogy: Imagine you want to open a door. You don't care if the human pushes it with their elbow or their hand; you care that the door opens.
What it does: It looks at what the human achieved (the door opened) and figures out how this specific robot can achieve that same result with its own unique body. It preserves the "goal" but changes the "how."

3. The "Physics-Grounded World Model" (The Simulator)

The Problem: Some AI models are great at making pretty videos of the future (e.g., "The cup will fall"). But they might ignore physics. They might show the cup falling through the table or floating in the air. A robot needs to know if the cup will actually break or if it will slide.
The Solution: We need a "crystal ball" that doesn't just guess what the picture will look like, but predicts the physics.

What it does: It answers questions like: "If I push this block here, will it tip over? Will it hit the wall? Will the friction be enough to stop it?"
Why it matters: It allows the robot to "imagine" different outcomes and learn from mistakes without actually breaking anything in the real world.

4. The "Reward Grounding Loop" (The Coach)

The Problem: When a robot fails, it just sees a video of a mess. It doesn't know why it failed. Was it too slow? Did it push too hard? Did it misunderstand the goal?
The Solution: We need a system that acts like a sports coach. It watches the robot's attempt and gives specific feedback based on the goal.

The Analogy: If a basketball player misses a shot, a generic observer just says "Missed." A coach says, "You released the ball too early, and your elbow was out."
What it does: It looks at the outcome and says, "You failed because you didn't apply enough force to the handle," or "You succeeded because you aligned the cup correctly." This turns a failure into a specific lesson for the next try.

The Main Takeaway

The paper concludes that the future of robotics isn't just about building bigger, smarter AI models (VLAs) that can talk and see. It's about building the infrastructure that connects those models to the messy, physical reality.

The Final Metaphor:
Think of the current state of robotics as having a brilliant student (the AI model) who is sitting in a library with only one very specific, old textbook (robot demonstrations). The student is smart, but they are limited by the book.

The paper argues that the real world is a massive, noisy, chaotic library with billions of books, videos, and experiences. To let the student learn from this massive library, we don't just need a bigger student. We need:

Librarians to organize the messy books (Autolabelling).
Translators to explain how to read them (Retargeting).
Mental Maps to understand the stories inside (World Models).
Tutors to grade the student's homework and explain the errors (Reward Grounding).

Without these four tools, the robot will remain stuck in its small, expensive library, unable to learn from the vast, wonderful, and messy world around it.

Technical Summary: Robots Need More Than VLAs & World Models

Problem Statement

The paper argues that the current framing of generalist robot intelligence as a "policy-scaling problem"—collecting more robot demonstrations and training larger Vision-Language-Action (VLA) models—is incomplete. While large-scale VLA models and cross-embodiment datasets have shown promise, they face a fundamental scaling bottleneck: the absence of mechanisms to convert the world's abundant unstructured behavioral data into grounded robot supervision.

Physical interaction data (human motion, internet video, factory workflows, simulation rollouts) contains rich information about tasks, goals, contacts, and failures. However, this data lacks the specific structure required for robot learning: embodiment-specific action labels, force signals, task semantics, and reward structures. Unlike text and images, which are naturally digitized and densely supervised, physical experience is "messy" and not directly executable by robot policies. The central challenge is not merely how to scale policies, but how to transform broad, weakly labeled physical experience into robot-usable supervision (actions, contacts, object states, goals, and rewards).

Methodology and Proposed Framework

Rather than proposing a single new algorithm or dataset, the paper surveys the current landscape of robot learning to identify gaps and proposes a grounding-centric pipeline as the necessary next step. The authors organize the field not by model architecture, but by the supervision bottleneck each approach exposes. They argue that the future robotics stack requires four missing components to bridge the gap between raw physical experience and deployable intelligence:

1. Physical Data Engines and Embodied Autolabelling

Current robot learning relies on "robot-native" supervision where observations are already paired with actions. The paper proposes a Physical Data Engine to ingest heterogeneous, asynchronous streams (video, wearable sensing, tactile data, robot logs, language) and automatically infer structured, robot-relevant labels.

Mechanism: This engine performs embodied autolabelling, inferring latent physical events ( $\zeta$ ) from raw data ( $x$ ). It aligns asynchronous modalities to a common timeline and recovers variables such as object-centric states ( $s$ ), contact events ( $c$ ), task phases ( $\phi$ ), latent physical actions ( $u$ ), and reward/progress signals ( $r$ ).
Goal: To convert raw, unstructured episodes (e.g., a human wearing a sensing suit) into a sequence of physical events that can be retargeted, simulated, or used for policy training, effectively turning "failures" and "corrections" into structured supervision.

2. Task-Preserving Retargeting

Once physical events are inferred, they must be translated into executable actions for a specific robot embodiment. The paper critiques simple pose-matching approaches, which fail to account for kinematic and dynamic differences between humans and robots.

Mechanism: The authors propose task-preserving retargeting, which maps latent physical actions or human demonstrations into executable robot actions ( $a^{(embodied)}$ ) while preserving the task-relevant physical effect ( $\Delta g$ ).
Goal: Instead of copying human joint trajectories, the system should ensure that the robot achieves the same physical outcome (e.g., drawer displacement, object alignment, contact establishment) regardless of the specific morphology used.

3. Physics-Grounded World Models

Current world models often focus on visual plausibility (generating realistic future frames). The paper argues that robot world models must be physics-grounded to predict actionable consequences.

Mechanism: These models must predict structured physical variables (geometry, contact, forces, constraints, material properties) rather than just pixels. They should answer "what would happen if" questions regarding object states, stability, and task progress.
Goal: To provide counterfactual experience. A robot needs to reason about whether an action will cause a slip, a jam, or a successful insertion. The model must be calibrated to estimate uncertainty and preserve the physical variables that determine success or failure.

4. Self-Improving Deployment Loops

The final component is a closed-loop system where real-world deployment generates new data for continuous improvement.

Mechanism: This involves task-conditioned reward grounding. Instead of treating deployment outcomes as simple pass/fail records, the system must interpret outcomes relative to the specific goal (e.g., distinguishing between a failed grasp and a successful one based on contact and object state).
Goal: To create a compounding system where failures, human corrections, and successful rollouts are fed back into the Physical Data Engine to update the policy, reward model, world model, and retargeting mechanisms. This transforms isolated engineering efforts into a system that learns from the physical world itself.

Key Contributions

Reframing the Bottleneck: The paper identifies the central bottleneck in generalist robotics not as a lack of policy capacity or data volume, but as the lack of grounding mechanisms to convert unstructured physical experience into robot-usable supervision.
Taxonomy of Supervision: It provides a structured analysis of current approaches (robot-native datasets, weakly grounded video learning, simulation, and world models), highlighting that while each offers partial solutions, none fully solve the grounding problem.
The Four-Pillar Architecture: The paper proposes a specific architectural stack comprising:
- Physical Data Engines (for autolabelling).
- Task-Preserving Retargeting (for embodiment transfer).
- Physics-Grounded World Models (for consequence prediction).
- Self-Improving Deployment Loops (for reward grounding and continuous learning).
Shift from Feed-Forward to Closed-Loop: It argues that the next generation of robotics must move from feed-forward pipelines (collect data $\to$ train policy) to closed-loop systems where deployment traces continuously refine the underlying models.

Results and Evidence

As a position paper, the work does not present new experimental results or a single unified system. Instead, its "results" are derived from a comprehensive survey of recent literature:

Robot-Native Scaling: It cites progress in datasets like Open X-Embodiment, DROID, and RH20T, and models like RT-2, Octo, and $\pi^0$ , noting that while these show scaling benefits, they remain dependent on pre-grounded, robot-native data.
Weak Supervision: It reviews works on learning from video (R3M, VIP, LAPA, UniVLA) and reward inference (PROGRESSOR, ReWiND), demonstrating that while these methods extract representations or latent actions, they still struggle with the correspondence problem and lack direct robot action labels.
Simulation and World Models: It surveys simulation environments (RLBench, ManiSkill) and learned world models (Dreamer, Genie, 3D-Gaussian Splatting approaches), arguing that while they generate experience, they often lack the physical fidelity (contact, force, stability) required for reliable control.
Synthesis: The paper synthesizes these findings to demonstrate that no single existing approach solves the full grounding pipeline, justifying the need for the proposed four-component architecture.

Significance and Claims

The paper claims that the field is entering a "foundation-model moment" but lacks the "internet" equivalent for robotics—the ability to leverage the vast, unstructured physical data of the world.

Beyond VLAs: The authors assert that VLAs are necessary but insufficient. They should be viewed as one layer in a larger physical-intelligence stack that depends on upstream grounding (data, embodiment, dynamics) and downstream feedback (rewards, deployment).
The Grounding Agenda: The significance of the paper lies in shifting the research agenda from "which model architecture to train" to "which mechanisms are needed to make the world's physical experience learnable."
Future Direction: The paper posits that the next foundation model for robotics will not be a monolithic VLA, but a compounding system that integrates autolabelling, retargeting, physics-grounded prediction, and deployment feedback. This approach aims to move robotics from reliance on curated, robot-native datasets to learning directly from the broader, messy physical world.