💻 computer science

HumanCLAW: Can Vision-Language Models Act Through a Body?

This paper introduces HumanCLAW, a framework that decouples vision-language model decision-making from motor execution to evaluate embodied intelligence, revealing that current models fail at long-horizon physical tasks primarily due to a lack of embodied self-awareness rather than recognition capabilities.

Original authors: Siyao Li, Jiawei Gu, Shuai Liu, Kairui Hu, Zekun Li, Linjie Li, Chengcheng Tang, Po-Chen Wu, Ivan Shugurov, Lingni Ma, Michael Zollhoefer, Sizhe An, Abhay Mittal, Amy Zhao, Ranjay Krishna, Manling Li

Published 2026-07-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Siyao Li, Jiawei Gu, Shuai Liu, Kairui Hu, Zekun Li, Linjie Li, Chengcheng Tang, Po-Chen Wu, Ivan Shugurov, Lingni Ma, Michael Zollhoefer, Sizhe An, Abhay Mittal, Amy Zhao, Ranjay Krishna, Manling Li, Ziwei Liu, Chuan Guo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a brilliant, super-smart robot to play a video game. But this isn't just any game; it's a game where the robot has to control a real, physical body that can trip, fall, bump into walls, and knock over vases. In the world of artificial intelligence, we have two main types of "brains" for robots. One type is like a pilot who has memorized millions of specific flight paths; it's great at doing exactly what it's seen before, but if the wind blows differently, it might crash. The other type is a "Vision-Language Model" (VLM). Think of this as a robot with a massive library of books and a camera for eyes. It can read a story, look at a picture, and understand complex ideas like "find the red sofa and sit on it." It's amazing at reasoning and talking, but it has never actually moved a body before.

The big question scientists are asking right now is: Can this book-smart robot actually do things in the real world? Can it take a high-level idea like "go to the kitchen" and turn it into the tiny, split-second muscle movements needed to walk without falling over? Usually, when a robot fails, it's hard to tell if the "brain" made a bad decision (like walking into a wall) or if the "body" just couldn't do the job (like losing balance). To solve this mystery, researchers needed a way to separate the thinking from the moving, so they could see exactly where the robot's brain was getting stuck.

This is where a new study called HumanCLAW comes in. The researchers built a special testing ground to see if these smart AI brains could control a human-like body in a physical world. They didn't train the AI to move; they just gave it a camera, a goal, and a list of basic moves like "walk forward," "turn," or "sit." Then, they let the AI try to navigate through 41 different virtual houses to find objects and sit on them.

The results were a bit of a shocker. Even the smartest, most advanced AI models out there today failed almost every time. The best model only succeeded in about 16.8% of the attempts. That means it failed more than 83% of the time.

Here is the weird part: The AI wasn't failing because it was "blind." When the target object (like a sofa) actually appeared on the screen, the AI almost always recognized it. It knew what it was looking at. The problem wasn't seeing; it was knowing where it was.

The researchers discovered that these AI models suffer from a severe case of "body amnesia." They are like a ghost that can see the furniture perfectly but has no idea where its own legs are. When the AI tried to walk, it would often keep walking even after it had already arrived at the sofa, or it would stop way too early, thinking it had arrived when it was still far away. When it tried to sit, it would often just lower its body into thin air, sitting on nothing because it didn't realize the sofa wasn't directly underneath it. It would also walk right into walls or trip over obstacles without ever realizing it had hit them.

The study shows that while these AI models are incredible at describing the world, they are terrible at understanding their own place in it. They can tell you what a chair looks like, but they can't figure out how to sit on it without falling over. The researchers conclude that to make robots that can truly act in the real world, we need to teach them not just to see, but to "feel" where their body is and how it interacts with the world around them. Until we solve this "embodied self-awareness" problem, even the smartest AI brains will remain clumsy ghosts in a physical machine.

Technical Summary: HumanCLAW

Problem Statement

Evaluating whether Vision-Language Models (VLMs) possess "action intelligence"—the ability to decide moment-by-moment what a physical body should do next—is challenging because action outcomes couple decision-making with low-level motor control. In existing evaluations, failures are often ambiguous: it is difficult to distinguish whether a VLM made a poor strategic choice or if the motor controller failed to execute the command (e.g., losing balance). Conversely, end-to-end Vision-Language-Action (VLA) models entangle reasoning with specific control policies, making it hard to isolate the generalist's reasoning capabilities. Furthermore, many agent benchmarks abstract away physical motion, executing commands via pre-defined scripts or simulators that ignore physical consequences like gravity and collisions.

The paper posits that current VLMs lack embodied self-awareness: the capacity to maintain an online estimate of their own body state, its relation to the scene, and the physical consequences of its actions. To address the evaluation gap, the authors introduce HumanCLAW, a framework that decouples action decision-making from low-level execution while preserving physical consequences.

Methodology

1. The HumanCLAW Framework

HumanCLAW creates a closed-loop, sub-second interaction cycle where an off-the-shelf VLM acts as a frozen decision-maker in a physical world. The system operates as follows:

Input: At each timestep, the VLM receives an egocentric RGB observation, a textual history of past decisions, and a high-level instruction (e.g., "find a sofa and sit on it").
Reasoning Harness: The VLM is guided by a contextual reasoning scaffold that structures its output into:
1. Ego-Perception: Explicitly describing visible objects, distances, and obstacles.
2. Mid-Level Planning: Deriving a temporary objective (e.g., "approach the sofa") valid for several seconds.
3. Low-Level Skill Selection: Choosing an atomic skill (e.g., walk, turn, sit) with continuous parameters.
4. Verification: A skill-specific verifier checks if the proposed action is spatially safe (e.g., not walking into a wall) before execution.
Skill-Conditioned Motion Generation: The VLM's atomic skill command is passed to a motion generator. This generator uses a Motion Base DiT (Diffusion Transformer) trained on AMASS data to provide a kinematically continuous prior. A plug-and-play ControlNet adapter, specific to the chosen skill, injects the skill parameters (e.g., stride length, turning angle) to realize the command as a 0.5-second chunk of full-body motion.
Locomotion-Decoupled Physical Simulation: The motion chunk is executed in a "half-physics" simulator (built on AI Habitat and Bullet). The human body is driven by kinematic velocities derived from the generated motion, not by simulated joint torques. This design ensures the agent does not fail due to balance loss or motor tracking errors, but still interacts physically with the world (collisions, gravity, object displacement).

2. HumanCLAW-Bench

The authors construct a benchmark consisting of 1,218 long-horizon, egocentric episodes across 41 indoor scenes (derived from HSSD).

Task: "Find-navigate-interact." The agent must find a target object, navigate to it, and interact (specifically, sit on it for furniture like sofas/beds/toilets, or simply navigate for others).
Difficulty Stratification: Episodes are categorized into Easy, Medium, and Hard based on geodesic distance, navigational choices (turns/rooms), and obstacle density.
Metrics: Evaluation includes staged success rates (Find, Navigate, Interact), body awareness (collision frequency), environmental disturbance (object displacement), action quality (motion jerk), and computational cost.

Key Contributions

HumanCLAW Framework: A controlled setting where an off-the-shelf VLM acts freely in a physical world. The decision is decoupled from motor execution, allowing failures to be attributed strictly to the decision-maker's action intelligence.
HumanCLAW-Bench: A dataset of 1,218 progressive episodes testing long-horizon, closed-loop whole-body decision-making with staged metrics.
Comprehensive Evaluation: An extensive analysis of nine state-of-the-art VLMs, identifying embodied self-awareness as the primary bottleneck rather than perception or motor control.

Experimental Results

The authors tested nine frontier VLMs (including GPT-5.5, Gemini-3.1, Claude-4.8, and various open-source models) on the benchmark.

Low Success Rates: No model solved the benchmark reliably. The best model, Gemini-3.1, achieved a full task success rate (Find $\to$ Navigate $\to$ Interact) of only 16.8%.
Perception vs. Action: Perception is not the bottleneck. Once a target is objectively visible in the egocentric view, models recognize it with high accuracy (within 5 percentage points of visibility). The failure occurs in the transition from seeing to acting.
Navigation Bottleneck: The "landing rate" (ratio of reaching a found target to finding it) is low. The primary failure mode is egocentric self-localization: agents fail to know where their body is relative to the target. They either stop prematurely while far away or continue navigating after arriving.
Interaction Failures: Even when agents reach the target, 81% of interaction failures are due to body placement errors (e.g., sitting on thin air or the wrong object), rather than a failure to decide to sit.
Body Awareness: Agents consistently fail to perceive their own body parts. Collisions concentrate on limbs (legs/feet and arms/hands) that the agent does not "watch," while the head rarely collides. The agents behave like "ghosts," fluent in describing the world but oblivious to their own physical presence.
Ablation Findings:
- Verification is critical: Removing the skill verifier caused navigation success to collapse from 27% to 2%.
- Mid-level reasoning is essential: Removing the mid-level planning step caused interaction success to drop to 0%, as agents could not sustain the multi-step routine required for sitting.
- History length: Longer text history provides diminishing returns; a compact history is sufficient.
- Visual history: Adding too many past frames degrades performance, suggesting the model is bottlenecked by reasoning over the current view.

Significance and Claims

The paper claims that HumanCLAW reveals a fundamental gap in current VLMs: they possess strong spatial reasoning for static scenes but lack embodied self-awareness. They can describe the world but cannot reliably track their own body state, determine if they have arrived at a goal, or anticipate the physical consequences of their movements.

The authors argue that this limitation stems from the training paradigm of VLMs (Visual Question Answering), which teaches models to describe pixels but not to "feel" the body or predict physical consequences. They posit that generalizable action intelligence arises from reasoning rather than fitting action data. By keeping the decision-maker frozen and the motor layer a fixed prior, HumanCLAW demonstrates that current reasoning capabilities do not yet extend to the physical body. The benchmark serves as a stress test for the next generation of agents, highlighting that progress requires models to integrate proprioceptive-like reasoning and spatial self-localization, rather than simply improving perception or collecting more trajectory data.