💻 computer science

Position: Vision-Language-Action Models Cannot Be Verified to Perform Physical Reasoning

This position paper argues that current evaluation protocols for Vision-Language-Action (VLA) models fail to distinguish between semantic matching and genuine physical reasoning, rendering claims of physical generalization unverifiable and calling for new experimental designs that can causally isolate these distinct capabilities.

Original authors: Taozhao Chen, Ian Manchester, Huaming Chen

Published 2026-07-01

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Taozhao Chen, Ian Manchester, Huaming Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Smart Chef" Who Can't Cook

Imagine a very smart chef who has read every cookbook, watched every cooking show, and memorized millions of food descriptions on the internet. This chef knows exactly what a "spicy tomato soup" looks like, what it smells like, and what the recipe says.

Now, imagine you ask this chef to actually make the soup in a real kitchen.

This paper argues that we are currently giving this chef a passing grade just because they can describe the soup perfectly or point to the right ingredients. We assume that because they know the words and the pictures so well, they must also know how to handle the heavy pot, adjust the heat, and stir without burning their hand.

The authors say: "We haven't actually tested if they can cook yet. We've only tested if they can talk about cooking."

The Problem: Mixing Up "Knowing" with "Doing"

The paper focuses on a type of robot brain called a Vision-Language-Action (VLA) model. These robots use a massive "Vision-Language Model" (VLM)—the part trained on internet photos and text—to tell them what to do.

The authors break the robot's brain into two parts:

The Semantic Map (The "What"): This is the part that recognizes objects. It sees a red ball and knows, "That is a ball. The instruction says 'pick up the ball'." This part is great at understanding language and pictures.
The Physical Decision (The "How"): This is the part that figures out how to grab the ball. Is it slippery? Is it heavy? If I grab it too hard, will it break? This requires understanding physics, weight, and friction.

The Flaw: The paper claims that current robots are built on a hidden assumption: "If the robot understands the picture and the words perfectly, it will automatically know how to move its arm correctly."

The authors say this assumption is unproven. Just because a robot can identify a "heavy, slippery cup" in a photo doesn't mean it knows how much force to use to lift it without dropping it.

Why Can't We Tell the Difference? (The "Scorecard" Problem)

Right now, we test these robots by giving them a task (like "put the cup on the table") and seeing if they succeed. If the cup ends up on the table, we give them a "Success" score.

The paper argues this scorecard is broken because it doesn't tell us why the robot succeeded.

Scenario A: The robot understood the words, figured out the physics, and did a great job. (Real success).
Scenario B: The robot guessed the right action because it saw the same setup in its training data, even if it didn't understand the physics. (Lucky guess).
Scenario C: The robot understood the words perfectly but failed to pick up the cup because it was too heavy. (Physical failure).

The Analogy: Imagine a student taking a math test. If they get the right answer, we assume they know the math. But what if they just memorized the answer key? Or what if they guessed?
Current robot tests are like a test where we only look at the final answer. We don't check if they actually did the math (physics) or just matched the pattern (semantic guessing).

The "Narrative Drift" (The Story That Got Away)

The paper describes a phenomenon called "Narrative Drift."

First Robot: "We made a robot that uses internet knowledge to control arms. It works a little bit!"
Second Robot: "We made a bigger one! It works better! It must be because the internet knowledge is getting better at physics."
Third Robot: "We made an even bigger one! It's amazing! It must be a genius at physics!"

The authors say this is a story we keep telling ourselves. Every time a robot gets a slightly higher score, we assume it's because the robot is getting smarter at physics. But the authors argue we never actually isolated the physics part to prove it. We just kept assuming the "internet knowledge" (the VLM) was doing the heavy lifting, even though internet photos don't teach you how heavy a brick feels.

The Three Levels of Confusion

The paper explains why we can't tell what's happening using three levels:

Level 1 (The Result): We see a success, but we don't know if it was because the robot understood the task or just guessed the right move.
Level 2 (The Source): We don't know if the robot learned from its "internet reading" (VLM) or from its "robot training" (physical practice). They are mixed together.
Level 3 (The Brain): We don't know if the robot actually lost some of its smart "internet reasoning" skills when we taught it to move its arm. Maybe it forgot how to say "no" to impossible tasks because we forced it to just "do" things.

The Solution: A Better Test

The authors don't say these robots are useless. They just say we need to stop guessing. They propose a new way to test them:

The "Controlled Variation" Test:
Instead of just giving the robot a task and seeing if it wins, we need to change one thing at a time:

Test the "What": Keep the physical world exactly the same, but change the words or the pictures. Does the robot still know what to do? (This tests the language part).
Test the "How": Keep the words and pictures exactly the same, but change the physics (e.g., make the object heavier, or the table slippery). Does the robot still succeed? (This tests the physical part).

If we do this, we can finally say: "Okay, the robot is great at understanding words, but it fails when the physics change." Or, "The robot is great at physics, but it gets confused by new words."

The Bottom Line

The paper is a call to stop assuming that knowing about the world (from the internet) is the same as understanding how to move in the world (physics).

Currently, we are praising robots for being good at "talking about" tasks, while assuming they are also good at "doing" tasks. The authors want us to build better tests that separate these two skills, so we can actually figure out if robots are learning to be physical agents or just really good at pattern matching.

In short: Don't assume a robot can lift a heavy box just because it knows the word "heavy." We need to test if it can actually lift it.

Technical Summary: Vision-Language-Action Models Cannot Be Verified to Perform Physical Reasoning

Problem Statement
The paper addresses a foundational, yet unverified, assumption underpinning recent Vision-Language-Action (VLA) research: that semantic generalization learned from internet-scale vision-language pretraining is sufficient to support physical action decisions in robotic systems. While VLA systems (e.g., RT-2, OpenVLA, $\pi^0$ ) have demonstrated improving task success rates on manipulation benchmarks, the authors argue that current evaluation protocols cannot distinguish whether these gains stem from genuine physical reasoning or merely from improved semantic matching and distributional overlap. Consequently, the field lacks the ability to causally attribute performance improvements to either semantic capabilities or physical execution generalization.

Methodology and Structural Argument
The paper employs a theoretical and structural analysis rather than new empirical experiments. The core methodology involves:

Decomposition of VLA Policies: The authors formally decompose a VLA policy into two distinct components:
- Semantic Mapping ( $z_{sem}$ ): The transformation of visual observations and language instructions into task-relevant abstractions (object identity, spatial relations, goals) derived from static image-text data.
- Physical Action Decision: The selection of actions conditioned on physical context ( $z_{phys}$ ), such as mass, friction, and contact geometry, which are not recoverable from static visual data.
Formalization of the "Semantic Sufficiency Assumption": The paper posits that current architectures implicitly assume $\pi(a_t | z_{sem}, z_{phys}) \approx \pi(a_t | z_{sem})$ . The authors argue this is structurally unjustified because the supervision signals for VLM pretraining (static image-text pairs) explicitly exclude the interaction outcomes (force, resistance, dynamics) required to learn $z_{phys}$ .
Analysis of Identifiability: The authors identify three levels of non-identifiability in current evaluation protocols:
- Attribution Non-Identifiability: Task success rates conflate semantic grounding errors with physical execution failures.
- Source Non-Identifiability: Performance gains cannot be traced to VLM pretraining, robot interaction data, or their interaction because "out-of-distribution" (OOD) definitions often ignore VLM pretraining exposure.
- Representation-Level Non-Identifiability: Benchmarks exclude infeasible or unsafe instructions, preventing the detection of whether fine-tuning degrades pretrained capabilities like feasibility judgment.

Key Contributions

Identification of "Narrative Drift": The paper introduces the concept of narrative drift, where successive systems inherit and strengthen prior interpretations of performance gains (e.g., attributing scaling laws to physical reasoning) without isolating the underlying causal mechanism.
Structural Critique of Evaluation: It demonstrates that current benchmarks are structurally incapable of testing the semantic sufficiency assumption because they lack controlled variations that separate semantic content from physical configuration.
Reframing of VLM Roles: The paper proposes that VLM backbones should be viewed as semantic interfaces for grounding language and perception, rather than implicit sources of physical competence. It suggests that physical reasoning may require distinct representational capacities developed through interaction, not just semantic pretraining.

Results and Observations
The paper does not present new experimental results but synthesizes existing literature to reveal a consistent empirical signature:

Misattribution of Failures: When VLA systems fail under novel physical conditions (e.g., handling objects with different masses or friction), these failures are consistently attributed to engineering factors (e.g., insufficient data coverage, model scale, or spurious correlations) rather than the failure of the semantic sufficiency assumption.
Ambiguity of Success: High task success rates are shown to be consistent with multiple explanations, including genuine physical generalization, improved semantic matching, or increased overlap between training and test distributions.
Systemic Consequences: The inability to distinguish these factors leads to misallocation of research resources (e.g., scaling models when the bottleneck is physical interaction), false confidence in deployment readiness, and persistent misdiagnosis of architectural limitations.

Significance and Claims
The paper claims its significance lies in clarifying the conditions under which claims of physical generalization can be meaningfully evaluated. It explicitly states that its goal is not to refute the role of VLMs in robotics, but to show that the assumption of semantic sufficiency has not been independently verified.

Methodological Shift: The authors argue that progress in physical execution generalization requires evaluation designs that introduce controlled variation to separately measure semantic and physical generalization.
Architectural Implications: They propose a shift in design philosophy: instead of assuming physical capability emerges from semantic representations, systems should be designed with explicitly separated components for semantic grounding and physical state modeling.
Empirical Question: The contribution of VLM-based representations to physical performance should be treated as an empirical question to be tested via causal attribution, rather than an assumed premise.

The paper concludes that without evaluation protocols capable of providing identifiable signals for key capabilities, implicit assumptions risk becoming entrenched as de facto premises, potentially obscuring the true path toward general embodied intelligence.