💻 computer science

Does VLA Even Know the Basics? Measuring Commonsense and World Knowledge Retention in Vision-Language-Action Models

This paper introduces Act2Answer, a novel evaluation protocol that adapts VLM benchmarks into action-based tabletop tasks to systematically measure and analyze knowledge retention in Vision-Language-Action models, revealing that while these models maintain basic concepts, they exhibit significant gaps in richer semantic knowledge compared to their source VLMs.

Original authors: Nikita Kachaev, Andrey Moskalenko, Matvey Skripkin, Nikita Kurlaev, Daria Pugacheva, Albina Burlova, Mikhail Kolosov, Denis Shepelev, Andrey Kuznetsov, Elena Tutubalina, Aleksandr I. Panov, Alexey K.

Published 2026-06-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nikita Kachaev, Andrey Moskalenko, Matvey Skripkin, Nikita Kurlaev, Daria Pugacheva, Albina Burlova, Mikhail Kolosov, Denis Shepelev, Andrey Kuznetsov, Elena Tutubalina, Aleksandr I. Panov, Alexey K. Kovalev, Vlad Shakhuro

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant student who has read every book in the library and knows the world inside out. You call them "The Knowledgeable One." Then, you hire them to work as a robot butler. You teach them how to pick up cups, open doors, and move furniture.

The big question this paper asks is: After all that training on how to move their hands, does the robot butler still remember the facts they learned from the books? Or did the "how to move" training accidentally wipe their brain clean?

The authors of this paper, led by Nikita Kachaev and colleagues, decided to find out. They built a new test called Act2Answer (Action to Answer).

The Problem: The "Success" Trap

Usually, when we test robots, we ask: "Did the robot successfully pick up the cup?"

If the robot picks up the cup, we say, "Great job!"
If it drops the cup, we say, "Bad job."

But this is like grading a student only on whether they can write their name, without checking if they actually know the answer to the math problem. A robot might drop a cup because its hand is clumsy (bad motor skills), not because it doesn't know that the cup is full of hot coffee and shouldn't be touched. The old tests mix up "clumsy hands" with "empty brains."

The Solution: The "Action Quiz"

To fix this, the researchers created a game called Act2Answer.

Imagine a table with two pictures on it:

A picture of a sad person.
A picture of a happy person.

The robot gets a voice command: "Put the cube on the sad person."

Instead of the robot talking back and saying "The sad person is on the left," the robot has to physically move a cube and place it on the picture of the sad person.

If it puts the cube on the sad picture, it gets a point.
If it puts it on the happy picture, it gets zero.

This is a "low-stakes" test. The robot doesn't need to walk across the room or lift a heavy sofa. It just needs to point with a cube. This isolates the robot's knowledge from its clumsiness.

What They Found: The "Simple vs. Complex" Gap

The researchers tested 7 different robot brains (VLA models) and compared them to their "parent" versions (the smart text-and-image models before they were trained to move).

Here is the story their data tells:

1. The Robot is Great at "What" (Simple Stuff)
If you ask the robot, "Is this a red ball or a blue ball?" or "Is this a circle or a square?", the robots are almost perfect. They can still see colors and shapes perfectly. It's like the robot remembers how to recognize a stop sign.

2. The Robot is Lost at "Why" and "Who" (Complex Stuff)
But when the questions get deeper, the robots start to fail.

Emotions: They struggle to tell the difference between a "sad" face and a "happy" face.
Social Rules: They don't seem to know that you shouldn't put a cube on a picture of a baby if the instruction implies safety.
Facts: They forget who famous people are or how to count objects correctly.

It's as if the robot learned to move its arm so well that it forgot how to be a human. The "clumsy hands" training seems to have overwritten the "smart brain" training for anything that isn't a simple shape or color.

3. The "Parent" vs. The "Child"
The researchers found a huge gap between the "Parent" (the smart text model) and the "Child" (the robot).

The Parent could answer 90% of the complex questions correctly.
The Child (the robot) often dropped to near 50% (which is just guessing).

This suggests that when we turn a smart AI into a robot, we often lose a lot of its common sense.

The "Hidden Knowledge" Mystery

The researchers also peeked inside the robot's brain (its internal layers) to see if the knowledge was truly gone or just hidden.

They found that the knowledge was still there in the middle layers of the robot's brain. The robot "knew" the answer internally. But when it tried to translate that knowledge into a physical action (moving the cube), the signal got lost.

The Analogy: Imagine a person who knows the answer to a riddle but has a speech impediment that makes them stutter so badly they can't say it out loud. The knowledge is there, but the "action" part of the brain can't access it.

The Takeaway

The paper concludes that simply teaching a smart AI how to move a robot arm isn't enough. We are currently training robots to be great at "doing" but terrible at "thinking" about the world.

To build a truly helpful robot assistant, we need to figure out how to teach them to move without making them forget the common sense they started with. The paper suggests that keeping the robot connected to language and world knowledge while it learns to move might be the key, but right now, that balance is missing.

In short: The robots can pick up the cup, but they might not remember that the cup is hot, or that the person holding it is sad. They are excellent movers, but they are forgetting how to be smart.

Technical Summary: Does VLA Even Know the Basics? Measuring Commonsense and World Knowledge Retention in Vision–Language–Action Models

Problem Statement

Embodied Vision–Language–Action (VLA) models are typically constructed by fine-tuning powerful pretrained Vision-Language Models (VLMs) on robotics data. While these models are often advertised as open-world-generalizable, it remains unclear how much commonsense and factual knowledge they retain after adaptation to control tasks. Current evaluation paradigms are predominantly "manipulation-centric," focusing on task success rates in navigation or object manipulation (e.g., LIBERO, CALVIN). These benchmarks conflate knowledge deficits with failures in low-level motor control, perception, or long-horizon planning. Consequently, there is no principled method to determine whether a VLA has catastrophically forgotten the world knowledge present in its source VLM, or if failures on knowledge-sensitive tasks are merely due to poor generalization of control policies.

Methodology: Act2Answer

To address the ambiguity between knowledge loss and control failure, the authors introduce Act2Answer, a lightweight evaluation protocol that adapts VLM knowledge benchmarks for VLA models by requiring agents to answer through action rather than text generation.

Core Protocol

Action-Grounded Selection: Instead of generating text, each question is converted into a short, simulated tabletop episode. The agent receives a natural language instruction and must perform a single, simple object-placement action (moving a cube) to select the correct answer among candidate images placed on a table.
Reduced Confounds: By limiting the task to a single, short-horizon action, the protocol minimizes the influence of complex motor control, long-horizon planning, and environmental dynamics, isolating the agent's ability to use knowledge for decision-making.
Soft Success Rate (SR): Performance is measured using a "Soft Success Rate," which accounts for a tolerance region around the target answer. This metric distinguishes between:
1. Instruction/Perceptual Failure: The agent places the object out of bounds or consistently chooses the wrong option (SR < 0.5 - $\Delta$ ).
2. No Reliable Usable Knowledge: The agent grounds the action but fails to select the correct option, performing near random chance (0.5 - $\Delta$ $\le$ SR $\le$ 0.5 + $\Delta$ ).
3. Evidence of Usable Knowledge: The agent successfully leverages internal knowledge to skew action selection toward the correct target (SR > 0.5 + $\Delta$ ).

Data Curation

The authors curated a test suite of 1,720 unique binary questions across 12 knowledge categories, derived from existing VLM benchmarks (MLLM-CompBench, IconQA, MMBench, OK-VQA, VL-Think). These categories are grouped into seven broad domains:

Physical: Attributes, states, color, shape, symmetry.
Temporal: Time, counting.
Quantitative: Counting, magnitude.
Biological: Living world distinctions.
Social: Emotion, intent.
Normative: Traffic rules, public info.
Cultural: Celebrity recognition.

Layerwise Intent Probing

To investigate where knowledge is retained or lost within the model architecture, the authors employ layerwise intent probing. Linear classifiers are trained on hidden states from every transformer layer (both the VLM backbone and the Action Expert) to predict the correct answer. This allows for the quantification of how answer-relevant information is distributed and whether it attenuates as it passes from semantic representation to action generation.

Key Contributions

Act2Answer Benchmark Suite: A controlled protocol that converts VLM knowledge tasks into embodied action-selection episodes, enabling the direct probing of knowledge-sensitive behavior without relying on text decoding.
Systematic Knowledge Taxonomy: A curated dataset covering 12 categories across 7 domains, facilitating granular error analysis of commonsense and world knowledge in embodied agents.
Large-Scale Empirical Study: An evaluation of 7 modern VLA models (including $\pi_0$ , OpenVLA, Magma, SpatialVLA) against 9 strong VLM baselines, systematically ranking performance across knowledge domains.
Architectural Analysis: Introduction of layerwise intent probing to localize answer-relevant information, revealing the internal dynamics of knowledge retention versus action generation.

Results

The study reveals a significant disparity between VLM capabilities and their VLA counterparts:

Perceptual vs. Semantic Gap: VLA models perform strongly on simple perceptual categories (e.g., Color, Shape) but exhibit substantial performance drops on richer semantic categories (e.g., Emotion, State, Counting, Symmetry). Many VLA models perform at or near random chance on these complex tasks.
The VLM-VLA Gap: When compared to their source VLMs in an action-free setting, VLA models show a performance drop of roughly 20–40 percentage points across most knowledge domains. This suggests that fine-tuning for control significantly degrades the accessibility of world knowledge.
Role of VQA Co-training: Models trained with joint vision-language and robotics supervision (e.g., Magma, Xiaomi-Robotics-R0) demonstrate better knowledge retention and performance on knowledge-sensitive tasks compared to models trained primarily on robotics data (e.g., OpenVLA, $\pi_0$ ).
Internal Representation Bottleneck: Probing results indicate that answer-relevant information is often present in the intermediate layers of the VLM backbone but attenuates significantly in the upper layers and the Action Expert. This suggests a bottleneck where the model retains semantic information internally but fails to translate it into the correct action selection.
Impact of Fine-tuning: Additional downstream fine-tuning (SFT or RL) on pick-and-place datasets did not consistently improve knowledge-sensitive performance and, in some cases (e.g., State, Color), led to further degradation, indicating a bias toward task-specific optimization at the expense of general knowledge.

Significance and Claims

The paper argues that the current paradigm of simply fine-tuning VLMs on robotics data is insufficient for creating truly general-purpose embodied agents. While these models retain the ability to act on shallow perceptual cues, they often lose the ability to distinguish between complex semantic states (e.g., a "dirty" vs. "clean" cup, or a "sad" vs. "neutral" human).

The authors claim that Act2Answer provides a necessary diagnostic tool to measure this "knowledge erosion." The findings suggest that future architectures and training objectives must explicitly prioritize the alignment of the backbone's semantic understanding with learned motor policies. Without such alignment, embodied agents will remain limited in their utility for everyday environments where correct action depends on rich commonsense and world knowledge, not just low-level control. The paper concludes that preventing the erosion of world knowledge remains an open challenge, requiring strategies such as multi-task learning, representation preservation, or architectural decoupling of knowledge and control components.