💻 computer science

Mind-VLA: Instruction-Aware Spatial Representation Alignment for Vision-Language-Action Models

Mind-VLA is an instruction-aware spatial representation alignment method that enhances Vision-Language-Action models by specifically aligning latent representations with the 3D geometry of target objects identified by language instructions, thereby significantly improving performance on fine-grained manipulation and occluded target tasks.

Original authors: Xingyu Ding, Yuzhong Zhao, Yang Wu, Chunhai Zhao, Chaoyang Zhao, Yifan Zhang, Jian Cheng

Published 2026-08-31

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Xingyu Ding, Yuzhong Zhao, Yang Wu, Chunhai Zhao, Chaoyang Zhao, Yifan Zhang, Jian Cheng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Robots have long been masters of repetition, capable of performing the same precise motion thousands of times without error. Yet, when asked to navigate a cluttered room or pick up a specific item from a pile of similar objects, they often stumble. The challenge lies not just in seeing the world, but in understanding the three-dimensional shape and position of the specific object a human has asked them to touch. Modern robots are increasingly guided by Vision-Language-Action models, systems that take a visual scene and a spoken command, then decide on a physical movement. While these systems have become remarkably good at general tasks, they often struggle when the target object is partially hidden or when the robot must distinguish between two nearly identical items. The core difficulty is that these models tend to treat the entire visual scene as a single, uniform block of information, rather than focusing their geometric understanding on the specific item the human mentioned.

A new approach called Mind-VLA addresses this limitation by teaching robots to pay attention to the specific object named in a command. Instead of trying to map the 3D geometry of an entire room, the system learns to isolate the target object described in the language instruction and build a detailed mental model of just that item. This method allows the robot to understand the shape and position of a potato even if it is sitting next to a similar-looking apple, or to grasp a banana that is partially covered by a cloth. By shifting the focus from the whole scene to the specific target, the robot gains a sharper sense of where to reach and how to hold the object, leading to more reliable performance in complex, real-world situations.

The researchers behind this work identified a fundamental flaw in how current 3D-aware robots are trained. Existing methods often align the robot's internal understanding with the geometry of the entire scene, regardless of what the human is asking it to do. If a person says, "pick up the potato," the robot's training data might force it to encode the 3D structure of the table, the background wall, and the other fruits on the counter alongside the potato. This creates a muddy signal where the most important information—the shape of the potato itself—gets lost in the noise of the surrounding environment. This issue becomes critical when the target object is partially occluded, or hidden from view, because the robot has not been explicitly trained to preserve the 3D structure of the specific item it needs to manipulate.

To solve this, the team developed a training process that acts like a spotlight, illuminating only the object of interest. When the robot receives a command, the system first parses the language to identify the target object and its intended order of interaction. It then constructs a set of standard, or canonical, views of that specific object, effectively creating a clean, 3D blueprint of the item in isolation. During the training phase, the robot is shown these isolated views and asked to align its internal understanding with the geometry of the target, ignoring the rest of the scene. This process involves two main steps: predicting the hidden 3D structure of the target from its visual appearance and aligning the robot's intermediate processing layers with the detailed geometric features of that specific object. Crucially, this extra training supervision is only used while the robot is learning; once the training is complete, the robot operates just as it did before, without needing any additional 3D sensors or complex processing during its actual work.

The results of this approach were tested in both simulated environments and on real physical robots. In a series of standard benchmarks designed to test robotic manipulation, the new method achieved a success rate of 94.4 percent, outperforming previous models that used the same underlying architecture. The improvement was particularly noticeable in tasks requiring fine-grained discrimination, where the robot had to choose between similar objects. In the CALVIN benchmark, which tests a robot's ability to complete a sequence of five different tasks in a row, the system completed an average of 4.47 tasks, a slight but meaningful improvement over the previous best. These numbers suggest that by focusing on the specific target rather than the whole scene, the robot becomes more precise and less prone to confusion.

The true test of this method, however, came when the researchers placed the robot in a real-world scenario where the target object was partially hidden. They set up a dual-arm robot and asked it to pick up and place items like potatoes and bananas, sometimes covering about 25 percent of the target with an occluder. In these challenging conditions, the new system succeeded 54 percent of the time. This was a significant leap compared to a control version of the same robot that used the traditional, scene-wide approach, which succeeded only 28 percent of the time. The difference of 26 percentage points highlights the value of instruction-aware spatial understanding; when the target is partially blocked, the robot that knows exactly what to look for can still infer its shape and position, whereas the robot looking at the whole scene often fails to reconstruct the missing parts of the target.

Beyond the numbers, the study revealed that the robot was genuinely learning to represent the geometry of the specific object. When the researchers analyzed the internal layers of the model, they found that the system encoded much more detailed information about the target's shape than previous models did. Furthermore, the robot could reliably retrieve the correct object based on the language instruction, proving that its internal map was tied directly to the words it heard. This suggests that the robot is not just memorizing patterns but is developing a flexible understanding of how language relates to 3D space. By teaching the machine to focus its geometric attention on the object that matters, the researchers have taken a step toward making robots that can handle the messy, cluttered, and often hidden realities of human environments with greater confidence and skill.

Technical Summary: Mind-VLA

Problem Statement

Recent Vision-Language-Action (VLA) models have improved generalization by aligning their internal representations with 3D scene geometry. However, existing 3D-aware VLA methods are fundamentally instruction-agnostic. They typically align representations with the entire scene's geometry, regardless of which specific object is designated by the language instruction. This approach encodes target objects, distractors, and background structures together, weakening the connection between the instruction and the relevant 3D representation.

This limitation leads to failures in two specific scenarios:

Fine-grained manipulation: When the target object is visually similar to nearby distractors, instruction-agnostic supervision makes discrimination difficult.
Target occlusion: When the target is partially occluded, scene-level 3D information often captures only partial geometry of the target. Since the model is not explicitly trained to preserve the 3D structure of the instructed object, it becomes unreliable under occlusion.

Methodology: Mind-VLA

To address these issues, the authors propose Mind-VLA, an instruction-aware spatial representation alignment method. The core idea is to align the VLA model's latent representations specifically with the 3D features of the instruction-specified target object, rather than the whole scene.

The framework consists of four key components:

1. Instruction-Aware Target Acquisition

The system first parses the language instruction to identify the specific target objects and their manipulation order.

Parsing: A vision-language model (Qwen3-VL) parses the instruction into an ordered sequence of target phrases (e.g., "potato," then "banana").
Selection: Using a candidate-ranking strategy combining CLIP semantic similarity, SAM 3 segmentation confidence, and temporal motion cues from demonstrations, the system identifies the specific manipulated instances.
Reconstruction: The selected target is reconstructed into a 3D mesh using SAM 3D Objects. This process relies solely on RGB observations, language, and interaction cues, without requiring simulator object IDs or ground-truth meshes.

2. Canonical Target View Construction

To handle variable-length and unordered 3D meshes, the reconstructed object is centered, scale-normalized, and rendered into a fixed set of three orthographic views along its principal axes. These canonical target views provide a consistent, object-centric representation that reduces variations caused by camera viewpoint, scene pose, and visibility.

3. Instruction-Aware Spatial Representation Alignment

Mind-VLA introduces two auxiliary supervision modules during training to align the VLA model with the target object's geometry:

Canonical Target-View Latent Prediction ( $L_{tri}$ ): A group of object-level queries is introduced to predict the latent representation of the three canonical views (encoded by a frozen Stable Diffusion VAE). This encourages the model to encode the target object's geometry.
Target-Object VGGT Feature Alignment ( $L_{geo}$ ): The canonical views are fed into a frozen VGGT (Vision-Geometry-Transformer) model to extract geometric features. The intermediate representations of the VLA model are then aligned with these target-object features using a learnable projection. This directly constrains the VLA to encode the 3D geometric structure and cross-view information of the specific instructed object.

4. Temporal Target Assignment

For multi-step tasks, the system assigns each timestep to its corresponding active target. Transitions are localized based on gripper opening/closing cycles or significant motion changes in object trajectories, ensuring the supervision switches to the next target at the correct time.

Inference: Crucially, the auxiliary modules (instruction parsing, 3D reconstruction, VAE encoding, and VGGT extraction) are removed at inference time. The model retains only the learned object-level queries, meaning Mind-VLA adds negligible deployment overhead and requires no additional 3D inputs or preprocessing during execution.

Key Contributions

Identification of a Limitation: The authors identify that existing 3D-aware VLA methods are instruction-agnostic, which hinders performance in fine-grained manipulation and occluded target scenarios.
Mind-VLA Framework: A novel method that aligns VLA representations with target-object 3D features using canonical target-view latent prediction and VGGT feature alignment.
Performance: The method achieves state-of-the-art results on standard benchmarks with a compact backbone and demonstrates significant robustness to occlusion in real-world settings.

Experimental Results

Simulation Benchmarks

LIBERO: Mind-VLA achieves 94.4% average success on the LIBERO benchmark using a compact 345M-parameter backbone (GPT-2-M), outperforming the base DreamVLA (92.6%) and competing with much larger models (3–7B parameters). Notably, it improves success on the LIBERO-Object suite from 94.0% to 98.0%.
CALVIN: On the CALVIN ABC-D benchmark, Mind-VLA reaches an average completed length of 4.47 and 79.4% success on Task 5, surpassing the base DreamVLA (4.44 and 78.1%).

Real-Robot Evaluation

The authors evaluated the method on a dual xArm 6 setup with a static and wrist-mounted camera, focusing on tasks with ~25% target occlusion.

Occlusion Robustness: Mind-VLA achieved a 54% average success rate under occlusion.
Comparison: This outperforms the matched "Scene-VGGT" control (which uses scene-level alignment) by 26 percentage points (54% vs. 28%).
Degradation: While the Scene-VGGT control's performance dropped by 29 points when moving from normal to occluded settings, Mind-VLA's performance dropped by only 13 points.

Representation Analysis

Linear probing experiments confirmed that Mind-VLA learns significantly more target-geometry information (centered cosine similarity of 0.75 vs. 0.3–0.4 for baselines) and that this geometry is strongly associated with the specific instruction (29% retrieval accuracy vs. 10% chance).

Significance and Claims

The paper claims that instruction-aware spatial supervision is critical for effective VLA learning. By shifting the focus from global scene geometry to the specific geometry of the instructed target, Mind-VLA enables models to:

Better discriminate between similar objects.
Maintain robustness when the target is partially occluded.
Achieve these improvements without incurring additional computational costs or sensor requirements at inference time.

The authors position Mind-VLA as a complementary approach to existing methods, preserving the inference efficiency of standard VLAs while significantly enhancing their spatial reasoning capabilities through targeted training supervision.