Technical Summary: MuseVLA

Problem Statement

Current Vision-Language-Action (VLA) models for robotic manipulation predominantly rely on RGB observations. While effective for visual tasks, this limitation restricts the robot's ability to perceive physical properties that are difficult or impossible to infer from standard cameras, such as temperature, sound, or radar responses. Existing attempts to incorporate non-visual modalities (e.g., depth, tactile, audio) suffer from three primary limitations:

1. Lack of Adaptability: They typically require specialized model architectures or encoders for each specific sensor, making it difficult to scale to diverse or evolving sensor sets.
2. Data Scarcity: They depend on large-scale, expensive multisensory datasets, which are time-consuming to collect.
3. Static Sensor Usage: Existing models often assume a fixed sensor configuration, lacking the ability to adaptively select the most relevant sensory input based on the specific task, leading to inefficiency and suboptimal performance.

Humans, conversely, treat sensory modalities as task-oriented tools, adaptively selecting and invoking the right sensor (e.g., thermal for heat, audio for sound) only when necessary. MuseVLA aims to replicate this adaptive capability within a unified VLA framework.

Methodology

MuseVLA is an adaptive multimodal sensing VLA model designed to integrate diverse sensors as "on-demand tools." The architecture and pipeline are structured as follows:

1. Adaptive Sensor Selection Mechanism

Instead of processing all sensor inputs simultaneously, MuseVLA decomposes the task into three stages:

• Sensor and Target Generation: Given a language instruction ($l$) and an RGB observation ($o_t$), the VLA model ($\pi$) first generates a sensor token (e.g., <thermal>, <acoustic>, <mmWave>, or <none>) and a target description ($l_d$). This is analogous to a tool call with arguments, selecting the specific modality and the object of interest.
• Grounded Sensor Image Construction: The selected sensor measurement ($s_{i,t}$) is processed by a grounding function ($G$). Using the target description, a segmentation module (SAM3) creates a binary mask of the target object on the RGB image. The sensor data is then overlaid as a heatmap onto this masked region, creating a grounded sensor image ($m_{i,t}$).
• Action Generation: The grounded sensor image is appended to the model input. The VLA model then generates the action chunk ($A$) based on the instruction, RGB image, and the unified grounded sensor image.

2. Unified Representation

To avoid modality-specific encoders, MuseVLA uses grounded sensor images as a unified intermediate representation. By projecting heterogeneous sensor readings (thermal, radar, audio) onto the RGB camera plane over the target object, the model can utilize a single pretrained vision encoder (SigLIP) for all modalities. This decouples sensor-specific processing from the VLA backbone, allowing for the integration of new sensors without full retraining.

3. Data Synthesis Pipeline

To address the scarcity of real-world multisensory datasets, the authors propose a data synthesis pipeline:

• Keyword Injection: Physical property keywords (e.g., "hot," "ringing," "occupied") are injected into task instructions from existing RGB-only datasets.
• Synthetic Grounding: A VLM generates a target description, SAM3 segments the object, and a color-coded mask corresponding to the physical property is overlaid on the segmented region.
• Dataset Expansion: This process generates multisensory training data (9.6K episodes) from existing RGB datasets (MolmoAct, AgiBotWorld-Alpha, VITRA), enabling data-efficient training and zero-shot generalization.

4. Model Architecture and Training

• Backbone: The model uses PaliGemma-2 (Gemma-2 + SigLIP) as the VLM backbone and a Diffusion Transformer (DiT) as the action expert.
• Training Strategy: The model undergoes co-training. The VLM is trained to minimize cross-entropy loss for sensor token and target description generation. The VLA is trained end-to-end using a combined loss function: the VLM loss plus the diffusion MSE loss for action prediction. This ensures the model maintains accurate sensor selection while generating effective manipulation actions.

Key Contributions

1. MuseVLA Model: An adaptive multimodal sensing VLA that treats sensors as on-demand tools, enabling scalable and efficient sensor integration for dexterous hand manipulation.
2. Novel Representations and Mechanisms: The introduction of learnable sensor tokens for adaptive selection and grounded sensor images as a unified representation that eliminates the need for modality-specific encoders.
3. Data Synthesis Pipeline: A method to generate multisensory datasets from existing RGB data, facilitating training on unseen tasks without expensive hardware data collection.
4. Empirical Evaluation: Comprehensive evaluation on a real-world robot with a 12-DoF dexterous hand across thermal, acoustic, and radar-guided tasks.

Experimental Results

The model was evaluated on a real-world robot setup involving thermal cameras, mmWave radar, and microphone arrays.

• Task Success Rates: MuseVLA achieved an average success rate of 80.6% across thermal-guided pick-and-place, audio-driven object search, and radar-assisted hidden object retrieval.
  • This significantly outperformed RGB-only baselines (e.g., $\pi_0$-RGB at 20.8%) and baselines using raw sensor heatmaps without adaptive selection (e.g., $\pi_0$-Raw at 27.8%).
  • Specifically, MuseVLA outperformed the best RGB-only baseline by 58% and the best raw-sensor baseline by 47% in average success rate.
• Adaptive Selection Accuracy: The model demonstrated high accuracy in selecting the correct sensor token and target description, achieving 100% sensor selection accuracy on training tasks and 82.0% target description accuracy on unseen tasks when pretrained.
• Efficiency: By selecting only the relevant sensor, MuseVLA reduced inference GPU memory usage from 13.23 GB (processing all sensors) to 6.61 GB.
• Zero-Shot Generalization: When trained on the synthesized dataset, MuseVLA achieved a 66.7% success rate on unseen tasks (tasks with new object combinations or instruction structures), demonstrating strong generalization capabilities.
• Multi-Stage Tasks: The model successfully handled cascaded tasks (e.g., Radar $\to$ Thermal), achieving 75.0% success in switching modalities within a single end-to-end execution.

Significance and Claims

The paper claims that MuseVLA represents a scalable path toward general-purpose multisensory manipulation. By decoupling sensor processing from the VLA backbone and treating sensors as conditional tools, the approach allows robots to efficiently acquire task-oriented physical observations without the computational overhead of processing all available sensors. The authors assert that their data synthesis pipeline effectively mitigates the bottleneck of data scarcity, enabling robust zero-shot generalization to tasks requiring multimodal sensing.

The authors acknowledge current limitations, specifically the modest scale of their real-world dataset and dependence on a performant segmentation module (SAM3), noting that these constraints may be relaxed as larger datasets and stronger segmentation models become available. The work is presented as a step toward enabling robots to interact with the physical world with the same adaptive, task-conditioned sensory flexibility as humans.