💻 computer science

RoboMIND 2.0: A Multimodal, Bimanual Mobile Manipulation Dataset for Generalizable Embodied Intelligence

This paper introduces RoboMIND 2.0, a large-scale multimodal dataset of 310K real-world bimanual and mobile manipulation trajectories enhanced with tactile data and digital twins, alongside the MIND-2 hierarchical system that leverages offline reinforcement learning to achieve generalizable embodied intelligence across diverse robot embodiments.

Original authors: Chengkai Hou, Kun Wu, Jiaming Liu, Zhengping Che, Di Wu, Fei Liao, Guangrun Li, Jingyang He, Qiuxuan Feng, Zhao Jin, Chenyang Gu, Zhuoyang Liu, Nuowei Han, Xiangju Mi, Yaoxu Lv, Yankai Fu, Gaole Dai

Published 2026-03-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chengkai Hou, Kun Wu, Jiaming Liu, Zhengping Che, Di Wu, Fei Liao, Guangrun Li, Jingyang He, Qiuxuan Feng, Zhao Jin, Chenyang Gu, Zhuoyang Liu, Nuowei Han, Xiangju Mi, Yaoxu Lv, Yankai Fu, Gaole Dai, Langzhe Gu, Tao Li, Yuheng Zhang, Yixue Zhang, Xinhua Wang, Shichao Fan, Meng Li, Zhen Zhao, Ning Liu, Zhiyuan Xu, Pei Ren, Junjie Ji, Haonan Liu, Kuan Cheng, Shanghang Zhang, Jian Tang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot to be a master chef, a handyman, and a grocery shopper all at once. In the past, you'd have to teach the robot one specific recipe, then another, then another, like training a dog to sit, then stay, then roll over. It was slow, expensive, and the robot would get confused if you asked it to do something slightly different.

RoboMIND 2.0 is like the "Internet" for robot learning. It's a massive library of videos and data showing robots doing thousands of different jobs, so they can learn by watching and figuring things out on their own.

Here is a breakdown of what makes this paper special, using some everyday analogies:

1. The "Gym" for Robots (The Dataset)

Think of the dataset as a giant, high-tech gym where robots go to train.

The Scale: They collected 310,000 hours of robot movements. That's like watching a robot work 24/7 for over 35 years.
The Variety: Instead of just one type of robot, they used six different "bodies" (embodiments). Some are like human arms on a table, some are on wheels (like a Roomba with arms), and some are full humanoids. It's like training a basketball player, a swimmer, and a gymnast all in the same gym so they can learn how to move in any situation.
The "Feel" Factor: Most robot data only has eyes (cameras). This dataset includes touch sensors (tactile data). Imagine trying to pick up a raw egg. If you only have eyes, you might squeeze too hard. If you have "feel," you know exactly how hard to squeeze. This dataset teaches robots how to feel what they are holding.

2. The "Brain" and the "Hands" (The MIND-2 System)

The researchers didn't just collect data; they built a new way for robots to use it called MIND-2. They split the robot's mind into two parts, like a CEO and a skilled worker:

The Slow System (The CEO / The Brain): This part is like a project manager. It looks at a big, scary task (like "Clean the whole kitchen") and breaks it down into small, manageable steps ("Pick up the cup," "Put it in the sink," "Wipe the counter"). It uses a Vision-Language Model (VLM) to understand the goal.
The Fast System (The Worker / The Hands): This part is the muscle. Once the CEO says "Pick up the cup," the Fast System (a VLA model) instantly figures out exactly how to move the fingers, how much force to use, and how to avoid knocking over the salt shaker. It uses the massive dataset to know exactly what to do in split seconds.

Why this matters: Before, robots tried to do the whole job in one giant leap and often failed. Now, they plan first, then act. It's the difference between a human trying to solve a Rubik's cube by guessing randomly versus a human who knows the strategy and solves it step-by-step.

3. The "Digital Twin" (Simulation)

Collecting real robot data is expensive and slow. Robots break, batteries die, and humans get tired.

The Analogy: Imagine a flight simulator for pilots. They can crash a plane a thousand times in the simulator without anyone getting hurt.
The Innovation: RoboMIND 2.0 created a perfect digital copy (Digital Twin) of their real-world robots and rooms. They generated 20,000 more "fake" training videos in the computer.
The Result: They found that mixing real training with this "simulated" training made the robots even better in the real world. It's like a musician practicing on a digital keyboard at home before playing the real piano on stage.

4. The "Generalist" Goal

The ultimate goal of this paper is Generalization.

Old Way: Train a robot to open a specific red door. If you give it a blue door, it fails.
New Way: Because RoboMIND 2.0 has data on 1,139 different objects and 759 different tasks, the robot learns the concept of "opening a door." Now, if you give it a blue door, a heavy door, or a sliding door, it figures it out because it has seen so many variations.

Summary

RoboMIND 2.0 is a massive, high-quality "textbook" for robots that includes:

310,000+ hours of real-world robot videos.
Touch sensors so robots can feel what they grab.
Six different robot bodies so they learn to adapt to any shape.
A two-part brain system (Plan then Act) to handle long, complex tasks.
A perfect simulation to practice safely and cheaply.

By giving robots this massive library of experiences, the researchers are moving us closer to the day when robots can walk into a messy house, figure out what needs to be done, and actually do it without needing a human to hold their hand every step of the way.

1. Problem Statement

Despite advancements in data-driven imitation learning, robotic manipulation research faces critical bottlenecks:

Data Scarcity & Fragmentation: Existing large-scale datasets (e.g., Open X-Embodiment, DROID) often lack diversity in robot morphologies, focusing primarily on single-arm or fixed-base systems. They rarely cover complex bimanual coordination or mobile manipulation simultaneously.
Lack of Multimodal Feedback: Most datasets rely solely on visual observations and proprioception, omitting tactile feedback, which is crucial for contact-rich tasks, slip detection, and fine-grained manipulation.
Generalization Limitations: Current Vision-Language-Action (VLA) models struggle with long-horizon tasks, cross-embodiment transfer, and adapting to novel objects or environments due to insufficient training data diversity.
Simulation-Reality Gap: High-fidelity digital twins and large-scale simulated datasets aligned with real-world tasks are often missing, hindering cost-effective training and sim-to-real transfer.

2. Methodology

A. Dataset Construction: RoboMIND 2.0

The authors constructed a massive, multimodal dataset designed to bridge these gaps through a unified teleoperation and quality assurance pipeline.

Scale & Scope:
- 310,000+ trajectories covering 759 complex tasks and 129 fundamental skills.
- 6 Heterogeneous Embodiments: Includes fixed-base dual-arm (Franka, UR5e), mobile dual-arm (AgileX, ARX), and humanoid platforms (Tien Kung, Tian Yi).
- Diverse Environments: Balanced coverage of domestic (kitchens, living rooms, supermarkets) and industrial (logistics, labs, assembly) settings.
- Object Diversity: Involves 1,139 distinct objects across 6 major categories (Food, Daily Necessities, Kitchenware, etc.).
Multimodal Sensing:
- Tactile Integration: Uniquely includes 12,000 tactile-enhanced sequences using high-resolution Tashan tactile sensors (measuring normal/tangential forces) on grippers.
- Mobile Manipulation: Contains 20,000 mobile manipulation trajectories where robots navigate while manipulating objects.
Annotation & Quality Control:
- Natural Language: Every trajectory is annotated with fine-grained natural language descriptions using LLMs (Gemini 2.5 Pro) for semantic parsing.
- Rigorous QA: A multi-stage inspection workflow filters data based on 12 failure modes (e.g., unintended contact, re-grasping, visual artifacts).
Digital Twin & Simulation:
- Released high-fidelity USD assets and a 20,000-trajectory simulated dataset in Isaac Sim, perfectly aligned with real-world task structures and object configurations to facilitate sim-to-real transfer.

B. Proposed Framework: MIND-2

To leverage the dataset for long-horizon tasks, the authors propose MIND-2, a dual-system architecture:

Slow System (MIND-2-VLM): A high-level planner based on a fine-tuned Vision-Language Model (InternVL3-8B). It decomposes abstract natural language instructions into grounded, executable subgoals and monitors task progress.
Fast System (MIND-2-VLA): A low-level executor based on a Vision-Language-Action model. It generates precise motor actions using egocentric vision, proprioception, and language guidance.
- Training Strategy: The VLA is trained using Implicit Q-Learning (IQL), an offline reinforcement learning algorithm. Crucially, it utilizes both successful and failure trajectories to learn to avoid suboptimal behaviors, enhancing robustness.

3. Key Contributions

RoboMIND 2.0 Dataset: The first open dataset to jointly support bimanual coordination, mobile manipulation, dexterous hands, and high-fidelity tactile sensing across six diverse robot morphologies.
Multidimensional Diversity: Provides simultaneous coverage of morphology, environment, task semantics, and multimodal sensing (vision, proprioception, force-torque, tactile), enabling truly generalizable policy learning.
High-Fidelity Digital Twin: A photorealistic simulation environment with a 20K-trajectory dataset, validated to effectively augment real-world training and reduce costs.
MIND-2 Framework: A novel slow-fast dual-system architecture that successfully solves long-horizon, collaborative mobile manipulation tasks where standard VLA models fail.
Comprehensive Benchmarking: Extensive evaluation across single-task imitation learning, multi-task VLA models, object generalization, and sim-to-real transfer.

4. Experimental Results

A. Imitation Learning & VLA Baselines

3D vs. 2D: 3D-aware imitation learning methods (e.g., DP3, Dense Policy) significantly outperformed 2D methods (ACT, UVA) on bimanual tasks, highlighting the importance of spatial modeling.
VLA Performance: Among VLA models, XR-1 demonstrated superior cross-embodiment generalization compared to $\pi_0$ and $\pi_{0.5}$ , successfully transferring skills across fixed, mobile, and humanoid platforms.
Tactile Impact: Integrating tactile data into VLA inputs consistently improved success rates, particularly in contact-rich tasks (e.g., XR-1 saw a significant boost with tactile input).

B. MIND-2 Performance

Long-Horizon Tasks: In challenging multi-robot collaboration scenarios (e.g., supermarket checkout, industrial sorting), MIND-2 significantly outperformed single-task baselines and existing VLA models.
- Example: On the "Supermarket Task," MIND-2 (Offline RL) achieved a 90% success rate, compared to 40% for XR-1 and 10-30% for other baselines.
Offline RL Benefits: Training with failure data via IQL further refined the policy, with the best performance observed when the ratio of successful to failed trajectories was balanced (1:1).

C. Generalization & Sim-to-Real

Object Generalization: Models trained on RoboMIND 2.0 showed strong robustness to novel objects (different shapes, colors, materials) in UR5e tasks.
Sim-to-Real: Training on a hybrid dataset (Real + Sim) consistently outperformed training on real data alone. Increasing simulation data (up to a 1:5 ratio) further improved real-world performance, validating the high fidelity of the digital twin.

5. Significance

Advancing Embodied AI: RoboMIND 2.0 addresses the critical "data bottleneck" for generalist robots, providing the necessary scale and diversity to train models that can operate in unstructured, real-world environments.
Tactile Integration: By making tactile data a core component, the dataset enables research into contact-rich manipulation, a key capability for dexterous human-like robotics.
Scalable Training: The release of high-fidelity digital twins and simulated data offers a cost-effective pathway for scaling robot learning, proving that synthetic data can effectively bridge the sim-to-real gap.
New Paradigm: The MIND-2 framework demonstrates that separating high-level semantic planning from low-level control, combined with offline RL, is a viable strategy for solving complex, long-horizon mobile manipulation tasks.

In summary, RoboMIND 2.0 represents a significant leap forward in robotic data infrastructure, moving from single-modality, single-robot datasets to a comprehensive, multimodal, and cross-embodiment resource that enables the training of robust, generalizable, and dexterous embodied intelligence.