💻 computer science

RoboManipBaselines: A Unified Framework for Imitation Learning in Robotic Manipulation across Real and Simulation Environments

The paper introduces RoboManipBaselines, an open-source, unified framework designed to streamline the entire imitation learning pipeline for robotic manipulation across diverse simulation and real-world environments by offering a consistent, extensible, and reproducible interface for data collection, policy training, and evaluation.

Original authors: Masaki Murooka, Tomohiro Motoda, Ryoichi Nakajo, Hanbit Oh, Koshi Makihara, Keisuke Shirai, Tetsuya Ogata, Yukiyasu Domae

Published 2026-03-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Masaki Murooka, Tomohiro Motoda, Ryoichi Nakajo, Hanbit Oh, Koshi Makihara, Keisuke Shirai, Tetsuya Ogata, Yukiyasu Domae

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot how to fold laundry, cook a meal, or fix a broken toy. In the past, you had to be a genius engineer to write thousands of lines of code telling the robot exactly how to move its arm, how hard to grip, and what to do if it slips. It was like trying to teach a dog to play chess by writing a manual for every single paw movement.

RoboManipBaselines is a new, open-source "toolbox" that changes the game. Instead of writing code from scratch, researchers can now use this toolbox to teach robots by showing them what to do, just like teaching a child by demonstration.

Here is a simple breakdown of how it works, using some everyday analogies:

1. The "Universal Adapter" (Integration)

Think of the world of robotics as a chaotic room full of different types of video game consoles (PlayStation, Xbox, Nintendo) and different controllers. Usually, if you want to play a game on one, you can't use the controller from another.

RoboManipBaselines is like a universal adapter. It lets you plug in any robot (whether it's a real metal arm in a factory or a virtual robot in a computer game) and any "game" (task) into one single system. You don't need to rewrite the rules every time you switch robots. You can train a robot in a video game simulation, and then plug that same "brain" into a real robot, and it just works.

2. The "All-in-One Cookbook" (The Pipeline)

Teaching a robot usually involves three messy steps:

Gathering ingredients: Collecting video of humans doing the task.
Cooking: Training the AI model to learn from those videos.
Serving: Letting the robot try the task on its own.

Usually, these steps happen in three different, incompatible kitchens. RoboManipBaselines is a single, giant kitchen where you can do all three steps seamlessly. You collect the data, cook the model, and serve the result without ever leaving the room.

3. The "Lego System" (Extensibility)

Imagine you have a Lego set. If you want to add a new piece, like a dragon or a castle tower, you just snap it on. You don't have to melt down the whole castle to build it.

This framework is built like Lego. If a researcher wants to add a new robot arm, a new sensor (like a "touch" sensor), or a new way for the robot to think, they can just "snap" that new piece onto the existing framework. They don't have to rebuild the whole thing from scratch. This makes it incredibly easy for scientists to test crazy new ideas quickly.

4. The "Replay Button" (Reproducibility)

In science, it's frustrating when one lab says, "I built a robot that can fold a shirt!" and another lab tries to copy it and fails because they used different tools.

RoboManipBaselines provides a standardized "Replay Button." It comes with pre-recorded "demonstrations" (datasets) and standard tests (benchmarks). This means if Lab A and Lab B both use this toolbox, they are playing by the exact same rules. They can compare their results fairly, like two runners on the same track with the same starting line.

What Can It Actually Do? (The Cool Stuff)

The paper shows off some amazing things people built using this toolbox:

The "Magic Touch": They taught robots to use special "touchy" sensors (like a human fingertip) to feel if they are holding a spoon correctly, not just looking at it.
The "Chatty Robot": They connected the robot to a voice assistant. You can say, "Robot, put the banana on the red plate," and the robot understands the language and does the task.
The "Time Traveler": They used the toolbox to simulate thousands of hours of practice in a computer world (simulation) to teach the robot, then let the robot try it in the real world. It's like a pilot training in a flight simulator before flying a real plane.
The "DIY Sensor": One researcher even used a cheap, tiny keyboard attached to a robot's fingers to act as a touch sensor. It proved you don't need expensive gear to get good results; you just need the right framework to connect the dots.

The Bottom Line

RoboManipBaselines is the "operating system" for the future of robot learning. Before this, teaching a robot was like trying to build a car engine with a hammer and a screwdriver. Now, it's like using a 3D printer where you just select the part you need, and the machine assembles it for you.

It removes the headache of technical details so that scientists and engineers can focus on the fun part: teaching robots to do cool, helpful things.

1. Problem Statement

Imitation learning (IL) has emerged as a powerful approach for robotic manipulation, allowing robots to acquire skills from expert demonstrations without explicit environment modeling. However, the ecosystem for IL research faces significant fragmentation:

Lack of Unification: Unlike model-based robotics, which has mature, standardized frameworks, IL research relies on disparate software assets that are difficult to reuse across different pipelines.
Simulation-to-Reality Gap: Researchers often struggle to transition from simulation to real-world deployment because existing frameworks rarely support both environments with a consistent workflow.
Reproducibility Issues: The lack of standardized datasets and benchmarking protocols makes it difficult to compare new methods fairly.
Extensibility Barriers: Adding new robots, sensors, or policy models often requires rewriting large portions of code, hindering rapid experimentation.

2. Methodology: RoboManipBaselines Framework

The authors present RoboManipBaselines, an open-source Python framework designed to support the entire imitation learning lifecycle. The framework is built upon four core design principles: Integration, Generality, Extensibility, and Reproducibility.

A. Core Architecture

The framework systematically implements the three pillars of imitation learning:

Environment:
- Unified Interface: Adopts an OpenAI Gym-compatible interface, allowing seamless switching between simulators and real robots.
- Simulators: Supports MuJoCo, Isaac Gym, and PyBullet.
- Real Robots: Currently supports UR5e and xArm 7, with plans for Franka and Kinova.
- Advanced Features: Includes support for deformable objects (cables, cloth), parallel simulation for data collection, and tactile sensor plugins (e.g., TACTO).
- Modularity: Developers can extend the framework by inheriting abstract classes, allowing custom robot implementations without modifying the core code.
Dataset:
- Data Format: Uses a custom, efficient storage format combining HDF5 (for low-dimensional data like joint angles) and MP4 (for images).
- Depth Image Compression: Introduces a novel encoding scheme where 16-bit depth values are split into two 8-bit channels and stored as RGB video frames. This allows standard video codecs to compress depth data efficiently (e.g., 10 seconds of 640x480 depth data at 22Hz stored in ~14MB) while maintaining millimeter-level reconstruction accuracy.
- State Representation: Automatically stores both measured and commanded values (joint angles, end-effector poses) and their relative differences, enabling flexible state/action space definitions during training.
- Public Data: Provides over 1,500 episodes across 12 tasks, including language-conditioned Pick&Place scenarios.
Policy:
- Unified Training/Rollout: Provides a single interface for training and deploying diverse policy architectures.
- Supported Models: Includes baselines like MLP, ACT (Action Chunking with Transformers), Diffusion Policy, SARNN, 3D Diffusion Policy, Flow Matching policies (ManiFlow, Flow Policy), and language-conditioned models (MT-ACT, $\pi_0$ , GR00T).
- Efficiency: Supports loading entire datasets into CPU memory to accelerate training by several times.

B. Workflow

The framework enables a streamlined workflow via command-line tools:

Data Collection: Teleoperation via keyboard, 3D mouse, or leader-follower systems (e.g., GELLO).
Training: One-line commands to train specific policies on specific datasets.
Rollout: Deployment of trained policies in the same environment used for collection (sim or real).

3. Key Contributions

Unified End-to-End Framework: The first framework to provide a consistent workflow from data collection to policy rollout across multiple simulators and real-world robots.
Extensive Benchmarking:
- Simulation: Evaluated 8 manipulation tasks (including deformable objects like cables and cloth) using 4 different policies.
- Real-World: Evaluated 8 complex tasks (folding, grasping transparent/reflective objects) on a UR5e.
- Ablation Studies: Analyzed the impact of camera viewpoints (Front, Side, Hand) and dataset size on performance.
Novel Data Handling: A highly efficient depth image compression method that balances storage size and reconstruction precision.
Research Applications: Demonstrated the framework's versatility through five distinct applications:
- Data Augmentation: Automated trajectory generation improved success rates from 36% to 82%.
- Tactile Integration: Successfully integrated the Sparsh tactile foundation model with Diffusion Policy, outperforming vision-only baselines.
- Interactive Systems: Built a speech-enabled robot using Whisper/GPT-4o and Multi-Task ACT, allowing natural language task execution.
- Sensor Evaluation: Compared stereo (RealSense D455) vs. Time-of-Flight (Femto Bolt) cameras, showing ToF superiority for 3D Diffusion Policies under visual disturbances.
- Hardware Extension: Prototyped a low-cost contact sensor using a compact keyboard mounted on gripper fingers, enabling contact-aware imitation learning.

4. Experimental Results

Simulation Benchmarks:
- Policy Performance: Diffusion Policy achieved the highest average success rate (52.0%), followed closely by SARNN (50.8%) and ACT (45.0%). Simple MLPs performed poorly (16.4%).
- Camera Viewpoints: The combination of Front + Side cameras yielded the best performance (52.1%). Relying solely on the wrist-mounted "Hand" camera resulted in the lowest performance (17.1%) due to large viewpoint changes.
- Data Efficiency: Success rates did not always increase monotonically with more training episodes, suggesting that for single-task settings, high-quality, smaller datasets are often sufficient.
Real-World Benchmarks:
- Diffusion Policy again achieved the highest average success rate (47.9%), followed by ACT (43.8%) and SARNN (39.6%).
- The framework successfully bridged the sim-to-real gap, allowing policies trained in simulation to be evaluated on real hardware with consistent interfaces.

5. Significance

RoboManipBaselines addresses a critical bottleneck in robotic learning research by providing a standardized, extensible, and reproducible foundation.

Accelerated Research: By abstracting away environment and data management complexities, researchers can focus on developing novel policy architectures and learning algorithms.
Democratization: The inclusion of public datasets and support for diverse hardware lowers the barrier to entry for researchers without access to extensive proprietary infrastructure.
Industrial Relevance: The framework's ability to handle real-world complexities (deformable objects, tactile sensing, language instructions) and its adoption by industry partners indicate its potential for transitioning from academic research to industrial applications.
Future-Proofing: The modular design ensures that new simulators, sensors, and policy models can be integrated with minimal effort, ensuring the framework remains relevant as the field evolves.

In conclusion, RoboManipBaselines is not just a collection of tools but a cohesive ecosystem that facilitates the systematic advancement of imitation learning in robotic manipulation, bridging the gap between simulation theory and real-world application.