💻 computer science

IGen: Scalable Data Generation for Robot Learning from Open-World Images

IGen is a scalable framework that leverages open-world images and vision-language models to synthesize realistic 3D scenes and executable robot actions, enabling the training of generalist robotic policies with performance comparable to those trained on real-world data.

Original authors: Chenghao Gu, Haolan Kang, Junchao Lin, Jinghe Wang, Duo Wu, Shuzhao Xie, Fanding Huang, Junchen Ge, Ziyang Gong, Letian Li, Hongying Zheng, Changwei Lv, Zhi Wang

Published 2026-04-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chenghao Gu, Haolan Kang, Junchao Lin, Jinghe Wang, Duo Wu, Shuzhao Xie, Fanding Huang, Junchen Ge, Ziyang Gong, Letian Li, Hongying Zheng, Changwei Lv, Zhi Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot how to do chores, like watering plants or pouring tea. Traditionally, to teach a robot, you have to be its "teacher" in real life. You have to physically guide its arm, record every movement, and do this thousands of times for every single new task. It's like trying to teach a child to ride a bike by holding the seat and running alongside them for every single turn. It's exhausting, slow, and you can only teach them in your specific living room.

IGen is a new invention that changes the game. Instead of needing a real robot in a real room, IGen lets us teach robots using any photo you find on the internet.

Here is how it works, broken down with some simple analogies:

1. The Problem: The "Empty Photo" Dilemma

Think of a photo on your phone as a frozen moment in time. It shows a teapot and a cup, but it doesn't tell you how to pick up the teapot, where to pour the tea, or how hard to squeeze the handle. It's a beautiful picture, but it's "silent" regarding movement. Robots need a script, not just a picture.

2. The Solution: IGen (The "Imagination Engine")

IGen is like a super-smart movie director who looks at a single still photo and instantly writes, directs, and films a whole movie of a robot doing the task. It does this in three magical steps:

Step A: Turning a Flat Photo into a 3D World (The "Pop-Up Book" Effect)

First, IGen looks at a flat 2D image (like a picture of a kitchen) and uses AI to "pop" it open into a 3D world.

The Analogy: Imagine taking a drawing of a house and magically turning it into a 3D model you can walk around in. IGen does this by estimating depth (how far away things are) and creating a "point cloud" (a digital cloud of dots representing every object).
The Result: Suddenly, the robot has a 3D map of the room, not just a flat photo.

Step B: The Brain and the Hands (The "Chef and the Sous-Chef")

Now that the robot has a 3D map, it needs to know what to do.

The Brain (VLM): IGen uses a "Vision-Language Model" (a super-smart AI that understands both pictures and words). You tell it, "Please water the flowers." The AI acts like a head chef, breaking that big command into tiny steps: "1. Find the watering can. 2. Grab the handle. 3. Lift it up. 4. Tilt it over the plant."
The Hands (Motion Planner): Once the chef gives the orders, IGen calculates the exact math for the robot's arm to move. It figures out the precise angles and speeds needed to grab the can without knocking it over. It's like a sous-chef who knows exactly how to chop the vegetables based on the head chef's instructions.

Step C: Filming the Practice Run (The "Virtual Rehearsal")

Here is the coolest part. Before the robot ever touches a real object, IGen simulates the whole action inside the computer.

The Analogy: Imagine a video game where you can practice a level over and over again without breaking anything. IGen takes the 3D map and the movement plan, and it "renders" a video of the robot doing the task.
The Magic: It doesn't just guess; it uses physics. If the robot lifts the watering can, the water inside moves realistically. If the robot drops a cup, it shatters. It creates a perfect, safe "practice session" that the robot can learn from.

3. Why This is a Big Deal

Infinite Practice: You can take one photo of a messy desk, and IGen can generate 1,000 different practice scenarios (moving the cup to the left, the right, lifting it high, lifting it low). It's like having a robot that can practice a million times in the time it takes you to drink a coffee.
No Real Robots Needed: You don't need expensive robot arms or a lab full of sensors to collect data. You just need a computer and a photo.
Real-World Success: The paper proves that robots trained only on these computer-generated videos can actually go into the real world and do the job. It's like a pilot who has never flown a real plane but has practiced 10,000 hours in a perfect flight simulator, only to land the plane perfectly on their first real flight.

The Bottom Line

IGen is a bridge between the visual world (the billions of photos on the internet) and the physical world (robots that can move and touch things). It turns "passive" pictures into "active" training data, allowing robots to learn from the entire internet rather than just a few hours of human guidance in a single room.

It's the difference between trying to learn to swim by reading a book about water, versus jumping into a virtual reality pool where you can practice diving, floating, and swimming a million times before ever stepping into the real ocean.

1. Problem Statement

The development of generalist robotic policies relies heavily on large-scale, paired visual-action datasets. However, collecting such data via on-robot teleoperation is labor-intensive, expensive, and typically limited to specific, controlled environments. While the internet offers a vast repository of "open-world" images that capture diverse real-world scenes, these images lack the associated robot actions (executables) required for training visuomotor policies. Existing methods to bridge this gap face significant limitations:

Real-to-Sim-to-Real: Requires explicit 3D reconstruction of specific workspaces, limiting scalability to arbitrary open-world images.
Video Generation Models: Can predict future visual frames but often fail to generate physically grounded, explicit robot action sequences (e.g., SE(3) poses) and struggle with long-horizon or complex instruction-driven tasks.

IGen aims to solve this by creating a framework that automatically converts unstructured, single open-world images into high-quality, grounded visuomotor data (visual observations + executable actions) without human annotation.

2. Methodology

IGen operates through a unified three-stage pipeline that transforms a single 2D image into a dynamic, manipulable 3D simulation environment.

Stage 1: Scene Reconstruction (Pixels to Structured 3D)

Instead of operating on raw 2D pixels, IGen reconstructs the scene into a structured 3D representation suitable for robotic planning.

Depth & Geometry: Uses a monocular geometric foundation model (Metric3Dv2) to estimate depth and lift pixels into a dense 3D point cloud.
Object Segmentation & Reconstruction:
- Uses a Vision-Language Model (VLM) to identify task-relevant objects.
- Employs the Segment Anything Model (SAM) to generate masks.
- Utilizes a 3D generative reconstruction model (TRELLIS) to infer the full 3D shape and appearance of masked objects from the single view, addressing occlusion and incomplete geometry.
- Re-positions reconstructed objects using 6D pose estimation to align with the original scene.
Spatial Keypoints: Extracts DINOv2 features and applies K-means clustering on both feature embeddings and 3D coordinates to generate a set of spatial keypoints ( $K$ ) that serve as anchors for task planning.
Background Inpainting: Removes the target object from the original image to reconstruct a clean background point cloud ( $P_{bg}$ ).

Stage 2: Action Planning (Reasoning to Execution)

IGen leverages the reasoning capabilities of VLMs to generate robot behaviors without human demonstration.

High-Level Planning: The VLM takes the annotated image (with keypoints), their 3D coordinates, and a natural language task instruction as input. It decomposes the task into sub-stages (e.g., "Grasp," "Lift," "Move").
Low-Level Control: The VLM translates these sub-stages into executable Python control functions based on the end-effector's SE(3) pose. It infers parameters (e.g., lift height, translation distance) directly from the 3D keypoint coordinates.
Trajectory Generation: A motion planner (Curobo in Isaac Sim) generates feasible trajectories between sub-goals, producing a sequence of end-effector poses ( $T_t$ ) and joint states.

Stage 3: Experience Synthesis (Observation Generation)

To create training data, IGen synthesizes visual observations synchronized with the generated actions.

Dynamic Point Cloud Synthesis:
- Static Background: The background point cloud remains static.
- Robot Motion: A virtual camera captures the robot's motion point cloud sequence.
- Object Manipulation: During the "grasp" phase, the manipulated object's point cloud undergoes rigid-body transformations, strictly following the end-effector's pose trajectory ( $T_t$ ).
Rendering: The composite point cloud sequence (Background + Robot + Manipulated Object) is rendered frame-by-frame through a virtual camera to produce temporally coherent RGB and depth observations.
Output: The final output is a paired dataset of visual observations and action trajectories.

3. Key Contributions

Scalable Data Framework: IGen is the first framework to enable automated, grounded, and scalable visuomotor data generation directly from open-world images, eliminating the need for human teleoperation or specific scene reconstruction.
Simulation-Free 3D Synthesis: It introduces a novel approach to transform unstructured 2D images into actionable 3D scene representations (point clouds) and generates dynamic interactions via rigid-motion synthesis, bypassing the need for explicit physics-based simulation environments for the initial scene setup.
Policy Transferability: The paper demonstrates that policies trained solely on IGen-generated data (with zero real-world human data) can successfully perform real-world manipulation tasks, outperforming policies trained on limited human-collected data.

4. Experimental Results

The authors evaluated IGen across three dimensions: visual fidelity, action quality, and policy transferability.

Visual Fidelity: Compared against "Real-to-Sim" baselines (e.g., Simpler), IGen-generated scenes showed significantly higher similarity to original real-world images.
- Metrics: IGen achieved up to a 5.13× improvement in perceptual similarity (LPIPS) compared to baselines.
Action Quality: Evaluated on the DreamGen Bench using LLMs (GPT-4o, Qwen-3-VL) to assess instruction following and physics alignment.
- Results: IGen outperformed video-generation baselines (Cosmos, TesserAct) by a large margin, producing nearly 2× more instruction-following successes and demonstrating superior physical coherence.
Computational Efficiency: IGen is significantly more efficient than video-generation baselines.
- Speed/Memory: It requires only 8.3 GB GPU memory and ~18.6 seconds per sample, achieving 30× to 200× higher efficiency than TesserAct and Cosmos, respectively.
Real-World Policy Performance:
- Setup: A $\pi_0$ (Vision-Language-Action) model was fine-tuned on IGen data and deployed on a Franka Research 3 robot.
- Tasks: "Water Flowers," "Hit Box," "Place Toy."
- Outcome: A policy trained on 1,000 IGen-generated samples achieved a 66.7% success rate, surpassing a policy trained on 100 human-teleoperated samples (58.3%) and significantly outperforming zero-shot baselines. Notably, even with only 100 IGen samples, the policy (44.5%) outperformed the 100-sample human baseline (19.4%).

5. Significance

IGen represents a paradigm shift in robot learning by decoupling data collection from physical hardware constraints. By leveraging the abundance of open-world images, it offers a low-cost, scalable solution to the data bottleneck in robotics. The ability to train generalist policies entirely on synthetic data derived from single images suggests a future where robots can learn from the vast visual history of the internet, rapidly adapting to new environments and tasks without the need for expensive, time-consuming human demonstrations. This work paves the way for the rapid deployment of generalist robots in diverse, unstructured real-world scenarios.