💻 computer science

Imagine2Real: Towards Zero-shot Humanoid-Object Interaction via Video Generative Priors

Imagine2Real is a zero-shot framework for whole-body Humanoid-Object Interaction that overcomes data scarcity and retargeting complexity by unifying robot and object motions into 4D point trajectories and leveraging a Behavior Foundation Model's latent space for sparse keypoint tracking, enabling flexible, geometry-free physical deployment.

Original authors: Jiahe Chen, ZiRui Wang, Feiyu Jia, Xiao Chen, Xiaojie Niu, Weishuai Zeng, Tianfan Xue, Xiaowei Zhou, Jiangmiao Pang, Jingbo Wang

Published 2026-05-22

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiahe Chen, ZiRui Wang, Feiyu Jia, Xiao Chen, Xiaojie Niu, Weishuai Zeng, Tianfan Xue, Xiaowei Zhou, Jiangmiao Pang, Jingbo Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot to pick up a box, push a door, or even hit a pillar like Iron Man, but you don't have a library of thousands of videos showing robots doing these specific things. In fact, you have almost no data on how robots interact with objects. This is the problem the paper "Imagine2Real" tries to solve.

Here is the story of how they did it, broken down into simple concepts and analogies.

The Big Problem: The "Missing Manual"

Usually, to teach a robot to do something complex, you need a massive amount of high-quality 3D data (like a perfect digital twin of the robot moving exactly how you want). But this data is incredibly rare and expensive to make.

Without this data, robots are stuck. They can walk well, but the moment they try to touch or move an object, they get confused. Existing methods try to fix this by using 2D videos, but they run into two big headaches:

The Geometry Trap: To understand a video, old methods need a perfect 3D blueprint (CAD model) of the object. If you don't have the blueprint, the robot gets lost.
The "Stretchy Suit" Problem: To make a human video work for a robot, you have to mathematically "morph" the human body to fit the robot's body. This is like trying to stretch a human-sized suit onto a robot; it often rips, distorts, or creates awkward, unnatural movements.

The Solution: Imagine2Real

The authors propose a new way called Imagine2Real. Think of it as a "Dream-to-Reality" pipeline. Instead of needing a perfect 3D manual, they let the robot "imagine" the action first, then figure out how to do it.

Here is the three-step process:

1. The Dreamer (Video Generation)

First, you give the system a picture of a robot and an object, plus a text instruction like "Pick up the box."

The Magic: A video generator (an AI that makes videos) creates a short clip of the robot doing exactly that.
The Catch: This video is just pixels; it doesn't know the robot's joints or physics. It's just a visual dream.

2. The Tracker (The "Sparse" Eye)

Now, the robot needs to turn that video into actual movement.

Old Way: Try to track every single joint of the robot and the object. This is like trying to follow a dance by watching every finger and toe. It's messy and requires those "stretchy suits" (retargeting) mentioned earlier.
Imagine2Real Way: They only track three critical dots: the robot's feet (base), its left hand, and its right hand. They also track the object.
The Analogy: Imagine trying to follow a dancer in a foggy room. Instead of trying to see their whole body, you just follow the tip of their nose and their hands. If those move correctly, the rest of the body usually follows naturally. This avoids the "stretchy suit" problem entirely.

3. The Brain (The Behavior Foundation Model)

Here is the secret sauce. If you only tell a robot "move your hand here," it might flail its legs and fall over because it doesn't know how to balance.

The Solution: The robot uses a pre-trained "Brain" called a Behavior Foundation Model (BFM).
The Analogy: Think of the BFM as a master dancer who has practiced walking and balancing for years. The robot doesn't need to relearn how to walk. The BFM provides a "safety net" of natural movement. The robot only asks the BFM, "I need to move my hand this way; how do I keep my balance while doing that?"
This allows the robot to use those few "dots" (sparse keypoints) to generate smooth, natural, whole-body movements without falling over.

The Training Process: Learning in Layers

Since they don't have enough data for the specific "picking up boxes" task, they teach the robot in three stages, like leveling up in a video game:

Level 1 (The Walker): They train the BFM on thousands of hours of general human walking data. The robot learns to walk naturally.
Level 2 (The Tracker): They teach the robot to follow those three "dots" (hands and feet) while walking. It learns to coordinate its body to hit those targets.
Level 3 (The Interactor): Finally, they give it a tiny bit of specific data about holding boxes. The robot learns to add a little extra "twist" to its movements to actually grab the object, using the skills from Level 1 and 2 as a foundation.

The Real-World Test

They tested this on a real robot (a Unitree G1) inside a motion-capture room (a room with cameras that track movement perfectly).

The Result: The robot successfully watched a generated video, figured out where to put its hands and feet, and physically picked up boxes, pushed them, and even hit a pillar—all without ever seeing a real human do it first. It was zero-shot, meaning it learned the specific task just by "imagining" it from the video.

Limitations (The Fine Print)

The paper admits two main things that still need work:

The "Blind Spot": The system currently needs a motion-capture room to know where the robot is in the real world. If the robot pushes a box and the cameras can't see the markers (because the box is blocking them), the robot gets confused.
The "One-Way Street": The process is currently open-loop. The robot makes a plan based on the video and executes it. It doesn't constantly check "Is the box actually moving?" and adjust in real-time. Making a truly closed-loop system (where the robot and video generator talk to each other constantly) is the next big challenge.

Summary

Imagine2Real is a framework that lets a robot learn to interact with objects by:

Dreaming the action via video generation.
Watching only the most important parts (hands/feet) to avoid complex math.
Relying on a pre-trained "dance instructor" (the BFM) to keep its balance and move naturally.

This allows robots to learn new tasks instantly without needing a massive library of pre-recorded robot data.

Technical Summary: Imagine2Real

Problem Statement

Whole-body Humanoid-Object Interaction (HOI) is currently bottlenecked by the scarcity of high-fidelity 3D interaction data. While video generative models offer a promising alternative for providing visual and physical priors, existing methods attempting to leverage them for HOI face two critical challenges:

Representation Misalignment: Extracting 3D motion from 2D videos typically relies on strong geometric priors (e.g., explicit CAD models), which are rarely available in real-world settings. Furthermore, existing approaches often estimate robot and object motions independently using separate tools, leading to spatial and depth discrepancies that result in low-quality reference motions and tracking failures.
Retargeting Complexity: Traditional motion tracking requires dense, low-level signals (e.g., joint positions) that must be explicitly retargeted to a specific humanoid embodiment. In HOI scenarios, this process is exceedingly complex, requiring intensive morphing to align robot and object meshes, which inherently amplifies estimation errors from the video source. Conversely, methods that track sparse keypoints often fail to produce natural gaits due to a lack of joint-level information.

Methodology

The authors propose Imagine2Real, a zero-shot HOI framework designed to achieve flexible, geometry-free interaction by unifying robot and object motions and leveraging a Behavior Foundation Model (BFM). The framework operates through a "video-to-motion-to-real" pipeline involving three core components:

1. Unified 4D Point Trajectory Representation

To resolve representation misalignment, the framework avoids relying on geometric priors. Instead, it treats both robot and object motions as a unified 4D point trajectory.

Video Synthesis: Given an initial image and text instruction, a video generative model synthesizes an interaction video.
Extraction: Using segmentation (SAM3) and point tracking (SpaTrackerV2), the system extracts 4D point trajectories for sparse critical regions: the robot's base, hands, and the target object.
Alignment: By representing both entities as point trajectories within a common frame, the method ensures spatial alignment without requiring explicit CAD models or independent estimation pipelines.

2. Behavior Foundation Model (BFM) Based Keypoints Tracker

To overcome retargeting complexity and maintain natural gaits with sparse signals, the framework utilizes a Keypoints Tracker built upon a pre-trained BFM.

BFM Backbone: A BFM is first pre-trained on large-scale proprioceptive motion data (~68.5h) to construct a compact latent motion space. This model internalizes forward dynamics and natural motion transitions.
Sparse Tracking: The tracker does not relearn a whole-body controller. Instead, it conditions on a sparse reference motion (base, hands, object) and applies a latent residual planner on top of the frozen BFM backbone.
Mechanism: The frozen predictor generates a latent prior based on proprioceptive history to ensure natural locomotion and balance. The planner then outputs a low-dimensional latent residual to steer the robot toward the sparse tracking targets. This allows the system to bypass the error-amplifying retargeting process while preserving physical stability.

3. Progressive Training Strategy

To address data scarcity, the framework employs a three-stage progressive training strategy:

BFM Backbone: Trained on diverse whole-body motions to establish a robust motion prior.
Keypoints Tracker: Trained on loco-manipulation data (~8.86h) to learn tracking sparse points while maintaining balance.
Interaction Adaptor: Fine-tuned on specific HOI data (~0.43h) to learn residual joint-level actions for object manipulation. This adaptor bridges the gap between kinematic tracking and physical contact, enabling precise manipulation without complex reward engineering.

4. Real-World Deployment Pipeline

The deployment loop involves:

Video Generation: Synthesizing an interaction video conditioned on text.
Trajectory Extraction: Converting video pixels into stable 4D point trajectories.
Mocap Calibration: Using a motion capture (mocap) system to resolve scale ambiguity inherent in monocular video generation. The system captures initial global positions to scale and calibrate the extracted trajectories to real-world dimensions before executing the policy on a Unitree G1 humanoid robot.

Key Contributions

Unified 4D Point Trajectory Representation: A novel representation for HOI that removes dependency on geometric priors and resolves motion misalignment between robots and objects.
BFM-Based Keypoints Tracker: A tracking mechanism that focuses solely on interaction-critical sparse points (base, hands, object). By utilizing the BFM's latent space as the search domain, it enables flexible interaction while ensuring stable, natural gaits without complex retargeting.
Progressive Training Framework: A strategy that effectively utilizes multi-scale datasets (from large-scale non-interactive data to specific HOI data) to achieve robust zero-shot HOI, eliminating the need for hand-engineered mechanisms like complex contact rewards or physical state initialization.

Experimental Results

The authors evaluated the framework through Sim2Sim and real-world zero-shot deployments:

Motion Naturalness: In ablation studies, the BFM-based approach significantly outperformed direct joint-space tracking and DAgger baselines in terms of action smoothness and gait naturalness (lower $E_{mpjae}$ , $A_{rate}$ , and $A_{smooth}$ ), proving that searching the latent space yields more physically plausible motions.
Task Success: The framework achieved high success rates in whole-body HOI tasks:
- Carrying Boxes: 82.65% success rate with the Interaction Adaptor (vs. 0% without).
- Pushing Boxes: 64.91% success rate with the Adaptor (vs. 29.82% without).
- Tracking errors for hands, base, and objects were maintained within centimeter-level precision.
Zero-Shot Deployment: The system successfully executed diverse physical tasks on a real Unitree G1 robot, including lifting various boxes and performing semantically rich interactions (e.g., hitting an "Iron Man" pillar), demonstrating the efficacy of the video-to-motion-to-real pipeline.

Significance and Limitations

The paper claims that Imagine2Real represents a significant step toward generalizable humanoid control by decoupling interaction learning from the need for massive, task-specific 3D datasets. By leveraging video generative priors and a unified point representation, the framework enables robots to learn complex interactions from sparse visual cues while maintaining physical stability.

However, the authors acknowledge current limitations:

Dependency on Mocap: Real-world deployment currently relies on external motion capture systems, which are susceptible to marker occlusion during close-contact tasks.
Open-Loop Pipeline: The current system is open-loop. While iteratively prompting the video model offers a pseudo-closed-loop approximation, a true closed-loop system requires joint fine-tuning of the video generator and motion tracker using physical feedback, which remains challenging due to the closed-source nature of top-tier video models and the lack of physical consistency in open-source alternatives.