🤖 AI

SceneBot: Contact-Prompted General Humanoid Whole Body Tracking with Scene-Interaction

SceneBot is a unified motion-tracking framework that enables humanoid robots to seamlessly generalize across free-space locomotion and complex contact-rich tasks by conditioning a single policy on reference motions and inferred scene-interaction graphs derived from a novel hindsight scene reconstruction approach.

Original authors: Sirui Chen, Shibo Zhao, Zhen Wu, Jiaman Li, Guanya Shi, C. Karen Liu

Published 2026-06-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sirui Chen, Shibo Zhao, Zhen Wu, Jiaman Li, Guanya Shi, C. Karen Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine teaching a robot to walk, dance, and carry heavy boxes up a staircase. For a long time, robot trainers faced a tricky problem: they could teach robots to move perfectly in empty rooms (free space), but as soon as the robot had to touch a wall, step on a stair, or grab a heavy object, it would get confused. It's like telling a dancer to "move your hand to the box," but not telling them how to touch it—should they just brush against it, or grab it tight enough to lift it? Without that specific instruction, the robot doesn't know if it's supposed to be floating or pushing.

SceneBot is a new solution created by researchers from Stanford and Amazon. Think of it as a "universal translator" that teaches a single robot brain how to handle both empty rooms and messy, cluttered environments where touching things is necessary.

Here is how it works, broken down into simple concepts:

1. The "Contact Map" (The Secret Ingredient)

Most robots just try to copy a human's movements (kinematics). SceneBot adds a second layer of instruction: Contact Labels.

The Analogy: Imagine you are learning to carry a heavy suitcase up stairs. A normal instruction says, "Move your arm like this." SceneBot adds a sticky note that says, "Your hand must be pressing down on the handle," and "Your foot must be firmly planted on the step."
How it helps: This tells the robot exactly which body parts need to push, pull, or rest against the world. It turns vague movement instructions into a physical game of "connect the dots" with the environment.

2. The "Time-Travel" Data Engine

To teach a robot this, you need thousands of examples of robots interacting with stairs and boxes. But real-world data is rare, and filming robots doing this is slow and expensive.

The Problem: We have tons of videos of humans dancing, walking, and carrying things, but we don't have the 3D models of the stairs or boxes they were interacting with.
The Solution (Hindsight Reconstruction): The researchers built a clever "time-travel" system. They take a video of a human moving, and the computer works backward to invent the scene.
- If the human's hand stops moving near a spot, the computer says, "Ah, they must have been holding a box here," and it 3D prints a virtual box.
- If the human's foot steps up, the computer says, "They must have been on a stair," and it builds a virtual staircase.
The Result: They turned hours of human motion data into a massive training gym filled with virtual stairs, boxes, and uneven ground, all perfectly matched to the robot's movements.

3. The "One-Brain" Policy

Usually, you need one brain for walking, another for climbing stairs, and a third for carrying things. SceneBot trains one single brain to do it all.

The Magic: By feeding the robot both the "move like this" instructions and the "touch this" instructions, the robot learns a general skill. It can walk across a flat floor, then immediately switch to carrying a box up a flight of stairs without changing its "software."
Real-World Test: The paper shows the robot successfully picking up a box, carrying it, and walking up stairs all in one continuous motion.

4. The "GPS" for the Robot's Head

To keep the robot from getting dizzy or falling over, the researchers also improved how the robot knows where it is in the world.

The Issue: Robots often get confused about their own orientation (which way is up) when they move their heads quickly.
The Fix: They combined data from a laser scanner (LiDAR) and a sensor on the robot's hips (like a human's inner ear) to create a super-accurate "GPS" that keeps the robot balanced even when it's doing complex tricks.

Summary

SceneBot is like giving a robot a "super-sense" of touch. Instead of just blindly copying human moves, it learns where and how to touch the world. By using a clever trick to invent training scenes from human videos, the researchers taught a single robot policy to handle everything from dancing in an empty room to carrying heavy furniture up a staircase, all without needing a different program for each task.

The researchers state that this is the first framework to seamlessly unify free-space movement with complex, contact-heavy tasks, and they plan to share their code and data so others can build on it.

Technical Summary: SceneBot – Contact-Prompted General Humanoid Whole Body Tracking

Problem Statement

Current humanoid reinforcement learning (RL) policies have achieved significant success in tracking free-space motions (e.g., walking, running, dancing) but struggle with contact-rich tasks. Pure kinematic tracking is insufficient for interacting with objects and uneven terrain because it cannot resolve physical ambiguities. For instance, moving a wrist toward a box does not inherently specify whether the robot is reaching, touching, or lifting; similarly, placing a foot near a stair edge does not guarantee the generation of sufficient ground reaction force to step up. Existing pipelines lack the scene-awareness required to coordinate interactions with both objects and terrain over long horizons, often relying on high-level teleoperation or failing to generalize to complex loco-manipulation tasks.

Methodology

SceneBot is a unified motion-tracking framework designed to handle free-space locomotion, terrain traversal, and whole-body manipulation within a single policy. The methodology consists of three primary components:

1. Hindsight Scene Reconstruction

To address the scarcity of large-scale datasets containing paired motion and scene interaction data, SceneBot introduces a hindsight scene reconstruction pipeline:

Input: Retargeted human motion trajectories.
Scene Interaction Graph: The system constructs a heterogeneous graph capturing temporal interactions between robot key links (wrists, feet, pelvis) and scene types (terrain, objects). Edges are added based on low relative velocity/acceleration and pruned if they violate collision or force-closure constraints.
Asset Generation: Based on the interaction graph, the system reconstructs plausible 2.5D elevation maps for terrain and parallel plate objects for movable items. This transforms ordinary human motion data into contact-rich robot-scene interaction data suitable for training.

2. Contact-Conditioned Policy Training

The core policy is trained using Proximal Policy Optimization (PPO) to track both kinematic reference motions and explicit contact labels.

Inputs: The policy takes kinematic commands ( $g_t$ ), proprioceptive states ( $s_t$ ), root state estimates ( $p_{root,t}$ ), and a binary contact label vector ( $c_t$ ). The contact labels explicitly indicate which robot links should establish contact with terrain or objects.
Reward Structure: The training objective includes standard motion tracking rewards augmented with two contact-specific terms:
- Contact Correctness Reward ( $r_{cr}$ ): Encourages the robot to establish contact when the label is 1. This reward is asymmetric; it is only active when a contact is desired, preventing penalties for unintentional contacts in free-space scenarios.
- Contact Duration Reward ( $r_{dr}$ ): Encourages stable contact maintenance, clipped at 0.5 seconds to prevent the robot from refusing to release objects.
Stabilization: A heuristic "magic force" is applied to reconstructed objects during early training to ensure they follow the trajectory until a stable grasp is achieved, mitigating frequent grasp-failure terminations.

3. Infrastructure-Free State Estimation

For deployment, SceneBot utilizes SuperOdometry combined with onboard sensors to estimate the robot's global root state (position, orientation, linear velocity).

Sensor Fusion: Recognizing that head-mounted IMUs (e.g., Livox) may not accurately represent pelvis dynamics during aggressive motions, the system fuses a pelvis-mounted IMU (for pitch/roll) with SuperOdometry's yaw estimation via a Kalman filter.
Initialization: Root height is estimated using the lowest 5% of points from a gravity-aligned LiDAR scan.

Key Contributions

Unified Framework: SceneBot is presented as the first general motion tracking framework capable of seamlessly unifying free-space locomotion, uneven terrain traversal, and whole-body object manipulation within a single policy.
Contact Conditioning: The paper introduces a novel interface where a high-level decision maker provides contact labels (which links should touch the scene) alongside reference motions. This allows the low-level policy to resolve physical ambiguities without requiring a full vision-based task policy.
Data Synthesis Engine: The proposed hindsight scene reconstruction approach generates scalable, contact-rich training data from existing human motion datasets (AMASS, OMOMO, Bones, Lafan), overcoming the bottleneck of scarce annotated interaction data.
Zero-Shot Generalization: The system demonstrates the ability to generalize to unseen motions and unseen environments, including complex long-horizon tasks like carrying a box upstairs.

Experimental Results

Experiments were conducted on the Unitree G1 humanoid platform in a MuJoCo sim-to-sim environment and validated with real-world state estimation.

Performance Comparison: SceneBot achieves performance on par with state-of-the-art free-space trackers (SONIC) in flat-terrain scenarios but significantly outperforms them in scene interaction tasks.
- Terrain Interaction: Success rates dropped from 100% (with global state) to 45% when global root information was removed.
- Object Interaction: Without hand contact labels, grasp success rates dropped to near zero.
Ablation Studies:
- Global State: Incorporating global root position and velocity is critical; local tracking alone leads to drift and motion-terrain misalignment.
- Contact Labels: Training with contact labels is essential for success. Interestingly, a model trained with labels retains better performance (85% success) even when labels are zeroed out during testing, compared to a model trained without them (45% success), suggesting the contact conditioning simplifies the learning of balancing mechanics.
- Reconstruction vs. Retargeting: Direct scene reconstruction outperforms scene-aware retargeting (e.g., OmniRetarget) because it guarantees the generated scene is feasible for the motion, whereas retargeting often struggles with object alignment and penetration constraints.
State Estimation: The proposed state estimation pipeline achieved an average root position error of 0.032 m, orientation error of 0.018 rad, and linear velocity error of 0.092 m/s against OptiTrack ground truth, with an 80% task success rate in real-world trials.

Significance and Claims

The paper claims that SceneBot establishes contact conditioning as a powerful interface for humanoid control, effectively bridging the gap between free-space motion tracking and complex, contact-rich scene interactions. By synthesizing training data through hindsight reconstruction, the framework enables a single policy to execute diverse, long-horizon tasks (e.g., picking up a basket, climbing stairs, and placing it) without requiring task-specific tuning. The authors position this work as a foundational step toward physical intelligence where robots can intentionally leverage contacts for support and manipulation. All code and data are intended to be open-sourced.