💻 computer science

Pro-HOI: Perceptive Root-guided Humanoid-Object Interaction

This paper introduces Pro-HOI, a generalizable framework for robust humanoid loco-manipulation that combines optimized box-carrying motions, a novel root-trajectory-conditioned training policy, and a persistent object estimation module to enable reliable, long-horizon human-robot interaction in complex real-world scenarios.

Original authors: Yuhang Lin, Jiyuan Shi, Dewei Wang, Jipeng Kong, Yong Liu, Chenjia Bai, Xuelong Li

Published 2026-03-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuhang Lin, Jiyuan Shi, Dewei Wang, Jipeng Kong, Yong Liu, Chenjia Bai, Xuelong Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a humanoid robot as a clumsy but eager new employee. Its boss (the high-level planner) says, "Go pick up that box, walk over to the table, and put it down."

In the past, teaching a robot to do this was like trying to teach a toddler to walk by forcing them to copy a video of a professional dancer frame-by-frame. If the toddler tripped, they couldn't get up. If the box was in a slightly different spot, they froze. They were too rigid and couldn't handle surprises.

Pro-HOI is the new, smarter way to train these robots. Think of it as giving the robot a GPS navigation system combined with a super-internet-connected safety net.

Here is how it works, broken down into simple concepts:

1. The "Root" Compass (The GPS)

Most robots try to memorize every single muscle movement of a human carrying a box. Pro-HOI does something different. It tells the robot: "Don't worry about exactly how your knees bend. Just focus on where your center of gravity (your 'root') needs to go."

The Analogy: Imagine you are carrying a heavy tray of drinks through a crowded party. You don't think about "bend left knee 15 degrees, then right arm 30 degrees." You just think, "I need to move my body forward to the table, and I need to keep my tray level."
The Benefit: Because the robot only cares about the destination of its center, it can walk, run, or dodge obstacles while carrying the box, even if the path is different from what it practiced. It's flexible, not rigid.

2. The "Digital Twin" Safety Net

What happens if the robot trips and the box falls? In the old days, the robot would just stand there, confused, because the box was now out of its camera's view.

Pro-HOI has a secret weapon: a Digital Twin.

The Analogy: Imagine the robot has a twin brother living in a perfect video game simulation inside its brain. When the real robot drops the box, the twin brother instantly simulates the physics: "Okay, the box fell, hit the floor, and rolled three feet to the left."
The Result: Even if the real robot's camera can't see the box anymore, the "twin" knows exactly where it is. The robot then turns its head, finds the box, and picks it up again. It recovers from mistakes on its own, just like a human would.

3. The "No-External-Devices" Rule

Many advanced robot systems need giant cameras and sensors all over the room (like a motion-capture studio) to work. Pro-HOI is designed to run entirely on the robot's own "brain" (a small computer on its back) and its own eyes (cameras).

The Analogy: It's like the difference between a movie actor who needs a full crew, lights, and a green screen to act, versus a street performer who can do an amazing show with just a hat and a microphone. Pro-HOI is the street performer; it's ready to work in a messy, real-world house, not just a perfect lab.

Why This Matters

The researchers tested this on a real robot (the Unitree G1) and found it could:

Carry boxes while running (high-speed agility).
Walk around obstacles without dropping the box.
Pick up a box it accidentally dropped and keep going, even after 15+ cycles in a row.

In a nutshell: Pro-HOI teaches robots to be less like rigid puppets copying a script, and more like skilled humans who know where they are going, can adapt to the terrain, and can fix their own mistakes when things go wrong. It's a major step toward having robots that can actually help us in our homes and workplaces.

1. Problem Statement

Humanoid robots face significant challenges in executing reliable Humanoid-Object Interaction (HOI) tasks, such as carrying boxes, in real-world environments. Existing methods suffer from three primary limitations:

Lack of Generalization: Traditional model-based methods require precise dynamics modeling and task-specific optimization, making them brittle in unknown scenarios. Reinforcement Learning (RL) approaches often rely on extensive reference motions or intricate reward engineering, limiting their ability to adapt to new object positions or task phases.
Absence of a Universal Control Interface: Most existing HOI frameworks lack a standardized command interface for high-level planners. This makes it difficult to integrate object manipulation with simultaneous navigation and obstacle avoidance.
Poor Robustness and Failure Recovery: Current systems often lack closed-loop perception mechanisms. If a robot drops an object (especially outside the camera's field of view), it cannot autonomously detect the failure or re-grasp the object, leading to task termination.

2. Methodology: Pro-HOI Framework

The authors propose Pro-HOI, a generalizable and robust RL framework that decouples motion tracking from the observation space and integrates persistent object estimation.

A. Root-Guided Policy Training

Instead of feeding the policy full-body reference motions (which limits generalization), Pro-HOI uses a Root-Guided Observation Paradigm:

Observation Space: The policy is conditioned solely on the desired root trajectory (deviations in position $\Delta p$ and orientation $\Delta r$ ) and a binary contact state ( $c_t \in \{0, 1\}$ ).
Reward Mechanism: The reference motion is used exclusively as a reward signal (tracking fidelity), not as an input observation. This forces the policy to learn the dynamic correlation between the root movement and the task state rather than blindly mimicking joint angles.
Benefits: This design eliminates the need for complex reward tuning, allows the robot to interact with objects at arbitrary locations, and establishes the root trajectory as a universal interface for high-level planners.

B. Data Preparation & Optimization

Dataset: A custom dataset of box-carrying motions was collected using Xsens inertial sensors and retargeted to the Unitree G1 robot.
SDF Loss Optimization: To resolve mesh penetration artifacts between the robot's hands and the object, a Signed Distance Field (SDF) loss was applied during data retargeting. This ensures physically feasible reference motions before training begins.

C. Persistent Object Estimation & Failure Recovery

To handle real-world uncertainties like accidental drops:

Digital Twin Module: Combines real-time object detection (FoundationPose) with a physics-based simulator (MuJoCo).
Drop Detection: The system monitors the consistency between the contact state and the relative object pose. If the object is expected to be held ( $c_t=1$ ) but the pose deviates significantly, a drop is detected.
Re-grasping: Upon detection, the Digital Twin simulates the object's fall trajectory to predict its final resting position. The robot then autonomously adjusts its gaze and replans its path to re-acquire the object, even if it has fallen out of the camera's field of view.

D. System Architecture

Hardware: Deployed on a Unitree G1 humanoid robot.
Compute: All modules (perception, planning, control) run onboard on a single NVIDIA Jetson NX.
Sensors: Utilizes an Intel D435i camera and a Mid-360 LiDAR.
Planner: Integrates a Task-Specific Planner (TEB) for navigation and obstacle avoidance, generating 6-DoF root trajectories for the policy.

3. Key Contributions

Root-Guided RL Framework: A novel training paradigm that decouples root commands from motion tracking rewards. This achieves generalizable HOI without extensive reference motions or complex reward engineering, while enabling seamless integration with high-level planners for simultaneous navigation and manipulation.
Persistent Object Estimation: A Digital Twin-based module that enables autonomous detection of drops and re-grasping of objects, significantly enhancing system reliability in unstructured environments.
Real-World Deployment: The first demonstration of a complete, robust HOI system running entirely onboard a humanoid robot (Unitree G1), achieving long-horizon execution (15+ consecutive cycles) and successful sim-to-real transfer.

4. Experimental Results

The framework was evaluated on the Unitree G1 in both simulation (MuJoCo) and real-world scenarios.

Generalization (OOD Performance):
- In Out-of-Distribution (OOD) tests (random object positions and orientations), Pro-HOI achieved a 99.93% grasp success rate, significantly outperforming baselines like PhysHSI (82.54%).
- It maintained high placement precision (error < 7cm) even in novel scenarios.
Tracking Fidelity:
- The method achieved the lowest Root Position Error (RPE: 0.22m) and Root Orientation Error (ROE: 5.77 rad) compared to ablation studies, demonstrating precise adherence to root trajectories.
Robustness & Recovery:
- The system successfully completed 15+ consecutive carrying cycles in the real world.
- In failure recovery tests, the Digital Twin module enabled the robot to detect drops, predict the new location, and re-grasp the object autonomously.
Dynamic Agility:
- The robot successfully performed "running while carrying" tasks at speeds up to 0.6 m/s, demonstrating the policy's ability to handle high-dynamic motions.

5. Significance

Pro-HOI represents a critical milestone in transitioning humanoid robots from controlled laboratory demonstrations to practical, human-centric applications. By solving the generalization problem (via root-guided learning) and the reliability problem (via digital twin-based recovery), it provides a holistic solution for whole-body manipulation. The ability to run this complex stack entirely onboard a standard humanoid robot without external motion capture systems or cloud computing makes it a viable candidate for real-world deployment in logistics, service, and disaster response scenarios.