💻 computer science

Real2Sim in HOI: Toward Physically Plausible HOI Reconstruction from Monocular Videos

This paper introduces HA-HOI, a framework that reconstructs physically plausible 4D human-object interactions from monocular videos by adopting a "human-first, object-follow" formulation to ensure contact consistency and enable stable physics-based simulation.

Original authors: Yubo Zhao, Yujin Chai, Yunao Dong, Chengfeng Zhao, Zijiao Zeng, Yuan Liu, Chi-Keung Tang

Published 2026-05-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yubo Zhao, Yujin Chai, Yunao Dong, Chengfeng Zhao, Zijiao Zeng, Yuan Liu, Chi-Keung Tang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a home video of someone picking up a coffee mug and taking a sip. If you asked a standard computer program to "reconstruct" this scene in 3D, it might draw a 3D person and a 3D mug. But here's the problem: the computer might make the mug float just above the person's hand, or the person's fingers might pass right through the mug like a ghost. To the naked eye, it looks okay, but if you tried to use this 3D scene in a video game or a robot simulation, the physics would break—the mug would fall through the floor, or the robot would try to grab air.

This paper, titled "Real2Sim in HOI," introduces a new system called HA-HOI (Human-Anchored Human-Object Interaction) to fix exactly that problem.

Here is the breakdown of how it works, using simple analogies:

The Core Problem: The "Ghost Hand" Issue

Current methods for turning 2D videos into 3D animations treat the human and the object as two separate things moving around. It's like trying to choreograph a dance by telling the dancer where to go and the prop where to go, without telling them to actually touch each other. The result is a 3D scene that looks visually correct from the camera's angle, but physically impossible. The hand might hover near the cup, or the cup might float in mid-air.

The Solution: The "Anchor" Strategy

The authors propose a new way of thinking: The Human is the Anchor, the Object is the Follower.

Instead of trying to guess where the human and the object are independently, HA-HOI says: "Let's figure out exactly what the human is doing first, and then figure out where the object must be to make sense of that action."

Think of it like a magnet and a paperclip.

The Magnet (The Human): The system first locks onto the human's movement. It treats the human's body as the solid, stable center of the universe for that specific video.
The Paperclip (The Object): Once the human's motion is set, the system figures out where the object is relative to the human. If the human's hand is closed in a "grabbing" shape, the object must be inside that hand. It doesn't just float nearby; it snaps into place.

How It Works (The Three Steps)

1. Building the Foundation (Human-First)
The system watches the video and builds a 3D model of the person moving. It uses this person as the "coordinate system." Instead of asking, "Where is the object in the room?" it asks, "Where is the object relative to the person's hand?" This keeps the scale and timing consistent, even if the camera is shaking or moving around.

2. The "Smart Guess" (VLM Contact)
Sometimes, the video is blurry or the object is hidden behind the person's arm. To solve this, the system uses a "Visual Language Model" (a type of AI that understands images and text).

Analogy: Imagine you are trying to guess where a hidden key is. You don't just look at the dark spot; you ask an expert, "If a person is holding a key, where is it usually located?" The AI suggests likely places where the hand and object should touch. The system then uses these "smart guesses" to nudge the 3D model so the hand actually grips the object, rather than just hovering near it.

3. The "Physics Test" (Simulation)
This is the most unique part. After building the 3D animation, the system doesn't just stop there. It runs the animation through a physics simulator (like a video game engine).

The Metaphor: Imagine you built a wooden puppet show. Before showing it to an audience, you put the puppets on a real stage with real gravity. If the puppet's arm is too heavy and falls off, or if the chair it's sitting on collapses, you know the design is wrong.
HA-HOI does this. It tries to make the human and object interact in a physics engine. If the hand slips off the cup because the grip was too loose, the system fixes the animation so the grip is strong enough to hold the cup under gravity. This ensures the final result isn't just "pretty to look at," but physically stable.

The Results

The authors tested this on a dataset called BEHAVE (which contains videos of people interacting with objects).

Before: Other methods created 3D scenes where the human and object looked close, but often had "ghostly" gaps or objects passing through bodies.
After: HA-HOI created scenes where the contact was solid. The human actually held the object, and the object stayed in the hand.
The Metric: They measured "penetration" (how much the hand went through the object). HA-HOI reduced this error significantly, meaning the 3D models are much more realistic and ready to be used as "teachers" for robots or video game characters.

Summary

In short, this paper argues that to turn a real-world video into a useful 3D simulation, you can't just track the person and the object separately. You have to treat the interaction itself as the most important thing. By anchoring the object to the human's movement and testing the result with physics, HA-HOI turns shaky, ambiguous videos into stable, realistic 3D interactions that robots and simulations can actually use.

Technical Summary: Real2Sim in HOI: Toward Physically Plausible HOI Reconstruction from Monocular Videos

1. Problem Statement

Recovering 4D human-object interaction (HOI) from monocular videos is a critical step for scalable 3D content creation, embodied AI, and simulation-based learning. While recent methods (e.g., VisTracker, CARI4D, THO) have achieved progress in reconstructing temporally coherent human and object trajectories, they suffer from a fundamental "interaction gap." These methods typically treat humans and objects as independent entities in an ambiguous 3D space, resulting in outputs that are visually plausible but physically unstable. Specifically, existing pipelines often fail to preserve stable contact, functional manipulation, or dynamic plausibility when used as reference motions for humanoid-object simulation. The core challenge is that current reconstruction approaches do not explicitly recover the evolving functional relation that binds human and object trajectories into a coherent interaction, making them unsuitable as "teacher trajectories" for physical rollouts.

2. Methodology: HA-HOI

The authors propose HA-HOI (Human-Anchored HOI), a framework designed to reconstruct physically plausible 4D HOI animations from in-the-wild monocular videos. The core insight is that in most HOI scenarios, the human is the active agent, and object motion is conditioned on human action. Consequently, the framework adopts a human-first, object-follow formulation, treating the human body as the organizing coordinate system of the interaction rather than just another reconstructed entity.

The pipeline consists of four distinct stages:

3.1 Human-Centric Interaction Anchor

Instead of reconstructing human and object tracks independently, the system first establishes a temporally coherent human motion as the interaction anchor.

Human Recovery: The system recovers world-space body motion using SMPL-X parameters, fusing MANO hand poses to ensure fine-grained hand articulation critical for interaction.
Depth Calibration: Monocular depth predictions are calibrated to this human anchor. Rather than treating depth as a global scene representation, a scale factor is estimated to align unprojected depth points in the human region with the recovered mesh. This preserves the metric scale of human motion while using depth as local geometric evidence for object placement.

3.2 Object-Follow Reconstruction

With the human anchor established, the object is reconstructed relative to this frame.

Mesh Construction: A canonical object mesh is constructed from visible frames, utilizing image inpainting for occluded regions and single-image 3D reconstruction models.
Scale Estimation: Since monocular reconstruction lacks metric scale, the system estimates an optimal object scale ( $s^\star_o$ ) by aligning the object to observed image silhouettes and calibrated depth, penalizing depth mismatch relative to the human anchor.
Tracking: The scaled object mesh is tracked across the video using FoundationPose. Reliability is ensured via mask-depth consistency, with interpolation used for unreliable frames. The resulting trajectory is transformed into the human-centric world frame, producing a coarse HOI trajectory ( $\tau^0$ ).

3.3 Semantic Contact Refinement

The coarse trajectory may still lack physical coherence (e.g., floating objects or penetrations). To address this, the system employs semantic contact refinement.

Contact Proposals: A Vision-Language Model (InteractVLM) predicts likely contact regions on both the human and object meshes for representative frames.
Optimization: The system optimizes selected body joints, hand poses, and object poses to minimize a composite loss function. This includes:
- Contact Loss ( $L_c$ ): Initially uses bidirectional nearest-neighbor distance, then switches to a signed-distance-field (SDF) objective to pull human contact vertices onto the object surface.
- Penetration Loss ( $L_p$ ): Penalizes mesh intersections.
- Smoothness ( $L_s$ ), Anchoring ( $L_a$ ), and Regularization ( $L_r$ ): Ensure temporal stability and adherence to the coarse reconstruction.
Progressive Optimization: The process iterates from adjusting object depth under fixed human motion to jointly optimizing object pose, hand pose, and body joints, finally refining wrist and finger poses.

3.4 Physics-Based Interaction Projection

The final stage projects the kinematic trajectory into a physics-based simulation to ensure dynamic plausibility.

Simulation Setup: The optimized human sequence is converted into a humanoid reference motion, and the object is modeled as a dynamic rigid body.
Residual Controller: A sequence-specific residual controller is trained. It observes the simulated state and the reference state, outputting a residual action to modify the reference joint target.
Training Objective: The reward function balances motion fidelity (human tracking, object trajectory agreement, hand articulation) with interaction stability (active contact maintenance, velocity constraints).
Output: The final output is a physically plausible animation ( $\tau_{phys}$ ) that remains close to the input video but supports stable interaction under gravity and contact dynamics.

3. Key Contributions

HA-HOI Framework: A scalable, training-free pipeline that transforms in-the-wild monocular HOI videos into simulation-ready trajectories, shifting the focus from visually aligned 4D tracking to physically grounded interaction animation.
Human-First, Object-Follow Formulation: A novel reconstruction strategy that uses world-grounded human motion as the interaction anchor, reconstructing object motion relative to human action and refining contact through VLM-guided proposals and explicit optimization.
Physics-Grounded Evaluation: The demonstration that HA-HOI achieves accurate 4D HOI reconstruction while significantly improving physical plausibility, contact consistency, and temporal stability for humanoid-object simulation, moving beyond standard geometric metrics.

4. Experimental Results

The method was evaluated on the BEHAVE dataset, a multi-view RGB-D benchmark with pseudo-ground-truth registrations. The evaluation protocol used monocular RGB inputs without ground-truth object templates, contact annotations, or manual adjustments.

Quantitative Performance: HA-HOI achieved competitive or superior results across all metrics compared to state-of-the-art methods (CHORE, InterTrack, VisTracker, CARI4D, THO).
- Human Accuracy: Achieved the lowest Chamfer distance for humans (CD-h: 7.11 cm) and the lowest human acceleration error (Acc-h: 0.52 cm/s²).
- Interaction Plausibility: Achieved the lowest human-object penetration score (0.013 cm), a significant improvement over the best baseline (0.084 cm). This indicates a drastic reduction in physically invalid intersections.
Qualitative Analysis: Visual comparisons show that HA-HOI maintains functional hand-object relations (e.g., stable grasps, correct object placement) where baselines like CARI4D often produce floating objects or ambiguous contacts. The method successfully preserves interaction semantics even after physics simulation.
Ablation Studies:
- Removing contact refinement increased penetration from 0.013 to 0.097, confirming that contact optimization is essential for physical validity, even if it slightly alters geometric alignment scores.
- The method remains robust without ground-truth object meshes or camera parameters, validating its applicability to open-world scenarios.

5. Significance and Claims

The paper argues that monocular 4D HOI reconstruction must move beyond the independent tracking of humans and objects to recover the interaction relation itself. By treating the human as the organizing principle and enforcing physical constraints through simulation, HA-HOI bridges the gap between visual observation and physical reality.

The authors claim that this work represents a step toward turning general monocular HOI videos into scalable demonstrations for humanoid-object behavior. While the current work focuses on simulation-ready trajectories, the authors note that the ultimate goal is to deploy these reconstructions on real humanoid robots, though challenges regarding sim-to-real transfer and hardware limitations remain for future work. The significance lies in redefining the objective of HOI reconstruction: not just to "look right" in 3D, but to "act right" under physical laws.