💻 computer science

AGILE: Hand-Object Interaction Reconstruction from Video via Agentic Generation

AGILE is a robust framework that reconstructs dynamic hand-object interactions from monocular videos by shifting from traditional reconstruction to an agentic generation paradigm, utilizing a Vision-Language Model to synthesize complete object meshes and an anchor-and-track strategy to bypass fragile Structure-from-Motion initialization, thereby producing simulation-ready assets with high geometric accuracy and physical plausibility.

Original authors: Jin-Chuan Shi, Binhong Ye, Tao Liu, Xiaoyang Liu, Yangjinhui Xu, Junzhe He, Zeju Li, Hao Chen, Chunhua Shen

Published 2026-04-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jin-Chuan Shi, Binhong Ye, Tao Liu, Xiaoyang Liu, Yangjinhui Xu, Junzhe He, Zeju Li, Hao Chen, Chunhua Shen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a perfect, working digital twin of a person picking up a coffee mug, but you only have a single, shaky video of it happening. The video is messy: the hand covers the mug half the time, the lighting changes, and the mug spins fast.

Most current computer programs try to "reconstruct" this by piecing together the visible parts like a puzzle. But when the hand covers the mug, the puzzle is missing pieces. The result is a glitchy, broken 3D model that falls apart if you try to use it in a video game or a robot simulator.

AGILE is a new system that changes the game. Instead of trying to fix a broken puzzle, it acts like a super-smart director who watches the video, imagines the missing parts, and then builds a brand-new, perfect 3D movie prop from scratch.

Here is how AGILE works, broken down into three simple steps:

1. The "Agent" Director (The Brain)

Think of the Vision-Language Model (VLM) in AGILE as a film director with a sharp eye.

The Problem: In a video, the hand often hides the object. If you just look at one frame, you might only see half a mug.
The AGILE Solution: The "Director" scans the whole video and picks the best 4 or 5 moments (keyframes) where the object is visible from different angles. It then asks an AI artist to "paint" what the object looks like from the back, top, and sides—angles that were hidden in the original video.
The Quality Control: The Director doesn't just accept any painting. It acts as a critic, checking: "Does this back view look like the front view? Is the handle in the right place?" If the AI hallucinates (makes up) a weird shape, the Director rejects it and asks for a redo. This ensures the final 3D model is "watertight" (no holes) and looks exactly like the real object.

2. The "Anchor and Track" Strategy (The GPS)

Once the Director has built the perfect 3D model of the object, the system needs to figure out where it is in the video at every single second.

The Old Way (SfM): Traditional methods try to build a map of the room by triangulating points, like a surveyor. But if the object is moving fast or covered by a hand, the surveyor gets lost, and the whole system crashes.
The AGILE Way: Imagine you are tracking a friend in a crowded room. You don't need to map the whole room. You just find them once when they are clearly visible (the "Anchor").
- AGILE finds that one perfect moment where the hand grabs the object.
- It locks the 3D model onto the hand at that exact moment.
- Then, as the video plays, it simply tracks the object, using the strong visual similarity between its perfect 3D model and the video frames. It's like sticking a high-quality sticker on a moving car; even if the car speeds up, the sticker stays on because it knows exactly what the car looks like.

3. The "Physics Glue" (The Reality Check)

The final step is making sure the digital hand and object behave like real things.

The Problem: In many digital reconstructions, the hand might pass through the object (like a ghost) or the object might float away from the hand.
The AGILE Solution: The system applies "magnetic glue." It uses physics rules to say, "If the hand is holding the object, they must move together." If the video gets blurry or the hand covers the object again, the system relies on this "glue" to keep the object locked in the hand, preventing it from sliding or jittering.

Why Does This Matter?

Think of the difference between a paper cutout and a LEGO set.

Old methods give you a paper cutout of a hand holding a mug. It looks okay from the front, but if you try to turn it around, it's flat and broken. You can't use it to teach a robot how to pick up a mug because the robot would try to grab through the paper.
AGILE gives you a LEGO set. It builds a solid, 3D object with real texture and shape. Because it is physically accurate, you can drop this digital model into a robot simulator, and the robot can learn to pick up the mug just like a human.

In short: AGILE stops trying to "fix" broken videos. Instead, it uses AI to imagine the missing parts, builds a perfect 3D model, and then tracks it with a strategy that never loses its grip, even when the object is hidden. It turns messy, real-world videos into clean, simulation-ready digital assets.

1. Problem Statement

Reconstructing dynamic hand-object interactions (HOI) from monocular videos is critical for robotics, VR, and digital twin creation. However, existing methods face two fundamental barriers:

Geometric Fragmentation: State-of-the-art reconstruction methods relying on neural rendering (e.g., NeRF, 3D Gaussian Splatting) often produce fragmented, non-watertight geometries when objects are heavily occluded by hands. These representations are unsuitable for physics engines requiring explicit topology.
Brittle Initialization: Most pipelines depend on Structure-from-Motion (SfM) (e.g., COLMAP) for initial pose estimation. SfM fails frequently in dynamic scenes with textureless objects, rapid motion, or severe occlusion, causing the entire pipeline to collapse.
Generative Limitations: While generative models (e.g., diffusion) can hallucinate missing views, they often lack consistency with the specific video content, leading to "hallucinations" that do not match the actual object, and they typically lack the multi-view aggregation needed for high-fidelity 3D assets.

2. Methodology: The AGILE Framework

AGILE shifts the paradigm from traditional reconstruction to agentic generation. It consists of three main stages:

A. Agentic Textured Object Generation

Instead of optimizing a scene representation directly, AGILE synthesizes a complete, watertight 3D mesh using a Vision-Language Model (VLM) as an intelligent supervisor.

VLM-Guided Keyframe Selection: A VLM agent selects a minimal set of informative keyframes (1–4) from the video to maximize viewpoint coverage while minimizing occlusion.
Multi-View Synthesis & Rejection Sampling: The VLM guides an image generation model to synthesize orthogonal views of the object. Crucially, the VLM acts as a critic, evaluating the generated views against the original video frames for geometric and texture consistency. Views failing a strict consistency threshold are rejected and re-generated.
3D Lifting & Refinement: Validated multi-view images are lifted into a 3D mesh using a feed-forward 3D generator. An automated texture refinement phase follows, where an image-to-image model enhances high-frequency details. The VLM again supervises this step to ensure the texture matches the video observations, resulting in a photorealistic, watertight mesh.

B. SfM-Free Initialization (Anchor-and-Track)

AGILE eliminates the dependency on fragile SfM pipelines.

Metric Initialization: The hand is initialized using a robust estimator (WiLoR) and metric depth (MoGe-2) to recover scale and pose.
Interaction Onset Frame (IOF) Anchoring: The object pose is initialized only at a single frame where interaction begins (the IOF). This is done using a foundation model (FoundationPose) which leverages the high-fidelity generated mesh and metric depth.
Temporal Propagation: Instead of re-initializing every frame, the pose is propagated temporally. The system relies on the strong visual similarity between the generated textured mesh and the video observations.

C. Contact-Aware Optimization

A bi-directional online optimization refines the trajectories frame-by-frame from the IOF.

Hand Refinement: The hand translation is optimized using joint reprojection loss ( $L_{joint}$ ) to align with 2D keypoints, keeping rotation and scale fixed.
Object Tracking: The object pose is optimized using a composite loss function:
- $L_{mask}$ : Ensures silhouette alignment with segmentation masks.
- $L_{dino}$ : Uses DINOv3 semantic features to maintain consistency in textureless or occluded regions.
- $L_{interact}$ : A novel interaction stability loss. It enforces that the relative spatial configuration between the hand and object remains stable during a grasp. It uses a Signed Distance Function (SDF) to weight vertices, "locking" the object to the grasping parts of the hand to prevent sliding or interpenetration.

3. Key Contributions

Agentic HOI Pipeline: The first framework to integrate VLM-guided quality assessment with generative models, enabling the production of high-fidelity, watertight meshes independent of video occlusions.
Robust Anchor-and-Track Strategy: A novel initialization method that bypasses SfM entirely by anchoring pose at a single contact frame and propagating it via semantic and geometric alignment.
Physics-Aware Constraints: The integration of interaction stability constraints ( $L_{interact}$ ) ensures physical plausibility, preventing unnatural sliding and interpenetration, which is critical for simulation-ready assets.
Real-to-Sim Capability: The method produces assets that are directly usable in physics simulators (e.g., Isaac Gym) without manual post-processing.

4. Experimental Results

The authors evaluated AGILE on DexYCB, HO3D-v3, and a curated In-the-Wild dataset.

Quantitative Performance:
- Geometric Accuracy: AGILE achieved a Chamfer Distance (CD) of 0.52 cm² on DexYCB, outperforming MagicHOI (2.05 cm²) by nearly 75%.
- Interaction Stability: The Hand-relative CD ( $CD_h$ ) was 94.60 cm² for AGILE, compared to 661.90 cm² for MagicHOI, indicating significantly better physical contact maintenance.
- Robustness (Success Rate): AGILE achieved a 100% Success Rate on challenging sequences. In contrast, MagicHOI failed on 75% of sequences, and HOLD failed on 55%. Notably, AGILE's accuracy on the entire dataset (including hard cases) surpassed MagicHOI's accuracy on its easy subset.
Qualitative Results: AGILE successfully reconstructed complex objects (e.g., scissors, game controllers) with fine textures and watertight geometry, whereas baselines produced fragmented meshes or "floating" objects.
Ablation Studies: Removing the agentic multi-view generation caused a catastrophic rise in geometric error (CD from 0.27 to 2.85). Removing texture refinement led to initialization drift. Removing interaction constraints caused severe physical violations (sliding/penetration).

5. Significance and Impact

Scalability for Robotics: By converting unstructured internet videos into simulation-ready digital twins, AGILE enables the large-scale collection of dexterous manipulation data for training robotic policies.
Paradigm Shift: It demonstrates that agentic generation (using AI agents to curate and validate data) is superior to traditional geometric optimization for handling severe occlusions in dynamic scenes.
Physical Validity: Unlike many generative methods that prioritize visual appearance over physical correctness, AGILE explicitly enforces physical constraints, making its output directly applicable to physics-based simulation and real-world robotic control.

In summary, AGILE solves the "occlusion and initialization" bottlenecks in HOI reconstruction by leveraging VLMs to generate consistent 3D assets and using contact-aware optimization to track them, resulting in a robust, simulation-ready pipeline.