💻 computer science

CHOIR: Contact-aware 4D Hand-Object Interaction Reconstruction

The paper presents CHOIR, a framework that reconstructs 4D hand-object interactions from monocular videos by leveraging contact as an explicit coupling signal to rectify misalignments and enforce geometric and temporal consistency, thereby enabling scalable mining of reusable interaction primitives from open-world scenes.

Original authors: Hao Xu, Yilin Liu, Yinqiao Wang, Chi-Wing Fu, Niloy J. Mitra

Published 2026-05-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hao Xu, Yilin Liu, Yinqiao Wang, Chi-Wing Fu, Niloy J. Mitra

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a home video of someone picking up a coffee mug, taking a sip, and putting it back down. To your eyes, it looks simple. But if you tried to turn that flat, 2D video into a 3D movie where you could walk around the hand and the mug, it would be a nightmare. Why? Because the hand often blocks the view of the mug, the mug might look like a different shape from different angles, and it's hard to tell exactly where the fingers are touching the ceramic.

The paper introduces CHOIR, a new computer program designed to solve this exact puzzle. Think of CHOIR as a "3D detective" that watches a regular video and reconstructs a perfect, physics-accurate 3D movie of the hand and object interacting.

Here is how it works, broken down into three simple steps using everyday analogies:

The Problem: The "Floating Hand" and "Ghost Mug"

Most current computer programs try to guess the hand's position and the object's shape separately. It's like asking two different people to draw a picture of a handshake: one draws the hand, the other draws the mug. When they put their drawings together, the hand might be floating in mid-air, or the fingers might be sticking through the mug like a ghost. This happens because the computer gets confused by shadows, clutter, and the fact that the video is only 2D.

The Solution: CHOIR's Three-Stage Process

CHOIR fixes this by working in three stages, treating the "touch" between the hand and the object as the most important clue.

Stage 1: The Rough Sketch (Open-World Analysis)

First, CHOIR watches the video and makes a "rough sketch." It uses existing AI tools to guess where the hand is and what the object looks like.

The Analogy: Imagine a child drawing a picture of a handshake based on a blurry photo. The lines are there, but the hand might be slightly too big, or the mug might be floating a few inches above the table. It's a "contact-agnostic" sketch, meaning it doesn't worry yet about whether the fingers are actually touching the mug; it just gets the general shapes and movements down.

Stage 2: The "Reality Check" (Spatial Rectification)

This is the paper's secret sauce. The rough sketch from Stage 1 is often physically wrong (e.g., the hand is too far from the mug). CHOIR uses a "generative" AI model trained on millions of fake hand-grasps to fix the depth.

The Analogy: Think of this as a strict art teacher looking at the child's drawing. The teacher says, "Wait, if you are holding that mug, your fingers must be this close to it, not that far." The teacher uses a special tool (called a "flow-matching" model) to push the hand closer to the mug or pull it back, correcting the 3D depth so the hand and mug are in the right place relative to each other. This happens before the computer tries to figure out exactly which finger is touching which spot.

Stage 3: The Final Polish (Contact-Aware Optimization)

Now that the hand and mug are in the right place, CHOIR looks for the specific points of contact. It then runs a final "polishing" process to make sure everything stays consistent.

The Analogy: Imagine the child and the teacher are now working together to refine the drawing. They add a "magnetic" rule: "If the finger is touching the mug, it cannot float away, and it cannot go inside the mug." The program constantly checks the video to ensure the hand follows the video's shadows (image alignment) while obeying the laws of physics (no floating, no passing through solid objects). It smooths out the movement so the hand doesn't jitter, creating a fluid, realistic 3D animation.

What Does CHOIR Actually Do?

The paper claims that CHOIR can take a messy, everyday video (even with cluttered backgrounds or objects the computer has never seen before) and output:

3D Hand Motion: A precise 3D model of the hand moving.
Object Shape & Pose: A 3D model of the object and how it moves through space.
Contact Evidence: A map showing exactly when and where the hand touched the object.

Why Is This a Big Deal?

Previous methods often failed when the object was hidden (occluded) or when the video was taken in a messy room. CHOIR succeeds because it uses contact as a guide. Instead of guessing the hand and object separately, it realizes that "touching" is a strong physical rule that forces the two to align correctly.

Limitations (What It Can't Do Yet)

The paper is honest about what CHOIR cannot do:

It only works with one hand at a time (no two-handed juggling yet).
It assumes the object is rigid (it can't reconstruct a squishy pillow or a bending piece of cloth).
It relies on the first few seconds of the video to get a good "anchor" of the object; if the object is completely hidden from the start, the program might get lost.

In short, CHOIR is a tool that turns flat, confusing videos into solid, physics-accurate 3D stories of how we interact with the world, using the simple act of "touching" as its compass.

Technical Summary: CHOIR – Contact-aware 4D Hand-Object Interaction Reconstruction

Problem Definition

The paper addresses the challenge of reconstructing 4D hand-object interactions (HOI) from monocular RGB videos captured in open-world, "in-the-wild" settings. The goal is to recover a coherent 4D representation comprising articulated hand motion, object shape, 6D pose trajectories, and explicit contact evidence (when and where contact occurs).

Current state-of-the-art methods struggle in this regime due to several factors:

Unseen Objects: Many pipelines assume known object models or categories, failing on unseen everyday objects.
Weak Coupling: Hands and objects are often estimated separately, leading to misalignment under occlusion, clutter, and monocular depth ambiguity.
Physical Implausibility: Reconstructions often suffer from interpenetration, floating artifacts, or temporal drift because they lack explicit physical constraints like contact.
Data Scarcity: High-fidelity, physically grounded 4D interaction traces with explicit contact are difficult to obtain at scale for real-world scenarios.

The authors frame the problem through the lens of contact, citing Jacob Bronowski: "The world can only be grasped by action, not by contemplation. The hand is more important than the eye ... The hand is the cutting edge of the mind." This motivates treating contact not just as a visual feature, but as the primary coupling signal between hand articulation and object geometry.

Methodology: The CHOIR Framework

CHOIR is a three-stage pipeline designed to transform noisy, contact-agnostic initial estimates into a physically consistent 4D HOI trace.

Stage 1: Open-World HOI Analysis

This stage establishes a coarse, contact-agnostic 4D sequence from visual evidence.

2D Cue Extraction: The system extracts hand bounding boxes, chirality, 2D joints, interacting object boxes, and modal/amodal masks. It employs a self-adaptation procedure to mine object pseudo-labels around motion-selected interaction frames to fine-tune detectors (WiLoR) and uses SAM 2 propagation for temporally complete masks.
3D Geometry Recovery:
- Hand: HaMeR initializes per-frame MANO hand poses and camera-relative translations.
- Object: SAM-3D-Objects reconstructs a canonical object mesh, metric scale, and initial 6D pose from an "anchor frame" (typically the first frame with reliable visibility).
4D Motion Tracking:
- Hand: Dyn-HaMR stabilizes the initial MANO estimates into a 4D trajectory.
- Object: A "guarded follow tracker" based on SAM-3D-Objects freezes the object shape and scale while updating rotation and translation poses bidirectionally. Outliers are rejected, and missing poses are interpolated.
Coarse Isolated Fitting: Before contact reasoning, the system performs isolated fitting to refine the sequence.
- Hand: Optimizes MANO parameters using 2D alignment, monocular depth consistency, anatomical constraints, and temporal stability.
- Object: Uses complementary silhouette terms (repulsion and attraction) to prevent leakage into the background and pull vertices toward uncovered mask regions, alongside temporal and static consistency losses.
- Alignment: A ray-scale alignment step slides the object sequence along the camera ray to match interaction-depth statistics with the hand.

Stage 2: Generative HOI Spatial Rectification

Recognizing that Stage 1 outputs are visually aligned but potentially physically invalid (e.g., hovering or penetrating), this stage corrects relative hand-object placement before constructing contact correspondences.

Generative Model: A flow-matching-based generative model is trained on a synthetic dataset (GraspPair) derived from DexGraspNet, filtered by a physics engine (PyBullet) to ensure stability.
Rectification Process: The model takes a noisy hand estimate, object geometry, and viewing direction as input. It predicts a 1D scalar offset along the camera ray to correct the depth ambiguity, mapping a noisy source grasp to a physically valid target grasp.
Contact Initialization: From the rectified geometry, initial per-frame contact correspondences are established. Hand contact vertices are matched to object surface points based on proximity and normal compatibility, parameterized as barycentric coordinates on the object mesh. These serve as initialization evidence, not final outputs.

Stage 3: Contact-Aware Joint Optimization

This final stage jointly refines the hand and object trajectories using the rectified initialization and contact evidence.

Optimization Objective: The system minimizes a weighted sum of five loss terms:
1. Contact Loss ( $L_{con}$ ): Pulls active hand vertices toward dynamically rebuilt soft contact anchors (barycentric coordinates).
2. Penetration Loss ( $L_{pen}$ ): A one-sided term penalizing hand vertices inside the object.
3. Silhouette Loss ( $L_{sil}$ ): Aligns rendered object silhouettes with amodal masks.
4. Anchor Loss ( $L_{anc}$ ): Anchors the solution to 2D joints, anatomical constraints, and the Stage 1/2 initializations.
5. Temporal Loss ( $L_{temp}$ ): Enforces smoothness in hand and object motion.
Dynamic Constraints: The contact cache is periodically rebuilt during optimization to adapt to the current geometry, ensuring robustness against drift.

Key Contributions

Open-World 4D HOI Reconstruction: A method that extracts 2D interaction cues, performs guarded object tracking, and produces a coarse contact-agnostic sequence without requiring known object templates.
Generative Spatial Rectification: A novel module that predicts ray-depth corrections to rectify hand-object placement before contact correspondence construction, addressing the "chicken-and-egg" problem of needing correct geometry to find contact.
Contact-Aware Joint Optimization: A test-time optimization framework that integrates dynamically updated contact constraints with physical and temporal priors to enforce geometric and temporal consistency.

Experimental Results

The authors evaluate CHOIR on controlled (HO3D) and challenging in-the-wild (TASTE-Rob, self-captured) datasets.

Quantitative Performance (HO3D): CHOIR outperforms state-of-the-art baselines (iHOI, DiffHOI, HOLD, EasyHOI, MagicHOI) in object reconstruction metrics (Chamfer Distance, F-scores) and scale accuracy. It achieves a lower hand-object distance and penetration ratio compared to baselines.
In-the-Wild Performance: On a reference-free benchmark of 100 videos, CHOIR demonstrates superior physical plausibility (lower penetration ratio, smaller hand-object distance) and temporal smoothness compared to baselines, which often fail to produce usable reconstructions on a significant portion of the data due to tracking breakdowns.
Qualitative Results: Visual comparisons show CHOIR produces tighter hand-object alignment and fewer artifacts (floating hands, interpenetration) than baselines, even in challenging scenarios involving small objects, transparent objects, and changing contact points.
Ablation Studies: Removing Stage 2 rectification doubles the hand-object distance, confirming its role in correcting relative placement. Removing contact or penetration constraints in Stage 3 leads to increased interpenetration and less smooth motion.

Significance and Claims

The paper claims that CHOIR successfully turns everyday monocular videos into reusable 4D interaction primitives. By explicitly modeling contact as a transferable coupling signal and employing a staged strategy (analysis $\to$ rectification $\to$ optimization), the method achieves robust joint recovery even when initial estimates are noisy.

The authors emphasize that their approach enables scalable mining of real interactions and supports scene-aware synthesis and planning. However, they maintain a modest scope, acknowledging current limitations:

The method assumes rigid objects and single-hand interactions.
It relies on upstream 2D/3D initialization; failures in segmentation or anchor frame selection can propagate.
It does not currently handle bimanual manipulation or highly non-rigid deformations.

The work positions itself as a step toward automated, physically grounded 4D reconstruction across the interaction lifecycle, moving beyond the restrictive assumptions of previous curated-dataset approaches.