Technical Summary: Towards in-the-wild Egocentric 3D Hand-Object Pose Estimation

Problem Statement

Estimating accurate 3D hand–object pose from in-the-wild egocentric RGB images remains a significant challenge due to severe occlusions, ambiguous contact regions, and the wide diversity of objects and hand configurations. While existing learning-based methods have achieved success in controlled, in-lab settings, they struggle to generalize to real-world scenarios. A primary bottleneck is the scarcity of high-quality supervision; obtaining ground-truth 3D poses for both hands and objects in unscripted environments typically requires expensive motion capture (MoCap) systems, which are impractical for large-scale data collection. Furthermore, existing datasets often feature minimalist, uncluttered backgrounds that fail to reflect the complexity of real-world egocentric interactions.

Methodology

1. EPIC-Contact Dataset

To address the lack of in-the-wild supervision, the authors introduce EPIC-Contact, a dataset comprising 2,300 video clips (62,300 frames) sourced from the EPIC-Kitchens dataset. The dataset features 9 object categories and captures diverse, cluttered real-world interactions.

The annotation pipeline involves three key stages:

• Hand Contact Annotation: Annotators "paint" contact vertices on a subdivided MANO hand mesh (3,106 vertices) using video context to infer occluded regions. This ensures high-precision labeling of contact regions.
• Bijective Contact Transfer: Contact regions on the hand are parametrized using 2-DoF contact axes. These axes are transferred to the object surface via an interactive tool, establishing bijective correspondences between hand and object vertices. Object scaling is refined using a Vision-Language Model (VLM) to match specific instances in the video.
• EC-fit Optimization: An optimization pipeline (EC-fit) derives posed 3D hand-object meshes from the contact annotations. It initializes poses using WiLoR and category-aware priors, then aligns meshes by minimizing contact loss, an occlusion-aware mask loss, and a penetration loss to ensure physical realism.

2. HOPformer Architecture

The authors propose HOPformer, an end-to-end transformer framework for joint 3D bi-manual hand and object pose estimation from a single RGB image in one forward pass.

• Feature Extraction: The model uses DINOv2 (ViT-G) for object feature extraction and WiLoR for hand pose features.
• Cross-Attention Decoder: A core innovation is the use of a cross-attention decoder that conditions object features on hand priors. Unlike previous methods that treat hand and object features independently or via simple concatenation, HOPformer iteratively refines object representations by attending to hand context. This allows the model to leverage robust 3D hand reconstruction priors to guide object pose estimation, particularly under occlusion.
• Object Retrieval: The network includes a classification head to predict the object category, retrieving the corresponding 3D mesh from a predefined pool. The regressed pose is then applied to the retrieved mesh.
• Training Signal: The model is trained using a combination of 2D/3D keypoint losses, MANO parameter losses, object classification loss, and a specific interaction loss based on Contact Deviation (CDev) to enforce physical consistency between hand and object meshes.

Key Contributions

1. EPIC-Contact Dataset: The introduction of a large-scale, in-the-wild egocentric dataset with dense, bijective 3D hand–object contact correspondences and posed meshes, enabling training and evaluation on challenging real-world scenarios.
2. HOPformer: A novel transformer-based architecture that jointly predicts bi-manual hand and object poses by leveraging hand priors to enrich object features via cross-attention.
3. State-of-the-Art Performance: The demonstration that HOPformer achieves superior results on both established in-lab benchmarks and the new in-the-wild dataset.

Experimental Results

In-Lab Performance (ARCTIC Dataset)

Evaluated on the ARCTIC dataset (egocentric split), HOPformer achieves a 82.4% Success Rate (SR@0.05), outperforming the previous state-of-the-art, JointTransformer (76.2%), by 6.2 percentage points. It also reduces Contact Deviation (CDev) from 35.0 mm to 31.9 mm and Motion Deviation (MDev) from 10.4 mm to 7.3 mm.

In-the-Wild Performance (EPIC-Contact Dataset)

On the newly introduced EPIC-Contact dataset, HOPformer nearly doubles the success rate compared to baselines (29.8% vs. 17.6% for JointTransformer) and reduces CDev by 75% (from 30.1 mm to 20.7 mm). The model demonstrates robustness to occlusions, clutter, and transparent objects where prior methods often fail to maintain contact consistency.

Ablation Studies

Ablation experiments confirm that the performance gains are driven by:

• Hand-conditioned Cross-Attention: Replacing the cross-attention mechanism with simple concatenation causes a drastic performance drop (SR from 82.4% to 29.4%).
• Specialized Hand Priors: Using generic DINOv2 tokens for hand features instead of WiLoR significantly degrades performance, highlighting the necessity of pose-specialized hand features.
• Interaction Loss: Removing the CDev-based interaction loss increases contact deviation, confirming its role in enforcing physical consistency.

Significance and Claims

The paper claims that EPIC-Contact and HOPformer represent a significant step toward scalable and robust 3D hand-object pose estimation in unscripted environments. By introducing a method to obtain dense contact annotations without MoCap and a model architecture that effectively leverages hand priors, the work addresses the generalization gap between in-lab and in-the-wild scenarios. The authors position their work as a foundation for future research in human-centric applications such as AR/VR, robotics, and assistive technologies, providing a strong baseline for evaluation on challenging, real-world data.