Leveraging Synthetic Data for Enhancing Egocentric Hand-Object Interaction Detection

1. Problem Statement

Hand-Object Interaction (HOI) detection in egocentric (first-person) vision is a critical task for applications in robotics, healthcare, and industrial analysis. However, training robust models is hindered by the scarcity of large-scale, pixel-wise annotated real-world datasets. The process of collecting and labeling HOI data (identifying hands, objects, contact states, and segmentation masks) is resource-intensive.

While synthetic data has been successful in other domains (e.g., autonomous driving), its application to egocentric HOI remains underexplored due to the difficulty of simulating realistic hand poses, object physics, and complex interactions. The paper addresses several open questions:

• How large is the performance gap between synthetic and real data?
• Can synthetic data replace real data, or is it only useful as a supplement?
• How does Domain Adaptation (DA) bridge the gap between synthetic and real domains?
• Does aligning synthetic data with specific real-world characteristics (objects, grasps, environments) improve performance?

2. Methodology

A. The HOI-Synth Benchmark & Data Generation Pipeline

The authors introduce HOI-Synth, a new benchmark comprising three existing datasets (VISOR, EgoHOS, ENIGMA-51) augmented with a novel, automatically labeled synthetic dataset.

The Generation Pipeline (Figure 2):

1. Interaction Generation: Selects random hand-object grasps from DexGraspNet (1.32M robotic grasps). The hand pose is converted from ShadowHand rigging to match SyntheticHumans (customizable 3D human models with diverse ethnicity, clothing, etc.).
2. Environment & Placement: Places the human-object pair into environments from the Habitat-Matterport 3D (HM3D) dataset. A navigation mesh (NavMesh) ensures realistic placement without collisions.
3. Rendering & Annotation: Uses Unity HDRP for high-fidelity rendering.
   • Camera: Simulates a wearable camera (94.4° FOV) attached to the head.
   • Lighting: Randomized for realism.
   • Clutter: Adds distractor objects to increase complexity.
   • Automatic Labeling: Generates pixel-perfect ground truth including 2D bounding boxes, segmentation masks, contact states, and hand-object relations.
   • Output: 1280x720 RGB frames with full annotations.

B. Domain Adaptation Strategies

The study evaluates three adaptation scenarios using the Adaptive Teacher (AT) framework extended for HOI:

1. Unsupervised Domain Adaptation (UDA): Trains on labeled synthetic data + unlabeled real data. Uses a teacher-student architecture with pseudo-labeling and a domain discriminator (GRL) to align feature distributions.
2. Semi-Supervised Domain Adaptation (SSDA): Trains on labeled synthetic data + unlabeled real data + a small subset of labeled real data (10%, 25%, 50%).
3. Fully-Supervised Domain Adaptation (FSDA): Trains on labeled synthetic data + all available labeled real data.

C. Alignment Strategies

To minimize the domain gap, the authors propose aligning synthetic data generation with the target real dataset across three dimensions:

• Objects: Using DINOv2 features and DBSCAN clustering to select synthetic objects that match the visual distribution of real objects.
• Grasps: Using MMPose to extract 2D hand keypoints from real data and filtering synthetic grasps to match real hand configurations.
• Environments: Clustering real scene backgrounds to select synthetic environments that match the target domain's visual context.

3. Key Contributions

1. HOI-Synth Benchmark: The first benchmark specifically designed to evaluate synthetic-to-real domain adaptation for egocentric HOI detection across three diverse datasets (VISOR, EgoHOS, ENIGMA-51).
2. Novel Generation Pipeline: A scalable, automated pipeline capable of generating photorealistic, diverse, and accurately annotated HOI data, including complex contact states and segmentation masks.
3. Comprehensive Empirical Analysis: A systematic study answering 8 key research questions regarding the scale, alignment, and efficacy of synthetic data in various supervision regimes.
4. Open Source Release: Public release of the generated data, the simulator, and the codebase to facilitate future research.

4. Key Results

Performance Gains

• Synthetic-Only Baseline: Models trained only on synthetic data perform poorly (AP ~10-30%), confirming a significant domain gap (30-40% AP drop compared to real-only models).
• Unsupervised Domain Adaptation (UDA): Bridging the gap is possible. On VISOR, UDA improved Overall AP by +23.45% over the synthetic-only baseline, reaching 33.33%.
• Semi-Supervised (SSDA) - The Sweet Spot:
  • Using only 10% of real labeled data combined with synthetic data outperformed models trained on 100% of real data alone.
  • VISOR: +5.67% AP gain (44.22% vs 38.55%).
  • EgoHOS: +8.24% AP gain.
  • ENIGMA-51: +11.69% AP gain.
• Fully-Supervised (FSDA): Adding synthetic data to full real datasets yields modest but consistent gains (+1% to +4% AP).

Impact of Alignment

• Object Alignment: Had the most significant impact. Aligning synthetic objects to real object classes improved UDA performance by ~1% AP.
• In-Domain vs. Out-Domain: For datasets with unique environments (e.g., ENIGMA-51 industrial setting), in-domain synthetic data (matching the specific environment) was critical. In unsupervised settings, in-domain data outperformed generic data by ~10% AP. However, in semi-supervised settings, the advantage of in-domain data diminished as real labels were introduced.
• Scale: Performance plateaus at ~30,000 synthetic images. Increasing beyond this yields negligible gains.

Robustness

• Backbone Generalization: Results held true when switching from ResNet-101 to ConvNeXt-S, confirming the strategy's robustness to architecture changes.
• Simulation vs. Augmentation: Physically simulated data significantly outperformed simple "cut-and-paste" synthetic augmentation (projecting objects onto ImageNet backgrounds), proving that physical consistency (lighting, occlusion, contact) is crucial.

5. Significance and Conclusion

This work demonstrates that synthetic data is not a standalone replacement for real data but is a powerful complementary resource.

• Scarcity Solution: It effectively mitigates the data scarcity problem, allowing models to achieve state-of-the-art performance with as little as 10-25% of real labeled data.
• Domain Gap Mitigation: The study proves that while a gap exists, it can be effectively bridged through domain adaptation techniques and targeted alignment of synthetic data generation.
• Future Impact: The release of HOI-Synth and the generation pipeline provides a foundational tool for the community to advance egocentric vision, reducing the reliance on expensive manual annotation and enabling rapid prototyping for robotics and industrial applications.

In summary, the paper establishes that synthetic data, when combined with domain adaptation and strategic alignment, can significantly enhance HOI detection, particularly in low-data regimes.