Train, Test, Re-evaluate: Schedule-Sensitive Evaluation of Generative Data for Hand Detection

Technical Summary: Schedule-Sensitive Evaluation of Generative Data for Hand Detection

Problem Statement
In occupational safety monitoring, robust hand detection is critical for ensuring compliance with safety protocols. However, public datasets for hand detection are predominantly composed of bare hands, failing to represent the visual variation introduced by personal protective equipment (PPE) such as gloves, bandages, tattoos, and jewelry. This creates a significant distribution shift between training data and real-world deployment scenarios, particularly in safety-critical contexts where detecting accessorized hands is essential. While generating synthetic data to fill this gap is a promising solution, existing methods often struggle with the "simulation-to-real" gap, and the utility of synthetic data for object-centric tasks remains an open scientific question.

Methodology
The authors propose a framework to evaluate whether generative inpainting—editing the hand region of real photographs to add accessories while preserving the background—can effectively close this distribution shift.

1. Dataset Construction:

   • Real Data: A dataset of 6,507 training, 2,085 validation, and 3,591 test RGB images with YOLO-format annotations.
   • Synthetic Data: Using the open-source tool SemProbe, the authors generated paired synthetic images. The pipeline segments the hand region (via GroundingDINO and SAM2) and uses a rectified-flow transformer (FLUX.2) to inpaint accessories (gloves, jewelry, tattoos) based on text prompts. The background remains pixel-identical to the original.
   • Quality Assessment: Five paired-image metrics were computed to validate the synthetic data:
     • Pixel/Structural: PSNR, SSIM, and LPIPS (computed on full images and hand crops).
     • Anatomical: MediaPipe Hands detection rate, confidence, and landmark deviation.
     • Boundary: Boundary gradient ratio to detect visible seams between the inpainted hand and the original background.

2. Experimental Design:
   The study trained YOLOv8n detectors under six regimes (Experiments A–F), each repeated with three random seeds for statistical analysis:

   • Exp A (Baseline): Train on real data, test on real data.
   • Exp B (Robustness Probe): Evaluate Exp A models on synthetic-gloves and real-gloves test sets (no retraining).
   • Exp C (Two-Stage Augmentation): Train on Real $\cup$ Synthetic, then fine-tune on Real-only with a lower learning rate.
   • Exp D (Synthetic-Only): Train exclusively on synthetic data.
   • Exp E (Three-Stage Curriculum): Train on Real $\to$ Real $\cup$ Synthetic $\to$ Real-Gloves, with progressively decreasing learning rates.
   • Exp F (Meta-Analysis): Aggregates results to calculate signed gap metrics.

Key Results

• Image Quality: Full-image metrics (PSNR/SSIM) suggested high similarity, but hand-region metrics revealed significant differences, confirming the inpainting successfully altered the target region. Boundary gradient analysis showed a mean ratio of 1.088, indicating mostly invisible seams with a minority of visible artifacts.
• Detection Performance (mAP@0.5):
  • Exp C (Two-Stage) achieved the highest mAP@0.5 on both the standard real test set (0.954) and the real-gloves out-of-distribution (OOD) set (0.921), outperforming the real-only baseline (0.867) by +8.7 points on the real set and closing the OOD gap by +18.1 points.
  • Exp E (Three-Stage) achieved the highest mAP@0.5:0.95 (0.628 on real, 0.374 on gloves), demonstrating superior box-tightness, though with a slight trade-off in mAP@0.5 compared to Exp C.
  • Exp D (Synthetic-Only) reached 71.5% of the baseline mAP@0.5, proving synthetic data alone is insufficient but viable as a supplement.
• Statistical Significance: Paired t-tests (n=3) and Cohen's d analysis indicated that Exp C's improvement over the baseline on mAP@0.5 was a large effect (d=2.06), though the p-value (0.071) did not strictly meet the 0.05 threshold due to the small sample size. Exp E significantly outperformed Exp C on mAP@0.5:0.95 (p=0.012, d=5.29).

Significance and Claims
The paper claims that the utility of synthetic data for safety-critical hand detection is determined by the training procedure rather than the data alone.

• Schedule Sensitivity: Simple multi-stage training schedules (specifically "augment-then-fine-tune" and "curriculum learning") extract substantial real-deployment benefits from inpainted accessory data.
• Gap Closure: These methods significantly reduce the distribution shift between bare-hand training data and accessorized-hand deployment scenarios.
• Limitations: Despite improvements, a "stubborn" gap remains in strict-IoU performance (mAP@0.5:0.95) for real-gloved hands. The authors attribute this to the difficulty of maintaining box tightness on accessorized hands, suggesting that future improvements may require refining the synthetic generation pipeline itself rather than just adjusting training schedules.

The study concludes that while generative inpainting introduces machine-detectable signatures, it is a viable strategy for data augmentation in occupational safety, provided it is integrated via carefully designed, multi-stage training regimes.