HaDR: Applying Domain Randomization for Generating Synthetic Multimodal Dataset for Hand Instance Segmentation in Cluttered Industrial Environments

This paper presents HaDR, a framework that utilizes domain randomization to generate a synthetic RGB-D dataset and train multimodal instance segmentation models, demonstrating that such models trained exclusively on synthetic data outperform those trained on real-world datasets for robust hand localization in cluttered industrial environments.

The Gist

Imagine you are trying to teach a robot to "see" and grab objects with its hands in a messy factory. The problem is, factories are chaotic. There are tools everywhere, lights flicker, and workers wear gloves of all different colors (red, green, yellow, white). If you just show a robot a few photos of a hand in a red glove, it might get confused when it sees a hand in a blue glove or a dirty background.

This paper introduces a clever solution called HaDR (Hand Domain Randomization). Here is the story of how they did it, explained simply.

1. The Problem: The "Reality Gap"

Usually, to teach a robot, you need thousands of real photos of hands, and a human has to draw a line around every single hand in every photo. This is slow, expensive, and boring.

If you try to use a computer simulation (like a video game) to make these photos, the robot gets confused. Why? Because the simulation looks too perfect. It's like teaching a child to drive on a perfectly smooth, empty track, and then dropping them into rush-hour traffic. The child (or robot) panics because the real world is messy. This difference between the fake world and the real world is called the "Reality Gap."

2. The Solution: The "Chaos Chef" (Domain Randomization)

Instead of trying to make the simulation look realistic (which is hard and expensive), the authors decided to make it wildly unrealistic.

Think of their simulation as a Chaos Chef.

• The Ingredients: They put hands, tools, and backgrounds into a 3D kitchen.
• The Randomization: Instead of cooking a perfect meal, the Chef throws everything in randomly.
  • Sometimes the background is neon pink; sometimes it's a forest.
  • The lighting changes from blinding sun to pitch black.
  • The hands wear gloves that look like they are made of plastic, metal, or fur.
  • Random geometric shapes (cubes, spheres) float around to distract the robot.

The Magic Trick: By forcing the robot to look at these crazy, fake, and varied images, the robot stops paying attention to "tricks" like the color of the skin or the specific lighting. Instead, it learns the essential shape of a hand. It learns, "Ah, a hand is a hand, no matter if it's glowing green or covered in a red glove."

3. The Secret Weapon: Seeing with Two Eyes (Multimodal)

The researchers realized that just looking at the "color" (RGB) isn't enough. Sometimes a red glove blends in with a red background, and the robot gets lost.

So, they gave the robot a second pair of eyes: Depth.

• RGB (Color): Tells the robot what things look like.
• Depth: Tells the robot how far away things are (like a 3D map).

It's like wearing 3D glasses. Even if the colors are confusing, the robot can see that the hand is "sticking out" from the background. They tested this by feeding the robot data with just color, just depth, and both together. The combination (RGB-D) was the winner, acting like a superpower that helped the robot see through the clutter.

4. The Results: Beating the Experts

They trained their robot using only these 117,000 crazy, fake images. Then, they tested it on a real, messy industrial environment with real workers wearing real gloves.

The Scoreboard:

• The Old Way (Real Data): They compared their robot to models trained on famous real-world datasets. The old models failed miserably when the gloves changed color or the background got messy.
• The "Google" Way (MediaPipe): They compared their robot to MediaPipe, a very popular, high-tech hand-tracking tool used by millions.
  • The Result: Their robot beat MediaPipe.
  • Why? MediaPipe relies heavily on the color of human skin. If you wear a glove that looks like skin (or a weird color), MediaPipe gets confused. The HaDR robot didn't care about the color; it cared about the shape and the 3D position.

5. The Takeaway

This paper proves that you don't need to spend years taking thousands of photos of real hands to teach a robot. Instead, you can build a "chaos simulator," throw random nonsense at the robot, and teach it to focus on the shape rather than the details.

In a nutshell:
If you want a robot to be good at a messy job, don't train it on a clean, perfect day. Train it in a storm, with weird lights, and confusing colors. That way, when it faces the real world, nothing will surprise it.

Technical Summary

1. Problem Statement

The paper addresses the challenge of hand instance segmentation in unstructured, cluttered industrial environments. Key issues identified include:

• The "Reality Gap": Deep learning models trained on real-world data often fail to generalize when environmental conditions change (e.g., lighting, background clutter).
• Color and Texture Bias: Existing solutions (like MediaPipe) often rely heavily on skin tone and texture, leading to failures when workers wear gloves of various colors or when the background blends with the hand.
• Data Scarcity and Cost: Annotating pixel-level instance segmentation masks for real-world data is labor-intensive, expensive, and prone to human error.
• Limitations of Existing Datasets: Publicly available hand datasets often suffer from biases, such as hands appearing only in the center of the image, lacking distractor objects, or assuming the hand is the closest object to the camera.

2. Methodology

The authors propose a Domain Randomization (DR) approach to generate a fully synthetic, multimodal (RGB-D) dataset, training models to be "color-agnostic" and robust to environmental variations.

A. Synthetic Dataset Generation (HaDR)

• Platform: Built using CoppeliaSim, a 3D simulation environment.
• Domain Randomization Strategy: Instead of striving for photorealism, the system introduces high variability to force the network to learn essential shape features rather than specific textures or colors.
  • Randomization Parameters: Randomized hand poses, gestures, positions, and orientations; random textures, colors, and scales for distractor objects (tools, geometric shapes); randomized lighting (number, position, type); and random background clutter.
  • Multimodal Input: The system generates both RGB (color) and Depth images. Depth values are scaled to an 8-bit grayscale range [0.2, 1.0] meters.
  • Scene Constraints: The simulation ensures hands remain within the camera's field of view (truncated pyramid) and prevents distractors from overlapping the hand in the camera view.
  • Dataset Scale: Generated 117,000 images with pixel-level ground truth masks in COCO format. The dataset includes up to two hand instances per image to simulate a single user scenario.

B. Model Training

• Architectures: Two state-of-the-art instance segmentation models were trained: Mask R-CNN and SOLOv2.
• Backbones: ResNet-50 and ResNet-101.
• Training Strategy:
  • Models were trained solely on the synthetic dataset without fine-tuning on real data.
  • Data augmentation included horizontal/vertical flipping to handle both left and right hands (since the simulation used only right-hand meshes).
  • Pre-trained weights from MS COCO and ImageNet were used to initialize the models.

C. Evaluation Metrics

• Average Precision (AP): Standard COCO metrics (IoU at 0.5:0.95).
• Probability-based Detection Quality (PDQ): A metric that explicitly penalizes both false positives and false negatives, providing a more practical measure for robotics applications than AP alone.
• Qualitative Analysis: Visual inspection of mask quality, specifically focusing on performance with different glove colors (red, green, white, yellow) and background blending.

3. Key Contributions

1. HaDR Dataset: The creation of the first color-agnostic RGB-D synthetic dataset specifically designed for hand instance segmentation in industrial settings. It addresses the bias of existing datasets by including random distractors, varied hand positions (not just center), and variable depth.
2. Multimodal Analysis: A rigorous investigation into the synergy of RGB vs. Depth vs. RGB-D inputs. The study hypothesizes and proves that combining modalities improves robustness in cluttered environments.
3. Performance Benchmarking: Demonstrating that models trained only on synthetic DR data outperform models trained on state-of-the-art real-world datasets (EgoHands, HandSeg) and synthetic datasets (RHD, ObMan, DenseHands) when tested on real industrial images.
4. Superiority over MediaPipe: Showing that the proposed DR-trained models significantly outperform the industry-standard MediaPipe Hands solution in cluttered industrial scenarios, particularly regarding glove color independence.

4. Results

• Synthetic vs. Real Training: Models trained on the HaDR dataset achieved an AP of 52.5 (Mask R-CNN ResNet50 RGB) on real test data. In contrast, models trained on COCO-pretrained data or other existing datasets (even real ones like EgoHands) achieved significantly lower AP scores (e.g., EgoHands-trained Mask R-CNN achieved ~25.3 AP).
• Modality Performance:
  • RGB-D (Best): The Mask R-CNN ResNet50 (RGB-D) model achieved the highest PDQ (0.1576) and maintained high AP (45.5).
  • RGB: Performed well in AP (52.3) but struggled with false positives in qualitative tests when gloves blended with the background.
  • Depth: Performed worse than RGB in AP but showed better robustness in specific clutter scenarios compared to RGB alone.
• Comparison with MediaPipe:
  • MediaPipe: Achieved a PDQ of 0.0836 and AP of 18.1. It frequently failed to detect hands wearing green gloves or when the hand color matched the background.
  • HaDR Model: Achieved a PDQ of 0.1576 and AP of 45.5, demonstrating stable detection regardless of glove color or lighting.
• Qualitative Findings: The RGB-D models successfully identified 9/10 instances in difficult test cases (e.g., white gloves, background blending), whereas RGB-only models had higher false positive rates.

5. Significance and Conclusion

• Industrial Applicability: The study proves that high-quality, pixel-accurate segmentation models for human-robot interaction (HRI) can be built without collecting massive amounts of real-world annotated data. This is crucial for industries where data collection is difficult or where workers wear varying PPE (Personal Protective Equipment).
• Generalization: The Domain Randomization technique successfully forces the neural network to learn shape-based features rather than relying on color or texture, solving the "color bias" problem common in current hand-tracking solutions.
• Future Directions: The authors suggest expanding the variety of hand meshes and simulating interactions with tools to further bridge the reality gap.

In summary, HADR demonstrates that a carefully designed synthetic dataset using domain randomization can produce models that are more robust and accurate for industrial hand segmentation than those trained on existing real-world or photorealistic synthetic datasets.