IndustryShapes: An RGB-D Benchmark dataset for 6D object pose estimation of industrial assembly components and tools

Technical Summary: IndustryShapes

Problem Statement
Robotic 6D object pose estimation is fundamental for industrial automation, particularly in flexible manufacturing and collaborative environments. However, existing benchmarks often rely on controlled laboratory settings, household objects, or specific bin-picking scenarios that fail to capture the complexities of real-world industrial assembly. Industrial tools and components frequently exhibit challenging properties such as weak or absent textures, metallic and reflective surfaces, symmetries, and thin structures. Furthermore, current datasets often lack support for "novel object" pose estimation (generalizing to unseen objects) and do not provide the specific data modalities, such as static onboarding sequences, required by modern model-free approaches. This gap between lab-based research and real-world deployment leads to performance discrepancies when methods are applied in actual manufacturing scenarios.

Methodology and Dataset Composition
The authors introduce IndustryShapes, a new RGB-D benchmark dataset designed to bridge this gap. The dataset focuses on five industrial tools and components selected for their challenging geometric and surface properties. The data is organized into two complementary sets:

1. Classic Set: Designed for instance-level pose estimation. It contains 4,623 images (3,700 training, 923 test) and approximately 6,000 annotated poses. The training data includes both laboratory turn-table captures and real industrial scenes, while the test set features complex, cluttered industrial environments with occlusions and multiple object instances.
2. Extended Set: Tailored for novel object pose estimation (both model-based and model-free). This set includes:
   • RGB-D Static Onboarding Sequences: To the authors' knowledge, the first dataset to provide these. It consists of 10 sequences (2 per object) captured with a handheld camera to support the "onboarding" phase of model-free pipelines.
   • Extended Test Scenes: Three scenes containing all five objects in an office environment with unconstrained lighting and distractors.

The dataset comprises a total of over 22,400 annotations across 34 scenes. Data was captured using Intel RealSense RGB-D cameras (D455 and D405) at 640x480 resolution. Ground truth 6D poses were annotated using a combination of marker-based methods (ArUco) and a semi-automatic pipeline utilizing Structure-from-Motion (SfM) and CAD model anchor points. The annotation accuracy was validated by comparing rendered depth against captured depth, showing absolute errors of less than 12 mm for the classic set and approximately 5 mm for the extended set.

Key Contributions

• Realistic Industrial Benchmark: Provides a testbed with five challenging object types (reflective, symmetric, textureless) captured in realistic industrial assembly settings, moving beyond clean tabletop or household scenarios.
• Dual-Set Architecture: Uniquely supports both instance-level methods (via the Classic set) and novel object methods (via the Extended set).
• Novel Data Modalities: Introduces the first RGB-D static onboarding sequences, explicitly enabling the evaluation of model-free pose estimation pipelines.
• Comprehensive Benchmarking: The dataset includes rich annotations (poses, segmentation masks) formatted according to BOP specifications, facilitating the evaluation of detection, segmentation, and pose estimation.

Results and Evaluation
The authors evaluated the dataset using a representative set of state-of-the-art methods:

• Instance-Level: EPOS, DOPE, and ZebraPose.
• Novel Object: FoundPose (model-based) and FoundationPose (model-based and model-free).
• Detection/Segmentation: CNOS and SAM-6D.

Findings:

• Performance Gap: The challenging nature of the dataset resulted in generally low performance across most methods, highlighting the difficulty of industrial pose estimation.
• Novel vs. Instance: Recent novel object approaches achieved performance comparable to instance-level methods, particularly FoundationPose (Model-Based), which achieved the highest Average Recall (AR) of 0.67 on the Classic set and 0.69 on the Extended set.
• Object Variability: Performance varied significantly based on object characteristics. For example, Object 5 (reflective, thin) yielded significantly lower accuracy than Object 1 across all methods.
• Detection and Segmentation: SAM-6D demonstrated superior segmentation performance on the Classic set compared to CNOS, though both showed moderate detection capabilities.

Significance and Claims
The paper claims that IndustryShapes addresses a critical gap in the field by providing a realistic, application-relevant testbed that reflects the complexity of industrial robotics. By including onboarding sequences and supporting both instance-level and novel object paradigms, the dataset facilitates the development of methods that can generalize beyond controlled environments. The evaluation results demonstrate that while state-of-the-art methods show promise, there is significant room for improvement in handling the specific challenges of industrial components, reinforcing the dataset's role as a robust benchmark for future research. The authors note limitations regarding uneven frame distribution in some test scenes and varying training complexity, suggesting future work should focus on diversifying training data and increasing frame counts in underrepresented scenarios.