Technical Summary: Every9D-21M

Problem Statement

Estimating the 9D pose (3D rotation, 3D translation, and 3D object extent) of everyday objects from a single real-world image remains a significant challenge in computer vision. This difficulty stems primarily from a lack of large-scale, real-world supervision. Existing datasets suffer from a trade-off between category diversity and instance scale: they either rely heavily on synthetic renderings, which struggle to generalize to real-world noise and occlusions, or provide limited coverage of real-world objects. For instance, the largest existing real-world 9D pose dataset (Objectron) contains only 17,000 annotated objects across 9 categories, while the largest image-level 3D pose dataset (ImageNet3D) covers 200 categories but only 86,000 images. Furthermore, manual annotation of 9D poses is prohibitively expensive and often ambiguous from single views, hindering the development of robust foundation models for robotics, AR/VR, and 3D generative modeling.

Methodology

The authors propose Every9D-21M, a framework to generate 9D pose annotations for 21.8 million real-world images across 700 object categories. The core strategy leverages object-centric videos to overcome single-view ambiguities and scale annotation efforts. The pipeline consists of five iterative stages:

1. Clustering: Object-centric videos (sourced from uCO3D) are clustered using k-means on averaged DINOv3 features extracted from uniformly sampled frames. This category-agnostic approach groups visually similar instances, allowing for cross-category alignment and reducing reliance on manual category labels.
2. Reference Selection: For each cluster, a representative "medoid" video is selected as the reference instance. This choice, based on the video with the smallest average feature distance to others in the cluster, yields higher alignment accuracy than random selection.
3. Reference Annotation: A small set of reference objects (fewer than 0.01% of total images, approximately 1,000 objects) are manually annotated. Annotators use an interactive tool to align a 3D bounding box to a sparse 3D point cloud (reconstructed via VGGSfM) using multi-view masking (LangSAM + XMem). Crucially, the authors define cross-category orientation rules based on intrinsic properties, human-object interaction, and object-object interaction to ensure consistent canonical frames (e.g., defining the "front" of a chair as opposite the backrest).
4. Cross-Instance Alignment: The remaining videos in a cluster are aligned to the reference instance. This involves reconstructing a 3D surface (via alpha shapes) and projecting multi-view DINO features onto it. A rigid transformation (rotation, translation, scale) is estimated using RANSAC initialization followed by gradient-based refinement, minimizing a weighted combination of geometric (Chamfer distance) and appearance (feature space nearest-neighbor) distances.
5. Verification: All propagated poses are verified from multiple canonical viewpoints (front, top, right). Sequences are either filtered (removed due to poor geometry or multi-object leakage) or skipped (re-processed in subsequent iterations due to reference mismatches).

The resulting canonical poses are propagated to all frames within the object-centric videos, yielding dense 9D annotations.

Key Contributions

1. Scalable Real-World 9D Canonicalization: The authors introduce an iterative, reference-based framework that leverages object-centric videos for multi-view 3D reconstruction and cross-instance alignment. This approach propagates canonical frames from a tiny fraction of manually annotated references to millions of images, with every propagated pose verified from multiple viewpoints.
2. Every9D-21M Dataset: A large-scale dataset comprising 21.8 million images, 109,000 annotated objects, and 700 categories. It is two orders of magnitude larger than prior real-world 9D pose benchmarks in both image count and category diversity. The entire dataset was curated with approximately 199 hours of human effort (67 hours for annotation, 132 hours for verification).
3. Benchmark and Baselines: The paper establishes standardized, symmetry-aware evaluation protocols and provides baseline models trained on Every9D-21M. These baselines demonstrate that training on this dataset improves performance on existing benchmarks (PASCAL3D+, ImageNet3D) and yields substantially better generalization to the HANDAL dataset compared to models trained solely on ImageNet3D.

Results

Experiments validate the efficacy of the dataset and the canonicalization framework:

• Performance on Every9D-21M: Models trained on Every9D-21M significantly outperform state-of-the-art foundation models (OrientAnythingV1, OrientAnythingV2, WildDet3D) on the test split. Specifically, the proposed model improves coarse rotation accuracy (Acc@30°) by 16.6 to 17.7 percentage points over OrientAnythingV2 in both symmetry-unaware and symmetry-aware settings.
• Generalization: Training on Every9D-21M leads to stronger generalization to the HANDAL dataset (a real-world manipulable object dataset) compared to training on ImageNet3D, despite the domain gap.
• Complementarity: Joint training with Every9D-21M and ImageNet3D consistently improves performance on PASCAL3D+ and ImageNet3D, suggesting the datasets are complementary.
• Qualitative Improvements: Visual comparisons indicate improved robustness to unusual object orientations, more accurate estimation of subtle angular differences, and better resilience under challenging lighting conditions compared to prior methods.

Significance

The paper positions Every9D-21M as a critical resource for advancing 9D pose estimation and 3D understanding in unconstrained real-world environments. By providing a massive, real-world dataset with consistent canonical frames, the work addresses the "sim-to-real" gap that has limited the generalization of models trained on synthetic data. The authors argue that scalable canonicalized supervision is essential for:

• Robotics and AR/VR: Enabling spatial reasoning and physical interaction by inferring consistent object-centric coordinate frames.
• 3D Generative Modeling: Providing orientation-aligned training data to improve controllability and geometric consistency in text-to-3D and image-to-3D systems.
• Correspondence Learning: Ensuring stable alignment across instances for 3D morphable modeling.

The work demonstrates that high-quality 9D supervision can be achieved at scale with minimal human effort through the strategic use of object-centric videos and cross-instance alignment, setting a new standard for real-world 3D pose benchmarks.