Emergence of a Shared Canonical Object Frame from In-the-Wild Videos

Technical Summary: Emergence of a Shared Canonical Object Frame from In-the-Wild Videos

Problem Statement
Comparing object orientations and positions across different instances requires expressing their poses within a shared canonical frame. Traditionally, establishing such frames has relied on substantial manual supervision, including aligned CAD models, controlled synthetic rendering, or careful real-world pose annotation. This reliance creates a scaling bottleneck: synthetic data introduces a domain gap, while annotated real-world data remains limited in category and instance diversity. Consequently, learning canonical object representations that generalize broadly across real-world data without manual canonical pose labels remains an open challenge. Furthermore, existing self-supervised approaches often train separate models per category, failing to produce a shared frame across different object types.

Methodology
The authors propose a self-supervised, category-agnostic approach that learns a shared canonical frame from object-centric videos captured in the wild, utilizing only noisy camera poses recovered via Structure-from-Motion (SfM). The core mechanism involves routing all training sequences through a shared geometric bottleneck.

1. Shared Canonical Mesh: The method defines a fixed, coarse 3D mesh (e.g., a unit cube or sphere) in a canonical coordinate frame. This mesh carries no category-specific geometric detail, forcing the network to rely on appearance and semantic priors rather than shape matching.
2. Correspondence Network: A neural network learns to map image pixels to vertices of this shared mesh. It employs a feature extractor and a Transformer decoder, where learnable vertex embeddings act as queries against pixel features. The output is a dense distribution over mesh vertices for each pixel, effectively predicting 2D-3D correspondences.
3. Per-Sequence Alignment: Since SfM reconstructions exist in arbitrary coordinate frames, the method estimates a per-sequence rotation to align the SfM frame with the canonical mesh frame.
   • Initialization: Alignment is initialized via Principal Component Analysis (PCA) on the masked SfM point cloud to capture coarse geometric extent (e.g., elongation).
   • Refinement: During training, the network estimates a rotation that minimizes the distance between the network's predicted canonical coordinates and the SfM-derived coordinates (transformed by the estimated rotation). This is formulated as Wahba's problem, solved via SVD, with weights based on the entropy of the correspondence distribution to prioritize confident predictions.
4. Training Objective: The aligned SfM poses are used to rasterize the canonical mesh, generating pseudo-labels (visible mesh vertices) for the correspondence network. The network is trained using a cross-entropy loss to match these pseudo-labels, alongside a mask prediction head supervised by the visibility mask. The alignment and the correspondence network co-evolve during training.
5. Inference: At test time, the model processes a single RGB image to predict dense correspondences and a segmentation mask. The 6D pose is recovered via EPnP with RANSAC using the predicted correspondences, without requiring per-sequence alignment steps.

Key Contributions

• Self-Supervised Canonical Frame Emergence: The paper demonstrates that a shared canonical frame can emerge from object-centric videos using only noisy SfM camera poses, eliminating the need for canonical pose annotations.
• Geometric Bottleneck: The authors show that routing diverse training sequences through a shared, coarse canonical mesh induces cross-sequence consistency. This allows a common reference frame to emerge through multi-view alignment and the semantic priors of the feature extractor.
• Category-Agnostic Generalization: The method trains a single model across all categories without category conditioning. It achieves competitive performance with canonically supervised methods on multiple benchmarks and generalizes to object categories unseen during training.

Experimental Results
The method was trained on approximately 160,000 in-the-wild object-centric videos from the UCO3D dataset. Evaluation was conducted on five benchmarks: REAL275, Omni6DPose, Objectron, Pascal3D+, and ImageNet3D.

• Performance: Despite lacking canonical pose labels and training solely on real video, the method achieved the best macro-average performance across all five datasets compared to supervised baselines (e.g., OrientAnything V1/V2, QWEN3-VL) and category-specific self-supervised methods.
• Data Scale: Ablation studies confirmed that scaling the training data (using all available sequences rather than a capped subset) significantly improves performance, highlighting the importance of data volume for learning generalizable representations.
• Ablations: The study showed that while PCA initialization aids convergence, the learned alignment is critical for resolving semantic orientation (e.g., front-vs-back). A cube mesh provided a slight inductive bias over a sphere, but the geometric bottleneck principle remained robust across shapes. Multi-view coverage (K=4 views) was found essential for reliable alignment.

Significance and Limitations
The paper claims that this approach addresses the scaling bottleneck of canonical supervision, enabling category-agnostic pose estimation that transfers to unseen categories without test-time onboarding or manual labels. The learned frame is shown to be largely consistent across diverse object types, with per-category convention mappings clustering into a few modes.

However, the authors modestly note limitations regarding symmetry. The emerged frame is most consistent for objects with distinctive semantic axes. For continuously symmetric objects, where multiple orientations yield indistinguishable evidence, the method struggles because pseudo-labels become noisy. The paper suggests that resolving such ambiguities without canonical labels or symmetry priors remains a challenge for future work.