Technical Summary: Geometry Matters: 3D Foundation Priors for Learning Semantic Correspondence

Problem Statement

Semantic correspondence aims to match semantically equivalent object parts across different images, a task critical for vision and robotics. While recent foundation features from self-supervised vision transformers (e.g., DINOv2) and text-to-image diffusion models (e.g., Stable Diffusion) have established strong zero-shot baselines, they suffer from inherent limitations. Because these features are learned primarily from 2D image objectives, they lack explicit 3D awareness. This leads to systematic failure modes, particularly in confusing symmetric object sides (e.g., left vs. right wheels), repeated parts (e.g., multiple legs or windows), and visually similar but geometrically distinct structures.

Existing methods attempting to address these ambiguities often inject weak 3D priors, such as mapping features to a coarse spherical proxy. However, these approaches typically require manual human pose annotations and fail to capture the fine-grained geometric structure of specific object instances, limiting their scalability and effectiveness on symmetric or articulated objects.

Methodology

The authors propose a 3D-aware post-training framework that leverages priors from 3D foundation models to generate and filter high-quality pseudo-labels without requiring manual pose annotations. The pipeline consists of three main stages:

1. Canonicalized 3D Object Reconstruction

The method begins by reconstructing an object-centric 3D mesh for each instance from a single image.

• Initialization: Using SAM3 for 2D instance masking and SAM3D for single-image 3D reconstruction, an initial mesh and camera parameters are obtained.
• Pose Refinement: To correct scale and translation errors, a render-and-compare optimization is performed. This involves a two-phase process:
  1. Distance-Transform (DT) Phase: Optimizes alignment using a distance field loss to handle cases where the rendered silhouette and observed mask are disjoint.
  2. Soft-IoU Phase: Refines the alignment once overlap is established using a differentiable soft-IoU loss.
• Yaw Canonicalization: To resolve the four-fold yaw ambiguity common in symmetric objects (e.g., buses, trains), the method renders the mesh at eight known orientations and uses OrientAnything V2 to estimate the apparent orientation. A majority vote selects the correct discrete yaw correction, ensuring consistent canonical orientation across instances.

2. Pseudo-Label Generation and Filtering

Once the 3D geometry is reconstructed and canonicalized, the system generates semantic correspondences:

• Feature Fusion: The pipeline fuses 2D foundation features (DINOv2 + Stable Diffusion) with PartField descriptors rasterized from the reconstructed 3D mesh. PartField features are crucial as they encode geometric and part-level structure, distinguishing visually similar but geometrically distinct regions (e.g., front vs. rear wheels).
• Candidate Generation: Candidate matches are proposed via nearest-neighbor search in the fused feature space, followed by a relaxed cyclic consistency check to discard obvious mismatches.
• Geodesic Filtering: This is the core geometric verification step. Matched pixels are lifted onto their respective 3D meshes. The system computes a bicyclic geodesic error by comparing the image-space match against a cross-mesh correspondence predicted by PartField nearest-neighbor search. Candidates exceeding a geodesic distance threshold are rejected. This filtering is more reliable than spherical proxies because it operates on the actual instance-specific geometry.

3. Supervised Training

The filtered, high-quality pseudo-labels are used to train a lightweight adapter on top of frozen DINOv2 and Stable Diffusion features. The adapter is supervised using a combination of a sparse contrastive loss (on labeled pairs) and a dense regression loss (propagating gradients to unlabeled regions).

Key Contributions

The paper outlines four primary contributions:

1. 3D-Aware Post-Training Framework: A method for semantic correspondence that incorporates priors from 3D foundation models (SAM3D, PartField) without requiring human pose annotations.
2. Render-and-Compare Pose Refinement: A technique to refine object pose and canonicalize orientation, enabling the rendering of PartField features into the image plane to complement 2D features.
3. Geodesic Pseudo-Label Filtering: A filtering scheme based on geodesic distances on reconstructed 3D shapes, providing higher-quality supervision than coarse spherical geometry proxies.
4. State-of-the-Art Performance: The resulting geometry-aware refined features achieve state-of-the-art results on standard benchmarks with reduced manual supervision.

Experimental Results

The method was evaluated on four standard benchmarks: SPair-71k, SPair-Geo-Aware (a subset focusing on symmetric/repeated parts), AP-10K (animal pose), and SpairU.

• SPair-71k: The proposed method (3D-SC) achieved 73.0 PCK@0.1, outperforming the strongest weakly supervised baseline without human annotations (DIY-SC+OriAny) by 3.4 points.
• SPair-Geo-Aware: The gains were most pronounced here, reaching 70.8 PCK@0.1, significantly surpassing existing weakly supervised approaches. This confirms the hypothesis that instance-specific 3D structure is particularly effective for disambiguating symmetric and repeated parts.
• AP-10K: The method generalized well to articulated and shape-diverse animal categories, outperforming baselines without human annotations across intra-species, cross-species, and cross-family splits.
• Ablation Studies: Experiments confirmed that adding PartField features reduces the False Positive Rate of candidates and increases the number of retained correct matches. Geodesic filtering was shown to be the most effective filtering strategy, reducing the error rate of unfiltered predictions significantly.

Significance and Claims

The paper claims that instance-specific 3D structure serves as a more reliable geometric prior than the coarse spherical proxies used in previous post-training approaches. By automatically obtaining this structure from off-the-shelf 3D foundation models, the method eliminates the need for manual pose annotations while improving correspondence accuracy, particularly for rigid objects with symmetry.

The authors position this work as an early step toward a new class of self-supervised pipelines where 3D foundation models act as geometric teachers for 2D tasks. They note that as 3D reconstruction quality continues to improve, this direction will become increasingly powerful. The paper modestly acknowledges limitations, such as the dependency on SAM3D's pose estimates and the fact that PartField features are trained on rigid objects, which may limit performance on highly deformable categories or fine-grained keypoint localization within parts.