Technical Summary: DDMS – Discriminative Distillation of Multi-view Foundational Features into Single-view Models

1. Problem Statement

Recent visual foundation models (e.g., DINOv2) pretrained on large-scale single-image data have become powerful backbones for general-purpose vision tasks due to their strong semantic transferability. However, these features often exhibit limited 3D awareness when evaluated across multiple views. Specifically:

• Inconsistency: Features extracted from different viewpoints of the same scene may vary significantly.
• Lack of Separability: Visually similar but geometrically distinct regions may not be sufficiently separated in the feature space.
• Trade-offs in Existing Solutions:
  • Methods that enforce 3D consistency (e.g., Fit3D, SnD) often smooth out fine-grained distinctions, harming local discriminability required for image matching.
  • Methods that improve discriminability for matching (e.g., MEF) often degrade the semantic richness of the original foundation features, making them unsuitable as general-purpose dense features.

The core challenge is to adapt pretrained 2D foundation features into representations that are 3D-consistent and locally discriminative without sacrificing their semantic utility.

2. Methodology

The authors propose DDMS (Discriminative Distillation of Multi-view Foundational Features into Single-view Models), a two-stage framework that distills multi-view geometric knowledge into a single-view student encoder.

2.1 Multi-view-aware Teacher Construction

The first stage constructs a "teacher" model that fuses semantic and geometric information:

• Input: A set of $N$ input views.
• Components:
  • A pretrained 2D visual foundation model (e.g., DINOv2) providing semantic features ($F^v_m$).
  • A frozen multi-view geometry foundation model (e.g., Depth Anything 3) providing intermediate geometric features ($F^g_m$) that encode correspondence and epipolar cues.
• Fusion: The semantic and geometric features are concatenated on the same ViT patch grid. A lightweight projection module $\psi(\cdot)$ maps this concatenated feature back to the original foundational feature space and predicts a residual update:
  $$T_m = F^v_m + \psi([F^v_m; F^g_m])$$
  This keeps the teacher close to the original feature space while injecting cross-view geometric context.

2.2 Discriminative Feature Learning (Teacher Refinement)

The teacher is refined to ensure features are both 3D-consistent and locally distinctive:

• Geometry-Supervised Ranking: For a query pixel $x$, candidates are sampled from other views based on geometric validity (reprojection, depth consistency, surface normal agreement). Candidates are labeled as positive (3D proximity $< \tau_{pos}$) or negative ($> \tau_{neg}$).
  • A differentiable average-precision ranking loss ($L_{disc}$) is applied to encourage geometrically valid correspondences to rank higher than incompatible candidates in the feature similarity space.
• Feature-Space Anchoring: To prevent the refined features from drifting too far from the original semantic manifold, a semantic anchor loss ($L_{anchor}$) is applied. This penalizes large deviations from the frozen original 2D foundation features ($v_x$):
  $$L_{anchor} = \frac{1}{|Q|} \sum_{x \in Q} \max(0, d_{cos}(t_x, v_x) - \delta)$$
• Teacher Objective: $L_{teacher} = L_{disc} + \lambda_{anchor} L_{anchor}$.

2.3 Distillation into a Single-view Student

The final goal is a practical encoder that operates on a single image at inference time.

• Process: The refined multi-view teacher is frozen. A single-view student encoder is trained to mimic the teacher's feature maps using feature-level distillation ($L_{distill}$).
• Regularization: The student is also subjected to the same geometry-supervised ranking objective ($L_{disc}$) as a weak regularizer to preserve the discriminative structure learned by the teacher.
• Inference: Only the student encoder is used, requiring no multi-view inputs, camera poses, or geometry models during inference.

3. Key Contributions

1. Framework: Introduction of a 3D-aware geometric foundational feature distillation framework utilizing a multi-view teacher and a single-view student.
2. Training Strategy: A novel approach to train the multi-view teacher via geometry-supervised ranking (to ensure local separability) and feature-space anchoring (to preserve semantic structure).
3. Performance: Demonstration of consistent gains across correspondence, dense prediction transfer, and explicit 3D lifting evaluations, achieving a balanced profile of geometric consistency, discriminability, and semantic preservation.

4. Experimental Results

The method was evaluated against baselines including DINOv2 (frozen), Fit3D, MEF, and SnD across multiple dimensions:

• Direct Feature Quality:
  • Separation: DDMS achieved the largest margin between corresponding and non-corresponding points on ScanNet, indicating enhanced feature distinctiveness.
  • Geometric Correspondence: Substantial improvements in nearest-neighbor matching accuracy on ScanNet and Navi-Wild compared to all baselines.
  • Semantic Correspondence: Strong performance on SPair-71k and DAVIS, showing that geometric improvements did not come at the cost of semantic matching.
• Dense Prediction Transfer (Linear Probing):
  • Semantic Segmentation: DDMS matched or slightly outperformed SnD on ScanNet and ScanNet++, while significantly outperforming Fit3D and MEF.
  • Depth Estimation: The method produced coherent depth structures, outperforming SnD on NYUv2 and ScanNet.
• 3D Lifting and Rendering:
  • 3D Point Segmentation: After aggregating features onto 3D point clouds, DDMS maintained high segmentation accuracy.
  • Feature Splatting: In neural rendering scenarios (3D Gaussian Splatting), DDMS features remained coherent and consistent between 2D inputs and rendered views, whereas other methods showed discrepancies due to smoothing or mismatch.
• Ablation Studies: Confirmed the necessity of both the ranking objective (for discriminability) and the anchor loss (for semantic preservation), as well as the importance of the multi-view teacher conditioning.

5. Significance and Claims

The paper claims that DDMS successfully bridges the gap between semantic richness and 3D geometric awareness.

• Balanced Representation: Unlike prior works that optimize for either consistency (sacrificing discriminability) or matching (sacrificing semantics), DDMS produces features that are simultaneously 3D-consistent, locally distinctive, and semantically transferable.
• Practical Utility: By distilling multi-view knowledge into a single-view student, the method retains the efficiency and ease of use of standard 2D backbones while inheriting the geometric robustness of multi-view models.
• Generalizability: The approach is shown to work across different backbone scales (ViT-S, ViT-B, ViT-L) and foundation models (DINOv2, DINOv3), suggesting it is a general strategy for enhancing visual foundation features.

The authors emphasize that their method does not require multi-view inputs at inference time, making it suitable for real-world applications where only single images are available, yet 3D consistency is required for tasks like visual localization, SLAM, and scene understanding.