Splat and Distill: Augmenting Teachers with Feed-Forward 3D Reconstruction For 3D-Aware Distillation

Technical Summary: Splat and Distill (SnD)

Published at: ICLR 2026
Authors: David Shavin & Sagie Benaim (Hebrew University of Jerusalem)

1. Problem Statement

Vision Foundation Models (VFMs), such as DINOv2, have achieved state-of-the-art performance in 2D semantic tasks. However, they suffer from a critical lack of 3D awareness. They struggle with tasks requiring geometric understanding, such as monocular depth estimation, surface normal reconstruction, and multi-view feature correspondence.

Existing attempts to bridge this gap face two main hurdles:

• Optimization-based approaches (e.g., FiT3D): These require slow, per-scene optimization to lift 2D features into 3D, which is computationally expensive and leads to "semantic blurring" (averaging inconsistent features from different views).
• Relational approaches (e.g., MEF): These enforce feature similarity at corresponding points but lack the dense geometric supervision necessary for complex 3D tasks.

2. Methodology

The authors propose Splat and Distill (SnD), a framework that instills 3D awareness into 2D VFMs by augmenting a teacher model with a fast, feed-forward 3D reconstruction pipeline within a student-teacher distillation paradigm.

Key components of the pipeline:

• Student-Teacher Framework: The student and teacher share the same architecture. The teacher's weights are updated via an Exponential Moving Average (EMA) of the student's weights, ensuring a dynamic and evolving supervisory signal.
• Feed-Forward 3D Reconstruction: Instead of per-scene optimization, the framework uses a pre-trained, frozen model (MVSplat) to instantly predict 3D Gaussian primitives (geometry and appearance) from a few context views.
• Mask-Aware Feature Lifting: To solve the resolution mismatch between low-resolution patch features and high-resolution images, the authors introduce a mask-aware upscaling mechanism. It uses semantic masks to guide bilinear interpolation, ensuring that feature upscaling respects object boundaries and prevents "bleeding" across edges.
• 3D Feature Splatting: 2D features from the teacher are attached to the 3D Gaussians. These "feature-augmented" Gaussians are then "splatted" (rendered) from novel target viewpoints to create high-quality, geometrically grounded 2D feature maps.
• Semantic Blending: To mitigate artifacts caused by imperfect 3D reconstruction, the rendered features undergo a blending step. This enforces local consistency by averaging features within the same semantic mask, smoothing out geometric noise while preserving sharp edges.
• Distillation Loss: The student is trained to minimize the discrepancy between its 2D features and the teacher's rendered, blended, and downscaled 3D-aware features using a DINO-style cross-entropy loss.

3. Key Contributions

1. Architectural Innovation: Introducing a 3D reconstruction pipeline as a "teacher augmenter" to provide dense geometric supervision.
2. Efficiency: Replacing slow per-scene optimization with a feed-forward lifting mechanism, making the process scalable and significantly faster.
3. Boundary Preservation: Developing mask-aware upscaling and semantic blending to ensure that 3D-to-2D feature projection maintains high semantic fidelity and sharp object boundaries.
4. Dynamic Learning: Using an EMA-based teacher update to avoid the "least-squares compromise" (semantic blurring) found in static optimization methods.

4. Results

The framework was evaluated on a wide suite of tasks using ViT-Small and ViT-Base backbones:

• Monocular Depth Estimation: Significant improvements in RMSE and Relative error across ScanNet++, ScanNet, and NYUv2 datasets, outperforming Fit3D and MEF.
• Surface Normal Estimation: Achieved lower RMSE on NYUv2, producing smoother and more geometrically accurate normal maps.
• Multi-view Correspondence: Demonstrated higher recall at various viewpoint changes (0° to 180°), proving superior multi-view consistency.
• Semantic Segmentation: Improved mIoU and accuracy on both in-domain (ScanNet) and out-of-domain (ADE20k, Pascal VOC) datasets, showing that 3D awareness actually enhances 2D semantic richness.
• Generalization: Showed strong transferability to outdoor datasets (KITTI) despite being trained on indoor data.

5. Significance

"Splat and Distill" represents a shift from distilling 2D features into 3D to distilling 3D knowledge into 2D. By using 3D Gaussians as a bridge, the method provides a scalable way to create "3D-aware" foundation models. This has profound implications for robotics, augmented reality, and autonomous driving, where models must understand both the semantic identity of an object and its precise geometric structure in 3D space.