Geometric Foundation Model Distillation for Efficient Lunar 3D Reconstruction

Technical Summary: Geometric Foundation Model Distillation for Efficient Lunar 3D Reconstruction

Problem Statement
Large-scale 3D foundation models (e.g., MASt3R, VGGT) have achieved state-of-the-art performance in multi-view stereo reconstruction by leveraging massive transformer backbones and pre-training. However, these models typically contain hundreds of millions to over a billion parameters, requiring substantial memory and computational resources that are unavailable in resource-constrained environments. This limitation is particularly critical in planetary exploration, such as lunar missions, where onboard computing units face strict constraints on memory, power, and latency. Furthermore, lunar environments present unique challenges—low-texture surfaces, extreme illumination, and limited parallax—that standard terrestrial models struggle to address without domain adaptation. The paper investigates whether the dense geometric capabilities of a large, domain-adapted foundation model can be effectively transferred to lightweight student networks suitable for onboard deployment.

Methodology
The authors propose a knowledge distillation framework that compresses a 688.6M-parameter MASt3R teacher model (fine-tuned on lunar imagery) into a family of lightweight student networks. The approach involves three core components:

1. Student Architecture Design:

   • Encoders: The study evaluates different encoder backbones, comparing Convolutional Neural Networks (MobileNetV3-Large) against Vision Transformers (ViT-Small and ViT-Tiny).
   • Decoders: The authors experiment with varying decoder widths and depths. They test a "reduced" decoder configuration (embedding dimension 512, depth 6) and a more aggressively compressed variant (256, 4), while keeping the prediction heads (pointmap, descriptors, confidence) identical to the teacher.
   • Training Strategy: Experiments explore freezing versus fine-tuning the encoder and the use of feature-level versus output-level supervision.

2. Structured SVD-Based Initialization:
   To address the dimensional mismatch between the teacher and student weights, the authors propose a structured initialization strategy using truncated Singular Value Decomposition (SVD).

   • Teacher weight matrices are decomposed via SVD.
   • The student's weights are initialized by retaining the top $r$ singular components (where $r$ is the student's dimension), effectively projecting the teacher's knowledge into the student's smaller latent space.
   • For multi-layer decoders, a uniform stride mapping is used to align student layers with corresponding teacher layers, preserving the hierarchical structure of cross-view correlations.
   • This "warm start" significantly improves convergence stability and final performance compared to random initialization.

3. Distillation Losses:

   • Output-Level Supervision: The student is trained to reproduce the teacher's dense geometric predictions (3D point maps, confidence scores, and local descriptors) using a geometry loss ($L_{geo}$) that combines confidence-aware 3D regression and contrastive descriptor matching. The teacher's predictions serve as pseudo-ground truth, eliminating the need for camera pose annotations.
   • Feature-Level Alignment: To guide internal reasoning, a feature-level loss aligns intermediate encoder representations. A learnable linear projection maps student features to the teacher's feature space, followed by a margin-filtered cosine similarity loss that focuses optimization on misaligned regions.

Key Results
Experiments were conducted on the StereoLunar dataset, a benchmark of lunar stereo pairs with ground-truth 3D information. The results demonstrate:

• Compression Efficiency: The best-performing student (S2), utilizing a ViT-Small encoder and a standard decoder, reduces the model size by 4.4× (154.9M parameters) while retaining high reconstruction quality. It achieves a Chamfer error only ~15% higher than the teacher.
• Decoder Redundancy: A student with a reduced decoder (S3, 94.0M parameters, 7.3× smaller than the teacher) remains competitive, suggesting that much of the decoder capacity in the teacher is redundant for this task.
• Encoder Importance: Replacing the transformer encoder with a lightweight CNN (MobileNet, S1) results in clear degradation in slope artifacts and pose estimation, despite having a similar parameter count to S2. This indicates that transformer-based self-attention is critical for cross-view geometric reasoning in this domain.
• Superiority over Sparse Supervision: Distillation using the teacher's pseudo-ground truth outperforms a baseline trained directly with sparse ground-truth annotations, reducing Chamfer error by approximately 18%.
• Pose Estimation: The distilled models achieve accurate camera pose estimation, with the best student (S2) reaching a median Virtual Correspondence Reprojection Error (VCRE) of 3.56% of the image diagonal.

Significance and Claims
The paper claims to provide practical guidelines for deploying 3D reconstruction models in resource-constrained environments, specifically targeting planetary exploration. Its primary contributions are:

1. A General Distillation Framework: Demonstrating that a 688M-parameter teacher can be compressed into compact networks without requiring pose annotations, using teacher predictions as dense supervision.
2. Architectural Insights: Quantifying the trade-offs between encoder types (ViT vs. CNN) and decoder capacity, establishing that preserving encoder capacity is more critical than maintaining a large decoder for lunar stereo reconstruction.
3. Initialization Innovation: Introducing a structured SVD-based initialization that bridges dimensional mismatches, improving optimization stability and final performance.
4. Training Strategy Ablation: Isolating the impact of feature-level distillation and SVD warm-starts, showing they are essential for high-fidelity compression.

The authors conclude that their "recipe"—distilling a domain-adapted 3D teacher into efficient students—offers a viable path for deploying advanced geometric reasoning on lunar landers and rovers, bridging the gap between high-capacity research models and practical, hardware-constrained applications.