Semantic Segmentation and Depth Estimation for Real-Time Lunar Surface Mapping Using 3D Gaussian Splatting

1. Problem Statement

Robotic exploration of the lunar surface faces significant challenges for autonomous navigation and mapping:

• Environmental Conditions: The lunar surface features poorly textured terrain (regolith), high-contrast lighting, and deep, hard-edged shadows without atmospheric scattering.
• Hardware Constraints: Rovers are limited to radiation-hardened, low-power hardware. They often rely on cameras rather than active sensors like LiDAR due to power and mass constraints.
• Limitations of Current Methods:
  • Classical SLAM: Geometric feature-based methods struggle in low-texture and high-contrast environments.
  • Learning-based SLAM: While more robust, they often rely on discrete map representations (point clouds, voxels) that are memory-intensive and lack fine surface detail.
  • Neural Radiance Fields (NeRF): While capable of high-fidelity reconstruction, NeRFs are computationally expensive for real-time rendering, use fixed volumes that are hard to extend incrementally, and lack interpretability for explicit semantic information.

The paper aims to develop a real-time, vision-based mapping framework that creates detailed, large-scale, semantic 3D maps of the lunar surface suitable for resource-constrained rovers.

2. Methodology

The proposed framework integrates dense perception models with 3D Gaussian Splatting (3DGS), a novel scene representation that uses explicit, optimizable 3D Gaussians for fast rendering and flexible updates. The pipeline consists of three main stages:

A. Perception Frontend

The system processes raw stereo camera data to generate dense geometric and semantic inputs:

• Depth Estimation: The authors benchmarked various models (traditional stereo like BM/SGM, and learning-based like RAFT-Stereo, CREStereo, and monocular models like Depth Anything V2).
  • Selection: RAFT-Stereo was selected for its superior balance of accuracy and speed (4.4 FPS) in generating dense depth maps, outperforming traditional methods in low-texture regions and monocular models in domain adaptation.
• Semantic Segmentation: Various architectures (U-Net, MANet, Segformer, etc.) were benchmarked to classify lunar terrain (regolith, rocks, sky, rovers, landers, humans).
  • Selection: MANet was chosen for its high performance in detecting rocks (critical for hazard avoidance) and regolith, achieving a high mean IoU.
• Preprocessing: Sky pixels are masked to prevent erroneous 3D point generation. For monocular depth (if used), a scaling strategy using triangulated sparse features (SuperPoint/SuperGlue) is applied to convert relative depth to metric depth.

B. Incremental Mapping Backend

• 3DGS Initialization: For each new frame, the estimated pose, dense depth, and semantic labels are fused to create a per-frame 3D semantic point cloud.
• Gaussian Generation: New Gaussians are added to the global map. Their size is proportional to the distance to nearest neighbors to ensure adaptive coverage.
• Voxel Filtering: To prevent unbounded memory growth during long traverses, a voxel-based filtering strategy is used. Only points in previously unobserved space initialize new Gaussians.
• Semantic Assignment: Each Gaussian is initialized with the semantic label of its corresponding point, creating a continuous semantic field.

C. Optimization Backend

An asynchronous optimization engine runs in the background to refine the map and ensure global consistency:

• Keyframe Buffer: Stores a history of frames to prevent catastrophic forgetting.
• Custom Loss Function: The standard 3DGS loss is extended with specific terms for lunar environments:
  • Supervised Depth Loss ($L_{depth}$): Enforces geometric accuracy using the input dense depth map on valid geometry masks.
  • Volumetric Regularization: Includes an "empty loss" ($L_{empty}$) to penalize density in sky/void regions and a "dense loss" ($L_{dense}$) to ensure surfaces are opaque. This addresses the ambiguity between deep shadows and the black sky.
  • Semantic Loss ($L_{semantic}$): Uses Negative Log Likelihood (NLL) to supervise the semantic field, ensuring the rendered semantic probabilities match the input segmentation masks.
• Modified Densification: The standard global opacity reset is removed to prevent the pruning of stable, older map regions. Splitting/cloning is omitted as the dense depth input already provides sufficient geometry.

3. Key Contributions

1. Real-Time 3DGS Framework: A novel pipeline for building semantic, large-scale lunar maps in real-time, leveraging the fast rendering and explicit nature of 3DGS.
2. Perception Benchmarking: A comprehensive evaluation of state-of-the-art depth estimation and semantic segmentation models specifically for the lunar domain, identifying RAFT-Stereo and MANet as the optimal choices.
3. Impact Analysis: A quantitative analysis demonstrating how perception inputs directly affect the geometric and semantic quality of the final 3D reconstruction.
4. Novel Loss Formulation: The introduction of specific loss components (empty space, dense surface, and semantic supervision) tailored to the high-contrast, shadow-heavy lunar environment.

4. Results

Experiments were conducted on synthetic datasets generated by the LuPNT simulator, featuring a 120-meter traverse.

• Perception Performance:
  • RAFT-Stereo achieved the best trade-off, with a Mean Absolute Error (MAE) of 0.05m and 4.4 FPS.
  • MANet achieved a mean IoU of 83.1% and 153.8 FPS, with superior rock detection capabilities.
• Mapping Accuracy:
  • The full pipeline achieved a geometric height accuracy of approximately 2.8 cm (Mean Height Error).
  • At a 5 cm threshold, the system achieved 64.3% Precision and 68.6% Recall.
  • Comparison: The 3DGS approach outperformed a traditional point cloud baseline (3.5 cm error) by 20%, demonstrating the benefit of continuous Gaussian representation in smoothing noise and approximating surface geometry.
• Semantic Fidelity:
  • Using ground truth semantic labels resulted in negligible improvements to the final geometry, confirming the high accuracy of the selected MANet model.
  • Reconstruction errors were highest for rocks (4.0 cm) and landers (3.6 cm), primarily due to the difficulty of distinguishing sharp, shadowed edges from the dark sky.
• Efficiency:
  • The 3DGS map (23.9 million Gaussians) required 2.4 GB of memory, significantly compressing the raw dataset (19.2 GB of images) while retaining rendering capabilities.
  • The system operates at 1 Hz for mapping updates, with optimization running asynchronously.

5. Significance and Future Work

• Significance: This work demonstrates that combining learned perception with 3DGS is a viable path for creating detailed, semantic lunar maps without LiDAR. It overcomes the limitations of NeRFs (speed, incremental updates) and classical methods (texture dependence). The ability to jointly optimize map and pose suggests a strong foundation for a full SLAM system.
• Future Work:
  • Full SLAM Integration: Integrating a visual-inertial odometry front-end to remove reliance on ground truth poses and enable joint map/pose optimization.
  • Surface Extraction: Developing density-based techniques to extract continuous surfaces from Gaussians for better evaluation.
  • Deployment: Optimizing models for flight hardware via quantization and investigating alternative representations (2DGS, Neural Point Clouds).
  • Robustness: Testing with pose noise, different camera configurations, and incorporating perception uncertainty.

In conclusion, the paper establishes a robust, real-time framework for lunar surface mapping that effectively bridges the gap between high-fidelity neural scene representations and the practical constraints of planetary exploration.