OmniUnet: A Multimodal Network for Unstructured Terrain Segmentation on Planetary Rovers Using RGB, Depth, and Thermal Imagery

1. Problem Statement

Robot navigation in unstructured environments, particularly for planetary exploration (e.g., Mars), faces significant challenges due to steep slopes, uneven surfaces, and loose or rocky terrain. Traditional navigation systems relying on single-modality sensors (like RGB cameras) often fail under adverse conditions such as low lighting, glare, or when visual texture is insufficient to distinguish terrain types.

• The Gap: While multimodal sensor fusion (combining RGB, Depth, and Thermal) offers complementary data to overcome individual sensor limitations, there is a lack of:
  1. Specialized machine learning architectures designed to efficiently integrate heterogeneous data (RGB-D-T) for semantic segmentation.
  2. Publicly available, labeled multimodal datasets specifically for unstructured planetary-like terrains.
  3. Models optimized for deployment on resource-constrained embedded hardware used in rovers.

2. Methodology

A. Architecture: OmniUnet

The authors propose OmniUnet, a novel transformer-based neural network designed for semantic segmentation using RGB, Depth, and Thermal (RGB-D-T) inputs.

• Backbone: Utilizes the Omnivore architecture (based on Swin Transformers) for feature extraction. It employs a "shifted window" attention mechanism to capture both local context and global relationships across different modalities.
• Decoder: Adapts the U-Net decoding strategy. Feature maps from the encoder are upsampled and fused with outputs from higher stages via convolutional layers and GELU activations.
• Input Handling: Accepts a 5-channel input (RGB + Depth + Thermal) via a tokenizer layer that converts images into patch embeddings. This unified approach eliminates the need for separate models for different sensor configurations.
• Training Strategy:
  • Loss Function: A composite loss combining Cross-Entropy and Dice Loss (Sørensen–Dice coefficient) to balance precision and recall, particularly important for handling class imbalances in terrain segmentation.
  • Hardware: Training performed on an NVIDIA DGX Station (Tesla V100); Inference tested on embedded platforms (RTX 2070 and Jetson Orin Nano).

B. Data Collection and Dataset

• Sensor Setup: A custom 3D-printed sensor housing was developed to mount a Realsense D435i (RGB-D) and an Optris PI-640i (Thermal) camera on the MaRTA rover (a half-scale ExoMars testbed).
• Location: Data was collected in the Bardenas semi-desert (Northern Spain), a Mars-analog environment featuring sand, bedrock, and compact soil.
• Dataset (BASEPROD):
  • Recorded 36,000 multimodal images across 24 rover traverses (~1.7 km).
  • 8 Terrain Classes: Void, compact, bedrock, sandy, gravel, rock, bush, and grass.
  • Annotation: 950 representative samples were manually labeled for training, and 190 for validation.
  • Availability: The labeled dataset and training software are publicly released to support future research.

3. Key Contributions

1. OmniUnet Architecture: A flexible, transformer-based network capable of processing heterogeneous RGB-D-T data for semantic segmentation, demonstrating strong performance in complex, unstructured terrains.
2. Custom Sensor Integration: Development and integration of a multimodal sensor housing on the MaRTA rover, enabling synchronized capture of RGB, depth, and thermal data.
3. New Labeled Dataset: Creation and public release of a labeled RGB-D-T dataset from the Bardenas semi-desert, filling a critical gap in multimodal terrain perception research.
4. Edge Deployment Validation: Successful demonstration of the model on a resource-constrained embedded platform (Jetson Orin Nano), proving its viability for real-time on-robot deployment.

4. Results

Quantitative Performance

• BASEPROD Dataset (RGB-D-T):
  • Pixel Accuracy (PA): Achieved 80.37% total accuracy.
  • Class Performance:
    • Compact terrain (safest to traverse): 77.44% accuracy.
    • Gravel: 51.83% accuracy.
    • Bushes: 40.68% accuracy.
    • Rock: 18.40% accuracy (challenging due to thermal signature variability).
• RUGD Dataset (RGB Only):
  • Validated the model's flexibility by training on the standard RUGD dataset (25 classes), achieving 92.58% total PA and 39.72% Mean PA.

Qualitative Performance

• The model effectively distinguished between sandy and compact terrain by leveraging thermal inertia differences (sand retains less heat, appearing cooler/warmer depending on time of day, while compact soil has different thermal properties).
• In some instances, the network's output was more accurate than human ground truth in identifying transition zones between terrain types.

Inference Speed

• Jetson Orin Nano: Average inference time of 673 ms per image.
• RTX 2070: Average inference time of 137.50 ms.
• Significance: The 673 ms latency on the Jetson confirms the model is suitable for near-real-time navigation loops on planetary rovers.

5. Significance and Future Work

• Planetary Relevance: The use of thermal imagery is critical for Mars exploration, where low atmospheric pressure enhances thermal contrasts, aiding in slip detection and traversability estimation. OmniUnet leverages this specific advantage.
• Robustness: The ability to fuse thermal data allows navigation in low-light or high-glare conditions where RGB cameras fail.
• Open Science: By releasing the code and dataset, the authors facilitate reproducibility and accelerate the development of multimodal perception systems for robotics.
• Future Directions:
  • Improving detection of visually similar surfaces and obstacles (e.g., rocks).
  • Integrating a dedicated obstacle detection module.
  • Optimizing sensor housings for low-light stereo-thermal imaging to further enhance reliability in extreme environments.

In conclusion, OmniUnet represents a significant step forward in autonomous planetary navigation by providing a robust, multimodal solution that effectively bridges the gap between advanced transformer architectures and the practical constraints of embedded robotic systems.