💻 computer science

Deep Learning Aided Vision System for Planetary Rovers

This paper presents a scalable, compute-efficient vision system for planetary rovers that combines a real-time module using CLAHE-enhanced stereo imagery and YOLOv11n for object detection with an offline module leveraging Depth Anything V2 for dense 3D reconstruction, achieving a median depth error of 2.26 cm on Chandrayaan 3 NavCam data.

Original authors: Lomash Relia, Jai G Singla, Amitabh, Nitant Dube

Published 2026-03-31

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Lomash Relia, Jai G Singla, Amitabh, Nitant Dube

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are driving a car on the Moon. You can't call your mom for directions because the signal takes too long to get back to Earth. You also can't see very well because the ground is covered in gray dust, shadows are pitch black, and there are no road signs.

This paper describes a new "smart eyes" system designed to help a robotic rover drive itself safely in this tricky environment. It's like giving the robot a brain that can see obstacles instantly while driving, and then a super-powered memory to map out the terrain later.

Here is how the system works, broken down into simple parts:

1. The Two-Part Brain: "Drive Now" vs. "Map Later"

The system is split into two modes, like a human driver who has to react quickly to a squirrel crossing the road, but also has a GPS app that maps the whole neighborhood later.

The "Drive Now" Mode (Real-Time): This part runs on the rover's computer while it's moving. It needs to be fast and light, like a sports car, because the rover doesn't have a super-computer on board.
The "Map Later" Mode (Offline): This part runs back on Earth after the rover sends the pictures home. It uses massive computing power to build a detailed 3D model of the landscape, like a video game designer creating a high-definition world.

2. Fixing the "Gray Fog" (CLAHE)

The Moon's surface is mostly gray dust (regolith) with very little contrast. It's like trying to see a gray cat on a gray carpet in a dim room.

The Solution: Before the robot tries to see anything, the system applies a filter called CLAHE. Think of this as putting on special "night vision" glasses that brighten the shadows and make the edges of rocks pop out, turning a muddy gray image into something clear enough to read.

3. Spotting the Rocks (YOLO)

Once the image is clear, the robot needs to know what it is looking at. Is that a big rock? A crater? Or just a shadow?

The Tool: They use a model called YOLOv11n. Imagine a very fast, tiny security guard who can scan a crowd and instantly shout, "Rock! Rock! Artifact!" without getting tired.
The Trick: Instead of scanning the whole image (which is slow), the robot only looks closely at the specific boxes where it thinks an object is. It's like using a magnifying glass only on the suspicious spots rather than staring at the whole wall.

4. Guessing the Distance (The Neural Network)

Knowing what an object is isn't enough; the rover needs to know how far away it is to avoid crashing.

The Old Way: Traditionally, robots use complex math (triangulation) to measure distance by comparing two camera angles. It's like holding your thumb up and closing one eye, then the other, to guess how far away your thumb is. This math is heavy and slow for a small robot.
The New Way: The authors trained a Neural Network (a type of AI) to learn the pattern. They showed it thousands of examples where the math was already done. Now, the AI acts like a seasoned expert who can glance at a rock and instantly say, "That's 3 meters away," without doing the heavy math every time. It's the difference between calculating a tip in your head versus just knowing the rule of thumb.

5. Building the 3D World (Depth Anything V2)

While the robot is busy driving, it also takes pictures. Later, back on Earth, scientists use a powerful AI called Depth Anything V2.

The Magic: This AI can look at a single flat photo and guess how deep everything is, turning a 2D picture into a 3D sculpture.
The Result: They combine the robot's quick distance guesses with this detailed 3D map to create a "point cloud"—a digital cloud of dots that represents the Moon's surface. It's like taking a photo of a mountain and then printing it out in 3D so you can feel the peaks and valleys.

Why is this a big deal?

Speed: The robot can make decisions in real-time without waiting for Earth.
Accuracy: In tests, the AI guessed distances with an error of only about 2.26 centimeters (less than an inch) for objects between 1 and 10 meters away. That's like hitting a bullseye from across a room.
Efficiency: It uses less computer power, which is crucial because rovers run on batteries and solar panels.

In summary: This paper presents a smart, two-step vision system. It gives the rover "street smarts" to dodge rocks instantly using a lightweight AI, and "book smarts" to create a perfect 3D map of the Moon later for scientists to study. It's a major step toward sending robots that can explore the Moon and Mars completely on their own.

1. Problem Statement

Autonomous navigation for planetary rovers faces significant challenges due to:

Communication Delays: Real-time decision-making is required without constant ground intervention.
Environmental Constraints: Lunar and Martian terrains (regolith) often lack visual texture, making feature matching difficult for classical algorithms.
Hardware Limitations: Rover onboard computers have strict power and memory constraints, rendering computationally intensive classical stereo matching (e.g., Semi-Global Matching) or high-fidelity geometric triangulation (CAHVOR) impractical for real-time use.
Monochrome Imagery: Many rover cameras capture grayscale images, which reduces the effectiveness of texture-based deep learning models trained on RGB data.

The paper addresses the need for a vision system that balances real-time computational efficiency for obstacle avoidance with high-fidelity offline reconstruction for scientific terrain mapping.

2. Methodology

The authors propose a dual-pipeline architecture that decouples immediate perception from post-mission analysis:

A. Preprocessing

CLAHE (Contrast Limited Adaptive Histogram Equalization): Applied to monochrome stereo image pairs to enhance local contrast in low-texture regolith regions.
- Parameters: Clip limit of 2.0, tile grid size of 8×8.
- Goal: Improve feature visibility for subsequent detection and matching.

B. Real-Time Module (Onboard)

This module focuses on lightweight, low-latency obstacle detection and distance estimation.

Object Detection:
- Uses YOLOv11n (Nano version) trained on a converted grayscale dataset of Apollo mission images (1,579 images).
- Classes: Rocks, Craters, and Artifacts.
- Optimization: Input resolution reduced to 480×480 to balance precision/recall with inference speed and memory usage.
Region of Interest (ROI) Extraction:
- SIFT (Scale-Invariant Feature Transform) features are extracted only within the bounding boxes of detected objects, rather than the entire image, to reduce computational overhead.
Feature Matching:
- SIFT features are matched bidirectionally between the left and right stereo images.
- Objects with fewer than 4 correspondences are discarded to ensure triangulation reliability.
Neural Network Distance Estimation:
- Instead of traditional geometric triangulation, a lightweight Artificial Neural Network (ANN) estimates the 3D distance.
- Input: Four values representing the pixel coordinates of matched features ( $x_1, y_1, x_2, y_2$ ).
- Output: Real-world distance.
- Architecture: 3 hidden layers (128, 64, 16 neurons) with Leaky ReLU activation.
- Training: Trained on data generated by ISRO's CAHV utility using Mean Absolute Error (MAE) loss and the NAdam optimizer.
- Logic: The network learns the inverse proportionality between pixel disparity (difference in $x$ -coordinates) and real-world distance.

C. Offline Module (Ground Station)

This module generates high-resolution 3D terrain models for scientific analysis.

Monocular Depth Estimation: Uses Depth Anything V2 (metric variant) to generate dense depth maps from single grayscale images.
Point Cloud Fusion:
- Combines the metric distances from the real-time ANN with the dense depth maps.
- Uses Open3D to fuse these data points into a dense, metrically consistent 3D point cloud.
- Color-codes the point cloud based on input image intensity for visualization.

3. Key Contributions

Hybrid Architecture: Successfully integrates lightweight deep learning for real-time perception with advanced monocular depth models for offline reconstruction.
Efficient Feature Matching: Introduces a strategy of restricting SIFT feature extraction to object bounding boxes, significantly reducing computational load while maintaining accuracy.
Data-Driven Triangulation: Replaces traditional, iterative geometric triangulation with a feed-forward ANN, enabling batch processing and faster inference suitable for resource-constrained hardware.
Grayscale Adaptation: Demonstrates the viability of training modern object detection models (YOLO) on converted grayscale imagery for lunar environments.
Validation on Real Data: The system is evaluated using actual Chandrayaan-3 Pragyan rover NavCam stereo imagery.

4. Results

Object Detection:
- YOLOv11n (480×480) was identified as the optimal configuration, offering the best balance between precision (minimizing false positives) and computational footprint.
- The model successfully detects rocks, craters, and artifacts in low-texture lunar scenes.
Distance Estimation Accuracy:
- Tested against 3,855 unseen samples using ISRO's CAHV utility as the ground truth.
- Median Absolute Error: 2.26 cm within the critical 1–10 meter range.
- Interquartile Range: 0.91 cm to 5.58 cm.
- The ANN outperformed the CAHV tool in consistency for distant objects (>10m), where CAHV often produced spatially inconsistent values.
Performance:
- The ANN approach allows for scalable, batched inference in a single forward pass, whereas CAHV requires sequential processing for each feature pair.
- The system achieved full automation, eliminating the need for manual feature selection required by legacy tools.

5. Significance and Future Work

Significance: This work provides a scalable, compute-efficient solution for autonomous planetary exploration. It proves that deep learning can replace or augment classical geometric methods in harsh, low-texture extraterrestrial environments, enabling reliable obstacle detection without heavy onboard processing.
Future Directions:
- Physics-Informed Neural Networks (PINNs): To embed geometric constraints directly into the network for better generalization.
- Autonomous Control: Integrating lightweight Reinforcement Learning (RL) policies for onboard decision-making.
- SLAM Integration: Coupling the vision system with Visual Odometry (VO) and Simultaneous Localization and Mapping (SLAM) frameworks specifically tailored for lunar terrain.

In conclusion, the proposed system offers a robust pathway for next-generation planetary rovers to navigate autonomously while generating high-quality scientific data, effectively bridging the gap between real-time operational needs and post-mission scientific analysis.