Onboard-Targeted Segmentation of Straylight in Space Camera Sensors

This study presents an AI-based methodology using a lightweight DeepLabV3-MobileNetV3 model, pre-trained on diverse flare data to overcome space-specific data scarcity, for the onboard semantic segmentation of solar-induced straylight in space camera sensors, complete with system-level performance metrics for integration into spacecraft navigation pipelines.

The Gist

Imagine you are the captain of a spaceship traveling to a distant comet. Your ship relies on a high-tech camera to "see" where it is, much like your eyes help you walk down a street. But space is tricky. Sometimes, the Sun gets too close to the camera's view, creating blinding flashes, streaks of light, or "glare" that look like giant, messy sunspots.

If the ship's computer tries to navigate using a picture full of these blinding flashes, it might get confused, think a star is a rock, or crash into a wall. This is a straylight problem.

This paper is about teaching the spaceship's computer a new trick: how to instantly spot these blinding flashes and ignore them.

Here is the story of how they did it, broken down into simple parts:

1. The Problem: The "Blind Spot" in Space

Space cameras are great, but they have a weakness. When the Sun shines into the lens, it creates weird, bright patterns (flares) that look like noise.

• The Old Way: Engineers usually try to block this light with physical shields (baffles), but they aren't perfect.
• The New Way: Use Artificial Intelligence (AI) to look at the picture, say, "Hey, that bright spot is just a glare, not a real object," and tell the navigation system to ignore it.

2. The Challenge: Not Enough Practice Photos

To teach an AI to recognize something, you usually need thousands of photos of that thing.

• The Issue: There are very few photos of "space camera glitches" because space missions are rare and expensive. The authors had a small dataset of simulated space glitches, but it wasn't enough to make the AI a master.
• The Analogy: Imagine trying to teach a child to recognize a "dog" by only showing them three pictures of a Golden Retriever. They might think all dogs are Golden Retrievers. You need to show them many different dogs first.

3. The Solution: "School" on Earth, "Exam" in Space

To solve the lack of space photos, the researchers used a clever two-step training method:

1. Elementary School (Pre-training): They taught the AI using a massive dataset of Earth-based photos containing lens flares (like sun glare on a car windshield or a camera lens). This taught the AI the general concept of what a "glare" looks like (bright, streaky, messy).
2. Graduate School (Fine-tuning): Once the AI understood the basics of glare, they gave it the specific "Space Exam" using their small dataset of space camera glitches. This taught the AI how glare looks specifically in the vacuum of space.

The Result: The AI became an expert at spotting space flares, even though it had never seen a real space flare before it started training.

4. The Brain: A Lightweight Computer

Spacecraft have very limited computing power (they can't carry heavy, energy-hungry supercomputers).

• The Metaphor: Think of a standard AI model as a heavy, fuel-guzzling truck. It's powerful but can't fit in a small spaceship.
• The Fix: The researchers used a "compact" AI model (DeepLabV3 with a MobileNet backbone). Think of this as a sleek, electric scooter. It's lightweight, uses very little energy, but is still fast enough to do the job. This ensures the AI can run onboard the ship in real-time, not just on Earth.

5. The New Scorecard: "Did You Catch the Glare?"

Usually, when we test AI, we count how many pixels (tiny dots in the image) were labeled correctly.

• The Problem with Pixel Counting: Imagine a glare is a big, messy blob. If the AI misses one tiny dot in the middle of the blob, a standard score says, "You failed!" But for the spaceship, it doesn't matter if you missed one dot; what matters is that you saw the whole blob and ignored it.
• The New Metric: The authors created a new way to grade the AI. Instead of counting dots, they count objects.
  • Goal: Did the AI spot the entire glare? (Yes/No).
  • Why? If the AI misses a glare, the ship might crash. If the AI accidentally ignores a tiny bit of a real star (a false alarm), the ship just has slightly less data, which is okay. It's better to be safe than sorry.

6. The Big Picture: A Safer Journey

By combining a lightweight AI, smart training tricks, and a new way of grading success, this research gives spacecraft a "superpower."

• Before: The ship might get blinded by the Sun and lose its way.
• After: The ship sees the glare, says, "I see that, I'll ignore it," and keeps navigating safely using the clear parts of the image.

In summary: This paper teaches a spaceship's brain how to blink its eyes when the Sun shines too bright, ensuring it never gets lost in the dark.

Technical Summary

1. Problem Statement

Modern space missions, particularly autonomous planetary exploration (e.g., the Astrone KI mission concept for comet 67P), rely heavily on camera sensors for Guidance, Navigation, and Control (GNC). These sensors are susceptible to straylight effects (solar flares and glares) when the Sun enters or approaches the Field of View (FoV).

• Impact: Straylight creates artifacts that obstruct visibility and corrupt sensor readings. If processed by onboard navigation algorithms (like Kalman filters), these corrupted pixels can lead to erroneous state estimation and mission failure.
• Constraints:
  • Real-time Requirement: Unlike static faults (e.g., broken pixels), straylight is transient and dynamic, requiring immediate detection within a single image frame.
  • Hardware Limitations: Onboard processing must run on resource-constrained, space-qualified hardware (Commercial Off-The-Shelf or COTS components), limiting memory and computational power (FLOPs).
  • Data Scarcity: There is a lack of realistic, labeled datasets for space-specific camera faults, making traditional deep learning training difficult.

2. Methodology

The authors propose an AI-based semantic segmentation pipeline designed for onboard deployment.

A. Model Architecture

• Backbone: MobileNetV3 Large, chosen for its efficiency in memory and inference speed on embedded systems.
• Segmentation Head: DeepLabV3, utilizing Atrous Spatial Pyramid Pooling (ASPP) to capture multi-scale context without increasing computational overhead.
• Configuration: Output stride set to 16 with atrous rates of (6, 12, 18) to balance detail and context.

B. Training Strategy (Addressing Data Scarcity)

To overcome the lack of space-specific data, the authors employed a Transfer Learning approach:

1. Pre-training: The model was first trained on the Flare7k++ dataset (7,962 images of terrestrial flares/glare). This allowed the model to learn general flare textures and shapes across diverse backgrounds.
2. Fine-tuning: The pre-trained model was subsequently fine-tuned on a proprietary, synthetic dataset of space camera faults (1,000 images) containing specific straylight artifacts relevant to the mission scenario.
3. Dataset Composition: The final task was simplified to a binary segmentation problem: Nominal vs. Straylight.

C. Evaluation Framework

The authors introduced a novel evaluation paradigm shifting from pixel-level accuracy to artifact-level performance, reflecting the needs of the navigation subsystem.

• Conventional Metrics: Precision, Recall, and mean Intersection over Union (mIoU).
• Custom System-Level Metrics:
  • Per-artifact Recall (PaR): Measures the ability to detect entire flare objects (critical to prevent false negatives).
  • Per-artifact Precision (PaP): Measures the rate of false positive detections.
  • Per-artifact mIoU (PamIoU): Intersection over Union calculated per object rather than per pixel.
• Preprocessing for Metrics: To handle fragmentation in ground truth masks (caused by sequential rendering of faults in the dataset), a Gaussian smoothing algorithm was applied to connect fragmented regions before metric calculation. This ensures that a single flare is treated as one contiguous object.

3. Key Contributions

1. Onboard-Compatible Architecture: A DeepLabV3/MobileNetV3 model specifically optimized for space-grade hardware constraints (low memory, low FLOPs).
2. Transfer Learning for Space: A successful methodology demonstrating how pre-training on large terrestrial flare datasets can mitigate data scarcity for space-specific fault detection.
3. System-Aware Metrics: The development of Per-artifact metrics (PaR, PaP, PamIoU) and a preprocessing pipeline (Gaussian smoothing) that aligns model evaluation with the operational requirements of navigation systems (prioritizing the detection of whole artifacts over perfect pixel boundaries).
4. Integration Framework: A proposed interface between the AI segmentation output and the onboard navigation pipeline, enabling Fail-Operational states where corrupted pixels are excluded from Kalman filter updates rather than triggering a full system shutdown.

4. Results

• Performance Improvement: Fine-tuning on the proprietary space dataset significantly improved performance over the pre-trained-only model.
  • Conventional Metrics: Precision increased from 0.259 to 0.908; Recall from 0.403 to 0.958; mIoU from 0.188 to 0.873.
  • Custom Metrics: The fine-tuned model achieved a PaR of 0.991, indicating near-perfect detection of straylight artifacts, which is critical for mission safety.
• Artifact Detection: The model successfully learned to segment complex flare textures and distinguish them from the Sun itself (which was initially misclassified as a flare in the pre-trained stage).
• Metric Discrepancy: The study highlighted that conventional metrics (like mIoU) can be misleading for this application. The custom Per-artifact metrics provided a more accurate reflection of the model's utility in a navigation context, where missing an entire flare (False Negative) is more detrimental than a slight boundary error.

5. Significance and Future Work

• Mission Safety: This work bridges the gap between AI research and practical space operations, offering a solution to maintain navigation continuity during sensor anomalies (Fail-Operational capability).
• Autonomy: By enabling real-time, onboard fault detection, the system reduces reliance on ground intervention, which is crucial for deep-space missions with long communication delays.
• Future Directions:
  • Validating the model on real-world mission imagery to address domain shifts between synthetic and real data.
  • Hardware-in-the-Loop (HIL) testing on space-qualified FPGAs (e.g., Zynq UltraScale+) to verify real-time processing capabilities.
  • Refining the integration architecture to functionally demonstrate system resilience and availability during sensor-level anomalies.

In conclusion, the paper presents a robust, resource-efficient AI solution for detecting solar straylight in space cameras, validated through a novel system-centric evaluation framework that prioritizes mission safety over pure pixel-level accuracy.