A New Dataset and Performance Benchmark for Real-time Spacecraft Segmentation in Onboard Computers

Technical Summary: A New Dataset and Performance Benchmark for Real-time Spacecraft Segmentation in Onboard Computers

1. Problem Statement

As orbital infrastructure expands, there is an urgent need for autonomous spacecraft inspection systems to monitor damage and facilitate repairs. However, developing reliable computer vision models for this task faces three critical hurdles:

• Data Scarcity: Existing datasets (like NASA’s Pose-Bowl) lack diversity in spacecraft geometry and background conditions, and many lack the pixel-level segmentation masks required for training robust models.
• Hardware Constraints: Space-grade flight computers are extremely resource-constrained, typically limited to 4-core CPUs and less than 4GB of RAM, making heavy transformer-based or large CNN models impractical.
• Real-time Requirements: For proximity operations (docking or inspection), segmentation must occur in real-time, necessitating an inference latency of less than 0.95 seconds.
• Metric Limitations: Standard metrics like the Dice coefficient often fail to account for boundary localization precision, which is vital for safe spacecraft maneuvering.

2. Methodology

The authors introduce the SWiM (Spacecraft With Masks) dataset and a benchmarking framework designed to bridge the gap between laboratory models and orbital deployment.

A. Dataset Synthesis (SWiM):
The researchers created a massive dataset of nearly 64,000 annotated images using a dual-methodology approach:

1. Preprocessing Existing Data: They standardized the Pose-Bowl and Spacecrafts datasets. For Pose-Bowl (which only had bounding boxes), they used the Segment Anything Model 2 (SAM 2) to generate high-quality segmentation masks.
2. Synthetic Generation: To increase diversity, they used two synthetic methods:
   • Method 1 (Superimposition): Segmented spacecraft from existing datasets were superimposed onto augmented open-source backgrounds (ESA/Hubble/Sentinel-2) with various geometric and photometric distortions.
   • Method 2 (TTALOS + Stable Diffusion): They used NASA’s TTALOS pipeline to render 3D spacecraft models and combined them with photorealistic backgrounds generated via Stable Diffusion. This allowed for "extreme" scenarios like auroras, intense solar glare, and city lights.

B. Benchmarking Framework:
To simulate real-world deployment, the authors:

• Model Selection: Chose YOLOv8n and YOLOv11n (nano variants) due to their proven efficiency on edge devices.
• Hardware Simulation: Used a Docker container restricted to 3GB RAM and 3 CPU cores to mimic NASA’s inspector spacecraft flight computers.
• Optimization: Exported models to ONNX format and applied quantization via ONNX Runtime to ensure high-speed CPU inference.

3. Key Contributions

• The SWiM Dataset: The largest and most comprehensive spacecraft segmentation dataset to date, featuring high diversity in poses, lighting, and backgrounds.
• Hardware-Centric Benchmarking: The first benchmark that explicitly evaluates performance under strict orbital hardware (CPU-only, <4GB RAM) and latency (<0.95s) constraints.
• Dual-Metric Protocol: Implementation of a combined evaluation strategy using the Dice Coefficient (for regional overlap) and Hausdorff Distance (for boundary precision).

4. Results

The experiments yielded highly successful benchmarks for real-time onboard use:

• Accuracy: Both YOLOv8n and YOLOv11n achieved a high Dice score of ~0.92, demonstrating excellent overlap with ground truth.
• Boundary Precision: The models maintained low Hausdorff distances (0.69–1.07), indicating reliable edge detection.
• Inference Speed: The models achieved an average inference time of approximately 0.5 seconds (well below the 0.95s limit), with frame rates around 1.7 to 2.25 FPS.
• Robustness: The models showed high resilience to the noise and distortions introduced in the "Augmented" version of the dataset, with negligible drops in Dice scores.

5. Significance

This work provides a standardized foundation for the development of autonomous space inspection. By providing a massive, high-fidelity dataset and a realistic testing environment, the authors have moved the field closer to deploying "intelligent" vision systems on actual NASA inspector spacecraft. The methodology ensures that future researchers are not just optimizing for accuracy in a vacuum, but for the harsh, resource-constrained reality of deep space exploration.