SpaceSense-Bench: A Large-Scale Multi-Modal Benchmark for Spacecraft Perception and Pose Estimation

This paper introduces SpaceSense-Bench, a large-scale, multi-modal benchmark generated via high-fidelity Unreal Engine 5 simulations that provides 136 diverse satellite models with synchronized RGB, depth, and LiDAR data alongside dense semantic and pose annotations to address the scarcity of real-world space data and demonstrate the critical importance of dataset scale and diversity for advancing spacecraft perception and pose estimation.

The Gist

Imagine you are trying to teach a robot to be a space mechanic. Its job is to fly up to a broken satellite, grab onto it, fix it, or maybe push it out of the way so it doesn't crash into other satellites.

To do this, the robot needs "eyes" and a "brain." It needs to see the satellite, understand exactly what part it is looking at (is that a solar panel? a thruster?), and know exactly where the satellite is in 3D space.

The Problem:
You can't just send a camera to space and take a million photos of every possible satellite. It's too expensive, too dangerous, and there are too many different types of satellites out there. Plus, space is weird: sometimes it's blindingly bright, sometimes it's pitch black, and satellites are covered in shiny metal that reflects light in confusing ways.

Existing training data for robots is like a tiny, boring photo album. It usually only has pictures of one or two specific satellites. If you train a robot on just one model, it gets really good at recognizing that one satellite, but if you put it in front of a different one, it gets completely confused. It's like teaching a child to recognize only their own dog; if they see a cat, they might think it's a weird dog, or they might not recognize it at all.

The Solution: SpaceSense-Bench
The authors of this paper built a massive, ultra-realistic virtual space simulator (using game engine technology called Unreal Engine 5) to solve this. Think of it as a "Flight Simulator" for space robots, but instead of just flying planes, it's generating millions of training scenarios.

Here is what makes SpaceSense-Bench special, explained simply:

1. The "Giant Toy Box" (136 Satellites)

Instead of just one or two models, they created 136 different satellite models.

• The Range: Some are tiny, like a shoebox (CubeSats). Others are huge, as big as the International Space Station.
• The Variety: They are all different shapes, sizes, and functions. This forces the robot's AI to learn the general rules of what a satellite looks like, rather than just memorizing one specific picture.

2. The "Super-Senses" (Multi-Modal Data)

In the real world, if you close your eyes, you can still feel a wall or hear a car. Space robots need similar backup senses.

• RGB: Standard color camera photos (what you see).
• Depth: A map that tells the robot exactly how far away every pixel is (like a 3D ruler).
• LiDAR: A laser scanner that shoots out 256 beams to create a precise 3D point cloud of the object.
• The Magic: In this dataset, all three sensors capture data at the exact same time. This is like having a person who can see, feel, and measure an object simultaneously, perfectly synchronized.

3. The "Perfect Teacher" (Dense Annotations)

Usually, when you train an AI, you have to manually draw boxes around objects in thousands of photos. That takes forever.

• The Trick: Because this is a computer simulation, the "teacher" knows everything automatically. For every single frame, the system knows exactly which pixel is a "solar panel," which is a "thruster," and exactly where the satellite is in 3D space.
• The Labels: They broke satellites down into 7 specific parts: the main body, solar panels, dish antennas, stick antennas, science instruments, thrusters (engines), and the ring used to attach to rockets.

4. The "Stress Test" (What they found)

The researchers used this dataset to test the best AI models available today. They found two big things:

• The "Tiny Detail" Problem: The AI is great at spotting big things like solar panels (which take up most of the image). But it struggles terribly with tiny parts like small antennas or thrusters. These are like trying to spot a specific grain of sand on a beach from a helicopter. The AI often misses them or gets confused.
• The "More is Better" Rule: They tested what happens if they train the AI on more satellites. The result? The more satellites the AI sees, the better it gets at recognizing satellites it has never seen before.
  • Analogy: If you only show a child 5 pictures of cars, they might think all cars are red. If you show them 100 pictures of red, blue, truck, and sports cars, they learn what a "car" actually is. The same applies to space robots.

Why Does This Matter?

This dataset is like a gym for space robots. By training on this massive, diverse, and perfectly labeled virtual world, we can build robots that are actually ready for the real world.

Instead of sending a robot that might crash because it doesn't recognize a new type of satellite, we can send one that has "seen" 136 different types in a simulation and knows exactly how to handle them. This is a huge step toward making on-orbit servicing (fixing satellites in space) and debris removal (cleaning up space junk) a reality.

In short: They built the ultimate video game training ground so that real-life space robots won't get lost when they finally leave Earth.

Technical Summary

Here is a detailed technical summary of the paper "SpaceSense-Bench: A Large-Scale Multi-Modal Benchmark for Spacecraft Perception and Pose Estimation."

1. Problem Statement

Autonomous space operations, such as on-orbit servicing (OOS) and active debris removal, require spacecraft to perform robust part-level semantic understanding and precise relative navigation. However, developing these capabilities faces three critical bottlenecks:

• Data Scarcity: Collecting large-scale, multi-modal real-world data in orbit is impractical due to high costs and access constraints.
• Limitations of Existing Synthetic Datasets: Current benchmarks suffer from:
  • Low Diversity: Most contain only 1–16 satellite models, leading to models that overfit to specific textures rather than learning generalizable geometry.
  • Single Modality: They lack time-synchronized RGB, depth, and LiDAR data, hindering multi-modal fusion research.
  • Incomplete Annotations: They often lack dense, pixel-level and point-level part semantics and accurate 6-DoF pose ground truth.
• Generalization Failure: Current deep learning methods struggle to generalize to entirely unseen spacecraft configurations (zero-shot setting), particularly for small-scale components like thrusters and antennas.

2. Methodology: SpaceSense-Bench Construction

The authors present SpaceSense-Bench, a large-scale, high-fidelity synthetic dataset generated via a fully automated pipeline built on Unreal Engine 5 and the AirSim plugin.

A. Dataset Scale and Diversity

• Models: 136 diverse satellite models (ranging from 0.27 m CubeSats to 112 m ISS-class structures) sourced from NASA and ESA.
• Data Volume: Approximately 70 GB of data.
  • Sparse Sampling: 90,000 synchronized frames (used for current benchmarks).
  • Dense Sampling Potential: The pipeline can scale to >2 million frames without manual intervention.
• Trajectories: Two adaptive trajectory families (Approach and Orbit) scaled by target diameter to ensure uniform sampling density across all sizes.

B. Multi-Modal Sensing & Ground Truth

Every frame provides strictly time-synchronized and spatially aligned data:

1. RGB Images: 1024×1024 resolution.
2. Depth Maps: Millimeter-precision.
3. LiDAR Point Clouds: 256-beam.
4. Semantic Labels: Dense 7-class part-level annotations at both pixel and point levels. The classes are:
   • Main Body, Solar Panel, Dish Antenna, Omni Antenna, Payload, Thruster, Adapter Ring.
5. Pose: Accurate 6-DoF relative pose ground truth.

C. Simulation Environment

• Lighting: Simulates extreme orbital conditions including direct sunlight, Earth Albedo (diffuse reflection), and eclipse (near-total darkness).
• Physics: Uses hardware ray tracing for physically accurate specular reflections on metallic surfaces and sharp shadows.
• Automation: A four-stage pipeline handles asset decomposition, scene setup, trajectory planning, and automated quality control (filtering invalid frames based on label validity and cross-modal consistency).

3. Key Contributions

1. First Large-Scale Multi-Modal Benchmark: The first dataset to jointly provide synchronized RGB, Depth, and LiDAR for 136 satellite models.
2. Dense 2D/3D Annotations: Provides 7-class part-level semantic labels for both pixels and points, plus 6-DoF pose, automatically converted into mainstream formats (YOLO, MMSegmentation, SemanticKITTI).
3. High-Fidelity Simulation: A photorealistic environment in Unreal Engine 5 that captures complex orbital illumination and material properties.
4. Systematic Benchmarking: Established evaluation protocols for five tasks, revealing critical gaps in current state-of-the-art (SOTA) methods regarding small-object perception and zero-shot generalization.

4. Experimental Results & Analysis

The authors benchmarked five tasks on a zero-shot test set (14 satellites entirely unseen during training):

• 2D Semantic Segmentation: SOTA models (Mask2Former, SegFormer) achieved a mean Intersection-over-Union (mIoU) of only ~45.6%.
  • Performance Gap: Large components (Solar Panels, Main Body) scored >68% IoU, while small components (Omni Antenna, Thruster) scored <35% IoU.
• 3D Point Cloud Segmentation (RGB-LiDAR Fusion): PMFNet achieved 42.4% mIoU, confirming that multi-modal fusion helps but does not fully solve the small-object problem.
• Object Detection: YOLO26 showed similar trends, with high accuracy on large bodies but poor detection of small parts (e.g., Thruster AP@0.5 < 10% for smaller models).
• Depth & Orientation Estimation:
  • Depth: Depth Anything V2 (zero-shot) achieved high accuracy (AbsRel = 0.022) but struggled with relative depth ordering on metallic surfaces (Spearman correlation = 0.60).
  • Pose: Orient Anything achieved a Mean Axis Angular Error (MAAE) of 12.75°, with significant variance based on target symmetry.

Key Findings

1. Small-Object Bottleneck: Perceiving small-scale components (thrusters, omni-antennas) remains a critical failure point due to their tiny pixel footprint (<2% of foreground) and high morphological diversity.
2. Data Scaling Law: Increasing the number of training satellites from 9 to 117 resulted in a 73% relative gain in mIoU (from 24.4% to 42.4%). This proves that large-scale, diverse datasets are essential for zero-shot generalization, as models learn geometric priors rather than memorizing specific textures.

5. Significance

• Advancing Space AI: SpaceSense-Bench addresses the "data gap" that has hindered the application of deep learning in space perception, moving the field from single-target, single-modality research to generalizable, multi-modal systems.
• Enabling Robust Autonomy: By providing dense part-level semantics and 6-DoF pose, the dataset supports critical OOS tasks like robotic grasping, docking, and debris removal.
• Community Resource: The dataset, code, and toolkit are publicly available, providing a standardized benchmark to drive future research in sim-to-real transfer, 3D reconstruction, and cross-modal generation.
• Future Directions: The authors plan to expand the dataset using 3D generative models to synthesize novel geometries and validate sim-to-real transfer using real on-orbit data.

In summary, SpaceSense-Bench establishes a new standard for spacecraft perception research, demonstrating that scaling dataset diversity is the most effective path toward robust, autonomous space operations.