Simulation Based Reward Function Validation for Multi-Agent On Orbit Inspection

Technical Summary: Simulation Based Reward Function Validation for Multi-Agent On Orbit Inspection

Problem Statement
The accumulation of space debris necessitates active removal missions, which require prior data gathering to allocate resources effectively. Current orbital inspection missions often rely on manual planning for proximity operations, a process that is time-consuming and difficult to scale for large debris populations. While Multi-Agent Reinforcement Learning (MARL) has been proposed to automate the control of inspection spacecraft groups, existing approaches typically rely on reward functions that guide agents to a finite, predetermined set of inspection points. This rigidity limits the agents' ability to adapt to arbitrary inspection requirements. Furthermore, the effectiveness of these reward functions is often difficult to validate objectively without a direct link to the quality of the resulting data.

Methodology
This work proposes a generalized MARL framework for orbital inspection where agents are not constrained to fixed waypoints but can control the timing and location of image collection arbitrarily. The system employs a hierarchical control structure with two distinct agent types operating within a Proximal Policy Optimization (PPO) framework:

• Upper-Level Agents: Responsible for high-level decision-making, including selecting target points, determining when to capture images, and deciding when to terminate an episode.
• Lower-Level Agents: Responsible for low-level control, generating thrust commands to navigate to target points while minimizing fuel usage and avoiding collisions.

The environment utilizes Hill-Clohessy-Wiltshire (HCW) relative dynamics within a Local Vertical Local Horizontal (LVLH) frame. The reward function is the core innovation, designed to be generalized for any number of images at arbitrary locations. It is composed of several weighted components:

1. Image Value: A composite score based on distance, angular separation from other images, lighting conditions (sun angle), and avoidance of self-shadowing.
2. Team vs. Individual Rewards: A mixing coefficient balances the global team image value against individual agent contributions to encourage cooperation while preventing "laziness."
3. Penalties: Costs are applied for time elapsed, fuel consumption, target point errors, collisions, and drifting too far from the operational region.

To validate these reward functions, the authors utilize a high-fidelity simulation pipeline. Trajectories and image metadata generated by the trained MARL agents are exported to NVIDIA Isaac Sim to capture photo-realistic images. These images are then processed by COLMAP (photogrammetry) and Instant-NGP (neural radiance fields) to generate 3D reconstructions of the target. The quality of these reconstructions serves as the objective metric for reward function performance. The training process employs curriculum learning, gradually increasing fuel penalties and removing flat rewards for staying in the operational region to prevent agents from learning trivial policies (e.g., doing nothing or drifting immediately).

Key Contributions
The paper provides four primary contributions:

1. An open-ended MARL framework using PPO that accommodates images at any location and in any quantity, moving beyond fixed waypoint constraints.
2. A detailed methodology for testing and refining MARL inspection implementations, specifically focusing on the validation of reward functions through 3D reconstruction quality.
3. Identification of future improvements for RL-based inspection, including alternative observation representations and control architectures.
4. General takeaways applicable to both the specific MARL task and the broader field of orbital inspection.

Results
The study demonstrates that tuning reward function weights significantly impacts agent behavior. Specifically, increasing the fuel penalty weight from 3 to 4 shifted the policy from random, inefficient trajectories to fuel-efficient paths that leverage natural HCW dynamics for planar motion while using active control for out-of-plane adjustments. However, the authors note that excessive fuel penalties caused agents to rely entirely on natural dynamics, and overly strict image separation requirements discouraged image collection entirely.

In terms of reconstruction, COLMAP produced clean but incomplete reconstructions, likely hindered by changing lighting conditions during the orbit. Instant-NGP produced more complete but fuzzier reconstructions. The authors observed that while the generated policies were functional, they still exhibited deficiencies, and the training process was sensitive to parameter combinations.

Significance and Claims
The authors claim that while the presented MARL methods generate policies supporting the feasibility of automated orbital inspection, the techniques require further refinement before space deployment. The primary value of this work lies in the ability of MARL to allow mission designers to explore general lessons and strategies for inspection tasks. The insights gained regarding reward shaping, curriculum learning, and the trade-offs between fuel usage and data quality are applicable to inspection tasks utilizing traditional controllers as well.

The paper modestly concludes that while the current results are promising, they are not yet sufficient for immediate deployment. Future work is identified as necessary in the domain of RL safety assurances for proximity operations and the potential for full automation of the reconstruction analysis loop to provide direct RL rewards based on reconstruction quality, a goal currently limited by computational feasibility.