Optimal Multi-Debris Mission Planning in LEO: A Deep Reinforcement Learning Approach with Co-Elliptic Transfers and Refueling

This paper proposes a unified co-elliptic maneuver framework for multi-debris removal in Low Earth Orbit and demonstrates through comparative analysis that a Masked Proximal Policy Optimization deep reinforcement learning approach significantly outperforms Greedy heuristics and Monte Carlo Tree Search in mission efficiency and computational speed.

The Gist

Imagine the Low Earth Orbit (LEO) around our planet as a massive, chaotic highway. Instead of cars, this highway is filled with thousands of pieces of space junk—dead satellites, broken rocket parts, and tiny metal shards. If left alone, these pieces could crash into each other, creating a domino effect of explosions that makes space travel impossible for decades. This is known as the "Kessler Syndrome."

To stop this, we need "Space Janitors": special spacecraft designed to fly out, grab these pieces of junk, and drag them down to burn up in the atmosphere. But here's the problem: There are too many pieces of junk, and the Space Janitor has limited fuel and time.

This paper is about teaching a Space Janitor how to be the most efficient cleaner possible using Artificial Intelligence (AI).

The Challenge: The Ultimate Road Trip

Imagine you are a delivery driver in a giant city. You have a truck with a limited gas tank, and you need to drop off packages at 50 different houses scattered across the city.

• The Goal: Visit as many houses as possible before you run out of gas or time.
• The Catch: You can't just drive straight to the next house. You have to follow specific traffic rules (orbital mechanics), sometimes you need to stop at a gas station (refueling), and you must avoid crashing into other cars (safety zones).

The paper compares three different "drivers" (algorithms) to see who can clean up the most trash:

1. The "Greedy" Driver: This driver looks only at the house right next door. They pick the closest one, go there, then look for the next closest one. They don't think about the future.
   • Result: They are fast, but they often get stuck in a corner or run out of gas because they didn't plan the route ahead.
2. The "Super-Planner" (MCTS): This driver sits down and simulates millions of different possible routes in their head before making a single move. They think, "If I go here, then there, then maybe I should gas up..."
   • Result: They find a great route, but it takes them so long to think that by the time they decide where to go, the mission time is almost up. They are too slow for real-time use.
3. The "AI Learner" (Masked PPO): This is the star of the show. This driver has been trained by playing thousands of virtual versions of this game. They don't just look at the next house; they have "learned" the patterns of the city. They know when to take a shortcut, when to refuel, and how to chain trips together efficiently.
   • Result: They are almost as smart as the Super-Planner but move as fast as the Greedy driver.

The Secret Sauce: How They Move

The paper introduces a special way of moving called "Co-Elliptic Transfers."

Think of it like a race car on a track. If you want to catch a car in the next lane, you don't just swerve wildly (which wastes gas). Instead, you speed up or slow down slightly to get into a "shadow lane" (a safety ellipse) that runs parallel to the target. You drift along this lane until you are right next to the target, then gently merge.

The AI also uses "Safety Ellipses." Imagine approaching a fragile vase. You don't just grab it; you circle it slowly in a safe oval path to make sure you don't knock it over. This paper teaches the AI to do this with space junk, ensuring it doesn't accidentally crash into the debris it's trying to clean up.

The Results: Who Won?

The researchers ran 100 different "cleaning missions" with random junk locations. Here is what happened:

• The Greedy Driver cleaned up about 15–18 pieces of junk. They were too short-sighted.
• The Super-Planner cleaned up about 25–29 pieces. They were smart but took forever to think (sometimes hours to plan a 7-day mission).
• The AI Learner cleaned up 29–32 pieces. They were the most efficient!

The Big Win: The AI Learner visited twice as many pieces of junk as the simple Greedy driver, and it did it in just 1 or 2 seconds of computer time. The Super-Planner took thousands of seconds to do a slightly worse job.

Why This Matters

Space is getting crowded, and we can't afford to waste fuel or time. This paper proves that Deep Reinforcement Learning (a type of AI that learns by trial and error) is the future of space cleanup.

It's like upgrading from a human driver who gets tired and confused, to a self-driving car that has "seen" every possible traffic jam before it even happens. This technology could soon allow autonomous spacecraft to clean up our orbit, keeping space safe for future generations without needing a human to press every button.

In a nutshell: The paper teaches a robot how to be the ultimate space janitor, cleaning up the most trash in the least amount of time by learning from experience rather than just guessing or over-thinking.

Technical Summary

1. Problem Statement

The paper addresses the Multi-Target Active Debris Removal (ADR) problem in Low Earth Orbit (LEO). The proliferation of space debris threatens the sustainability of space activities, necessitating autonomous missions where a "chaser" spacecraft must rendezvous with, capture, and de-orbit multiple debris targets.

Key Challenges:

• Complex Constraints: Missions are bounded by strict $\Delta V$ (fuel) budgets, mission duration limits, and safety requirements (avoiding collisions with non-cooperative targets).
• Refueling Logic: The chaser must strategically decide when to return to a refueling station, balancing the time cost of the return trip against the benefit of continuing the mission with a full fuel tank.
• Dynamic Environment: The problem involves a high-dimensional state space with randomized debris fields, requiring adaptive planning strategies that traditional heuristics often struggle to optimize globally.

2. Methodology

A. Unified Co-Elliptic Maneuver Framework

The authors introduce a realistic orbital mechanics framework that integrates three distinct phases for every target visit:

1. Hohmann Transfers: Used for the bulk of the transfer. The first transfer covers 75% of the distance, and the second brings the chaser within 1 km of the target.
2. Co-Elliptic Phasing: Instead of isolated transfers, the chaser enters an intermediate co-elliptic orbit sharing the target's apogee or perigee. This allows for efficient phasing (timing adjustments) to rendezvous with debris clustered in similar orbital bands, reducing total $\Delta V$ and time.
3. Safety Ellipse Approach: For the final approach, a "safety ellipse" maneuver is employed. This ensures a controlled, slow, and safe approach to the non-cooperative debris, mitigating collision risks and accounting for navigation errors.

B. Mission Simulation Environment

• Scenario: A chaser starts at a 700 km circular refueling station.
• Targets: 50 debris objects per episode, randomly sampled between 700–800 km altitude with varying inclinations.
• Constraints:
  • $\Delta V$ Budget: Fixed at 3 km/s per refuel cycle.
  • Time Limit: Maximum mission duration of 7 days.
  • Refueling: The agent can return to the station to reset the $\Delta V$ budget, incurring a time penalty.
• State Space: Includes visited debris masks, remaining $\Delta V$, remaining time, and full Keplerian elements for the chaser and all debris.

C. Planning Algorithms Benchmarked

The study compares three distinct planning approaches:

1. Greedy Heuristic: A classical algorithm that selects the next target minimizing immediate cost ($\alpha \Delta V + \beta \text{Time}$). It is myopic and lacks long-term optimization.
2. Monte Carlo Tree Search (MCTS): A search-based method using Upper Confidence Bound (UCB) to balance exploration and exploitation. It simulates future trajectories to optimize cumulative reward but is computationally expensive.
3. Masked Proximal Policy Optimization (PPO): A Deep Reinforcement Learning (RL) agent.
   • Masking: Crucially, the action space is masked to exclude already-visited debris, ensuring valid policy distributions.
   • Reward Function: +1 for successful rendezvous, 0 for refueling/detours, and -1 for constraint violations (fuel/time exhaustion).
   • Training: Trained over 10 million steps using a custom OpenAI Gym environment.

3. Key Contributions

• Unified Framework: The integration of Hohmann transfers, co-elliptic phasing, and safety ellipse maneuvers into a single planning model that closely mimics real-world ADR operational constraints.
• Explicit Refueling Logic: The inclusion of a dynamic refueling mechanism that penalizes excessive returns, forcing the agent to learn optimal trade-offs between fuel capacity and mission duration.
• Masked PPO Implementation: The application of action masking within PPO to handle the combinatorial nature of the Traveling Salesman Problem (TSP) variant in space, effectively preventing the agent from selecting invalid actions.
• Comprehensive Benchmarking: A rigorous comparison of RL against both simple heuristics and complex search-based methods (MCTS) under identical, randomized orbital conditions.

4. Experimental Results

The algorithms were tested over 100 randomized scenarios (10 iterations per scenario with different debris fields).

Metric	Greedy Heuristic	Monte Carlo Tree Search (MCTS)	Masked PPO (RL)
Debris Visited	15 – 18	25 – 29	29 – 32
Computation Time	~1–2 seconds	1,000 – 10,000 seconds	~1–2 seconds
Strategy	Myopic (Local Optima)	Global Search (High Cost)	Learned Policy (Global Optima)

• Efficiency: Masked PPO visited up to twice as many debris objects as the Greedy heuristic and significantly outperformed MCTS in terms of debris count (approaching near-optimal counts).
• Speed: Masked PPO achieved inference speeds comparable to the Greedy algorithm (seconds), whereas MCTS was orders of magnitude slower due to extensive tree expansion and rollouts.
• Generalization: The RL agent demonstrated robust generalization across randomized debris fields without retraining.

5. Significance and Conclusion

The paper demonstrates that Deep Reinforcement Learning, specifically when combined with action masking and realistic orbital dynamics, is superior to both traditional heuristics and search-based methods for complex space mission planning.

• Practicality: The Masked PPO agent offers the "best of both worlds": the solution quality of complex search algorithms (MCTS) with the inference speed required for real-time, on-board execution.
• Scalability: The approach scales effectively to high-dimensional decision spaces where classical methods fail to capture long-term dependencies (like refueling cycles).
• Future Impact: This work paves the way for autonomous ADR missions, suggesting that future spacecraft can independently plan complex multi-target removal sequences while adhering to strict safety and resource constraints.

Future Work suggested by the authors includes integrating more detailed dynamical models (e.g., $J_2$ perturbations), transfer learning for adapting to changing debris fields, and further on-board validation.