Evaluating Robustness and Adaptability in Learning-Based Mission Planning for Active Debris Removal

This paper evaluates three mission planning approaches for active debris removal, demonstrating that while domain-randomized reinforcement learning offers a robust balance between speed and adaptability, Monte Carlo Tree Search provides superior constraint handling at the cost of significantly higher computational time, highlighting a critical trade-off between learned policy efficiency and search-based flexibility.

The Gist

Imagine you are the captain of a spaceship tasked with cleaning up a messy room filled with floating trash (space debris). You have a limited amount of fuel (like a gas tank) and a strict deadline (like a curfew). Your job is to visit as many pieces of trash as possible, stop at a gas station to refill your tank if needed, and get back on time.

This paper is a race between three different "brains" trying to figure out the best route to clean the room. The researchers tested how well each brain works when the rules of the game stay the same, and how well they handle it when the rules suddenly change (like running out of fuel faster than expected or having less time).

Here is how the three competitors stack up, using simple analogies:

The Three Competitors

1. The "Specialist" (Nominal PPO)

• What it is: This is a robot trained specifically for one perfect scenario. It's like a student who memorized the answers to a specific practice test.
• How it works: It learns by trial and error until it knows the exact best moves for a standard mission (7 days, full fuel).
• The Catch: It's incredibly fast. It makes decisions in a blink of an eye. However, if you change the test questions (e.g., "Now you only have half the fuel"), it panics. It tries to use the same memorized moves, runs out of gas, and fails miserably. It's great when things go exactly as planned, but brittle when things go wrong.

2. The "Generalist" (Domain-Randomized PPO)

• What it is: This is a robot trained on many different scenarios. It's like a student who didn't just memorize one test, but practiced with random fuel levels and random time limits every day.
• How it works: It learned to be flexible. It knows how to be aggressive when it has lots of fuel and how to be conservative when it's low on gas.
• The Catch: It's still very fast (just like the Specialist). When the rules change, it adapts much better than the Specialist. It doesn't perform quite as perfectly as the Specialist does in the perfect scenario, but it doesn't crash when the scenario gets tough. It's a good middle ground.

3. The "Calculator" (MCTS)

• What it is: This isn't a pre-trained robot; it's a super-computer that thinks through every possible future before making a single move. It's like a chess grandmaster who simulates 200 different games in their head before moving a piece.
• How it works: At every step, it asks, "If I go here, what happens next? If I go there, what happens then?" It constantly replans based on the current situation.
• The Catch: It is the smartest at handling surprises. If you cut the fuel in half, it instantly recalculates the best path and still gets the job done. However, it is slow. While the other two make decisions in less than a second, this one takes over four minutes to think through a single move. In a real emergency on a spaceship, waiting four minutes to decide where to turn might be too long.

The Race Results

The researchers ran 300 tests to see who won under different conditions:

• The "Perfect Day" Test (Normal Fuel & Time):
  The Specialist won by a tiny margin. It knew the route perfectly. The Generalist was almost as good, and the Calculator was slightly behind but still did a great job.

• The "Short on Time" Test (3 Days instead of 7):
  Everyone struggled because the clock was ticking faster. The Generalist adapted best and cleaned up the most trash. The Specialist got confused and cleaned up less. The Calculator did well but was slightly slower to react than the Generalist.

• The "Low Fuel" Test (1/3 of the fuel):
  This was the big shocker. The Specialist crashed hard; it tried to fly its usual route, ran out of gas immediately, and barely cleaned anything. The Generalist did much better, cleaning up more than double what the Specialist did, but it still couldn't beat the Calculator. The Calculator was the clear winner here because it could instantly see that it needed to be very careful with its fuel and changed its plan on the fly.

The Big Lesson

The paper concludes that there is a trade-off between speed and flexibility:

• If you know the rules won't change, use the Specialist. It's fast and efficient.
• If you think the rules might change a little, use the Generalist. It's a smart compromise that is fast but can handle some surprises.
• If the rules are chaotic and you need the absolute best plan no matter what, use the Calculator. But be warned: it takes a long time to think.

The authors suggest that the future of space cleanup might involve mixing these approaches: training robots to be "Generalists" (like the second robot) so they are smart and fast, but maybe giving them a little bit of the "Calculator's" ability to double-check their plans when things get really crazy.

Technical Summary

Technical Summary: Evaluating Robustness and Adaptability in Learning-Based Mission Planning for Active Debris Removal

Problem Statement
Active Debris Removal (ADR) in Low Earth Orbit (LEO) requires autonomous spacecraft to sequentially rendezvous with multiple debris objects under strict operational constraints. The core challenge is a constrained sequential decision-making problem where the spacecraft must maximize the number of debris removed while adhering to hard limits on total mission duration and cumulative $\Delta v$ (fuel budget). The spacecraft operates in a 700–800 km altitude band, utilizing co-elliptic Hohmann transfers and terminal safety ellipse maneuvers. A critical complication is the potential for "distributional shift," where deployment conditions (e.g., reduced fuel or shortened mission time) differ from the parameters assumed during the training of learned policies. The problem is further complicated by the need for mid-mission refueling, which resets the $\Delta v$ budget but consumes mission time.

Methodology
The authors evaluate and compare three distinct planning strategies within a high-fidelity orbital simulation environment (SpaceDebrisStressTestEnv) involving 50 randomly generated debris targets per episode. All methods utilize action masking to enforce feasibility constraints (remaining $\Delta v$, time, and visitation status).

1. Nominal Masked PPO: A Proximal Policy Optimization (PPO) agent trained on fixed, nominal mission parameters (7-day duration, 3 km/s $\Delta v$). It uses a Multi-Layer Perceptron (MLP) with two hidden layers (256 units) and is trained for 1 million timesteps.
2. Domain-Randomized Masked PPO: An identical PPO architecture trained for 5.5 million timesteps, but with mission parameters (duration and $\Delta v$ budget) randomized at the start of each episode. This approach aims to improve robustness against distributional shifts.
3. Plain Monte Carlo Tree Search (MCTS): An online search-based baseline using the Upper Confidence bounds applied to Trees (UCT) algorithm. It performs 200 simulations per decision step with a uniform random policy for rollouts. It replans dynamically at every step without prior training.

Key Results
The methods were tested across 300 cases (100 per scenario) covering nominal conditions, reduced fuel (1 km/s), and reduced mission time (3 days).

• Nominal Performance: Under training-matched conditions, the Nominal PPO achieved the highest average debris removal (29.1 objects), slightly outperforming the Domain-Randomized PPO (28.2) and MCTS (27.1). Both PPO variants demonstrated sub-second inference times.
• Reduced Mission Time: When the duration was cut to 3 days, the Domain-Randomized PPO showed the best adaptability (14.1 objects), outperforming both the Nominal PPO (12.6) and MCTS (11.9).
• Reduced Fuel ($\Delta v$): Under severe fuel constraints (1 km/s), the Nominal PPO degraded sharply, removing only 3.2 objects on average due to early fuel depletion. The Domain-Randomized PPO improved significantly (8.1 objects) but still lagged behind MCTS (15.0 objects).
• Computational Cost: MCTS incurred a massive computational penalty, averaging over 4 minutes per test case due to repeated environment cloning and rollouts. In contrast, both PPO variants required less than 1 second per episode.

Significance and Claims
The paper posits that there is a fundamental trade-off between the speed of learned policies and the adaptability of search-based methods in ADR mission planning.

• Learned Policies: Offer rapid inference suitable for real-time onboard execution but are brittle when deployment conditions deviate from training distributions.
• Search-Based Methods (MCTS): Provide superior adaptability to changing constraints through online replanning but are computationally prohibitive for real-time execution on resource-limited hardware.
• Domain Randomization: The study demonstrates that training with diverse mission parameters partially bridges this gap. While it incurs a moderate loss in nominal performance and requires significantly more training steps (5.5M vs. 1M), it yields a policy with significantly improved robustness to constraint changes compared to a nominal policy.

The authors conclude that while no single method currently offers both optimal speed and adaptability, combining training-time diversity (domain randomization) with online planning strategies represents a promising path for future resilient ADR systems. They suggest that hybrid frameworks, such as those blending neural policy predictions with tree search (e.g., AlphaZero or MuZero), could be a viable direction for future work to achieve both efficiency and adaptability.