Autonomous Planning In-space Assembly Reinforcement-learning free-flYer (APIARY) International Space Station Astrobee Testing

1. Problem Statement

Current robotic spacecraft operations rely heavily on classical control methods (e.g., PID, Proportional-Derivative) and significant human ground intervention. While effective for stabilization, these methods struggle with the dynamic complexities required for advanced autonomous maneuvers, such as docking, servicing, and assembling large structures in space.

• Limitations of Classical Control: Generating and verifying complex 6-DOF (Degrees of Freedom) waypoints for tasks like satellite servicing can take hours to days. Furthermore, spacecraft computers are often significantly slower than terrestrial counterparts, necessitating mathematically verifiable but often less flexible control algorithms.
• The Sim-to-Real Gap: While Reinforcement Learning (RL) offers potential for adaptable and optimized control, applying it to space robotics is hindered by the "Sim-to-Real" gap. Discrepancies between simulated physics (mass, friction, dynamics) and the real world often lead to degraded performance, requiring extensive real-world fine-tuning which is difficult and risky in space.
• The Challenge: The paper addresses the feasibility of deploying an RL-based control policy for a free-flying robot in a zero-gravity environment without extensive human intervention, bridging the gap between simulation training and on-orbit execution.

2. Methodology

The APIARY experiment utilized a pipeline moving from high-fidelity simulation to ground testing and finally to flight validation on the International Space Station (ISS).

• Platform: The experiment used the NASA Astrobee robot, a cube-shaped (12.5 inches), free-flying robot on the ISS that uses electric fans for 6-DOF propulsion and cameras for localization.
• Simulation Environment (Training):
  • Tool: NVIDIA Isaac Lab within the NVIDIA Omniverse physics simulator.
  • Approach: An actor-critic Proximal Policy Optimization (PPO) network was trained to control the Astrobee.
  • Domain Randomization: To enhance robustness and bridge the Sim-to-Real gap, the training environment randomized goal poses, mass distributions, and physics parameters. This allowed the policy to learn control strategies that generalize across varying conditions.
  • Task: The agent learned to move from a start pose to a desired end pose by minimizing position and orientation errors while penalizing high velocities.
• Control Architecture:
  • The standard Astrobee Guidance, Navigation, and Control (GNC) software was modified. The RL policy bypassed the internal control calculations to directly output force and torque (wrench) commands.
  • These commands were executed by the Astrobee's onboard fan-based thrusters via the Force Allocation Module (FAM).
• Validation Pipeline:
  1. Simulation: Tested in NVIDIA Omniverse (zero-G) and NASA's Gazebo-based Astrobee simulator (ISS and Granite Lab environments).
  2. Ground Testing: Validated on the Granite Lab at NASA Ames Research Center, a 2-meter granite table with air bearings simulating frictionless 3-DOF motion.
  3. Flight Testing: Deployed on the Bumble Astrobee robot aboard the ISS for 6-DOF zero-G testing.

3. Key Contributions

• First RL Control of a Free-Flyer in Space: To the authors' knowledge, this is the first instance of an RL policy controlling a free-flying robot in a zero-gravity environment on the ISS.
• Successful Sim-to-Real Transfer: The paper demonstrates a viable workflow for training RL policies in a randomized simulation environment (Isaac Lab) and deploying them directly to flight hardware without retraining, effectively bridging the Sim-to-Real gap for space robotics.
• Robust Safety Mechanism: The system implemented a critical fail-safe. If the RL policy caused positional errors to exceed defined thresholds, the system automatically switched back to the classical baseline controller to maintain a stable pose, ensuring mission safety.
• SWAP-Constrained Deployment: The experiment proved that complex RL policies can run on low Size, Weight, and Power (SWAP) flight computers (the Astrobee's flight computer) alongside existing mission-critical software.

4. Experimental Results

The study evaluated the RL policy against the standard NASA baseline controller across three environments:

• Simulation (Gazebo/Isaac):
  • In 1G Granite Lab simulations, the RL policy successfully executed 0.5m translations, though it exhibited slightly more off-axis drift (Y-axis) and orientation error compared to the baseline.
  • In 0G ISS simulations, the RL policy successfully undocked the robot but showed a tendency to overshoot the target position and had higher orientation errors than the baseline.
• Ground Hardware (Granite Lab):
  • The RL policy successfully commanded a 0.3m translation in 1G. While it exhibited drift consistent with simulation, the motion remained within the robot's defined success thresholds.
• Flight Hardware (ISS):
  • Maneuvers: Over a 7-minute window, the Astrobee completed 8 maneuvers, including undocking, rotations, translations, and docking attempts.
  • Performance: The RL policy successfully executed the undock and translation maneuvers within the success criteria, though with slightly higher error margins than the idealized baseline.
  • Safety Event: During the first docking attempt, a localization issue caused the RL policy to exceed error thresholds. The system's safety logic successfully triggered, reverting control to the baseline controller to prevent collision.
  • Success: A second docking attempt (initiated by a human operator) was successful, even during a loss of signal, demonstrating the system's resilience.

5. Significance and Future Outlook

• Paradigm Shift in Space Autonomy: This work validates that RL can replace or augment classical controllers for space robotics, enabling rapid development and deployment of tailored behaviors (in minutes to hours) rather than the months required for traditional algorithm verification.
• Scalability: The ability to train on 100,000s of parallel simulations allows RL to handle environmental variations that are impossible to replicate in a single deterministic simulation.
• Future Applications: The success of APIARY paves the way for more complex In-Space Assembly and Manufacturing (ISAM) tasks, including:
  • Rendezvous and Proximity Operations (RPOD).
  • Grasping and manipulation using robotic arms.
  • Adapting to unknown inertial parameters (e.g., manipulating unknown objects).
  • Fault recovery (e.g., regaining control after thruster failure).
• Conclusion: The experiment proves the concept of using RL for space robotics, establishing a foundation for future missions requiring high levels of autonomy in dynamic, unstructured space environments.