Deep Reinforcement Learning for Spacecraft Attitude Control During Atmospheric Re-Entry

Technical Summary: Deep Reinforcement Learning for Spacecraft Attitude Control During Atmospheric Re-Entry

Problem Statement
The paper addresses the challenge of controlling the attitude of a hypersonic lifting-body spacecraft during atmospheric re-entry (from 93 km to 10 km altitude). This is a safety-critical tracking problem involving highly nonlinear dynamics, significant uncertainties, and varying environmental conditions (Mach number, dynamic pressure). Traditional approaches rely on gain-scheduled Proportional-Integral-Derivative (PID) controllers, which require extensive expert tuning and formal verification for specific corner cases. The authors investigate whether Deep Reinforcement Learning (RL) can offer a more adaptive, precise, and robust alternative, while addressing the "black box" nature of learned policies which complicates verification and validation (V&V) for space-grade software.

Methodology
The authors formalize the attitude control problem within a continuous, off-policy RL framework. They utilize a high-fidelity six-degree-of-freedom simulation model that includes aerodynamic forces, thruster dynamics, and sensor lags.

• Control Architectures: Three architectures are compared:
  1. Baseline: An industry-standard gain-scheduled PID controller with feedforward inverse dynamics.
  2. Pure RL: A model-free agent (MR.Q, TD7, TD3, SAC) that directly outputs flap deflection changes and thruster commands.
  3. Hybrid Controllers:
     • Additive Hybrid: The RL agent outputs a residual command added to the baseline PID output.
     • Gain-Scheduling Hybrid: The RL agent outputs scaling factors (in decibels) to adjust the gains of the baseline PID controller.
• Algorithms: The study focuses on state-of-the-art continuous control algorithms, specifically comparing TD3-based extensions (TD7, MR.Q) and SAC-based extensions. MR.Q (Model-based Representation learning with Q-learning) is highlighted for its use of model-based representation learning and loss-adjusted prioritized experience replay.
• Reward Function: A strictly positive reward is designed to avoid early termination incentives. It combines an exponential decay of the attitude tracking error (angle of attack, sideslip, and bank angle) with a control cost penalizing excessive thruster usage and flap rate changes.
• Generalization Strategies: To address out-of-distribution (OOD) generalization, the authors employ:
  • Dynamics Randomization (DR): Randomizing physical parameters (mass, inertia tensor, actuator bandwidth) and initial attitudes during training.
  • Task Scheduling: Comparing strategies like "Hard tasks first" (SMT) and "Active Multi-Task" (AMT) against uniform random sampling.

Key Contributions

1. Application of SOTA RL to Hypersonic Re-Entry: The authors apply advanced continuous control algorithms (specifically MR.Q) to a challenging spacecraft attitude tracking problem requiring gain scheduling. They demonstrate that MR.Q excels without task-specific tuning, outperforming traditional gain-scheduled PID controllers in nominal conditions by continuously adapting to environmental changes.
2. Hybrid Control for Robustness: The paper explores hybrid architectures to mitigate the opacity and stability concerns of pure RL. It finds that while pure RL can outperform the baseline nominally, hybrid controllers (particularly additive hybrids) offer a safer fallback mechanism. The baseline PID acts as a verified safety layer, with the RL agent enhancing performance or adapting to variations.
3. Training Strategy Evaluation: The study systematically compares dynamics randomization against task scheduling approaches. It determines that continuous uniform dynamics randomization is sufficient and effective for enforcing generalization, whereas complex task scheduling strategies (like SMT or AMT) did not yield superior results in this specific domain and introduced training instabilities.

Results

• Nominal Performance: Under nominal conditions, the MR.Q algorithm (both pure and additive hybrid) achieves higher cumulative returns and tighter tracking of the angle of attack compared to the industry-standard baseline. The error distribution for the angle of attack is narrower for the learned controller, though the baseline shows a higher concentration of errors near zero.
• Out-of-Distribution Generalization:
  • Policies trained only under nominal conditions fail to generalize well to variations in initial attitude and inertia tensor, often performing worse than the baseline.
  • Policies trained with Dynamics Randomization show significantly improved robustness. They maintain high success rates (>93%) across variations in mass, inertia, and actuator bandwidth where the baseline controller fails (particularly at low actuator bandwidths).
  • The Additive Hybrid controller trained with DR demonstrates the best balance, achieving the highest success rates (up to 94-99%) and reducing the 90th, 95th, and 98th percentile errors for aerodynamic angles compared to the baseline.
• Task Scheduling: Task scheduling methods (SMT, AMT) did not outperform simple uniform dynamics randomization. In some cases, focusing on "hard" tasks first (SMT) led to training failures (0% success rate).

Significance and Claims
The paper claims that deep reinforcement learning, specifically when combined with hybrid control architectures and trained via dynamics randomization, can achieve superior robustness and tracking performance compared to traditional gain-scheduled PID controllers in the operational envelope of hypersonic re-entry.

The authors emphasize that while pure RL offers high performance, the hybrid approach is the most promising candidate for flight-ready systems. This architecture provides inherent safety by retaining a formally verified baseline controller, allowing the RL component to be bounded or disabled via supervisor logic if the system approaches the edge of its validated domain. The study concludes that MR.Q is particularly effective due to its model-based representation and multi-step returns, and that simple dynamics randomization is a more reliable training strategy than complex task scheduling for ensuring generalization in this safety-critical application. The work aims to provide statistical evidence of reliability within a strictly defined operational envelope, bridging the gap between adaptive AI and the rigorous V&V requirements of spaceflight.