⚡ electrical engineering

Safe Deep Reinforcement Learning for Spacecraft Reorientation with Pointing Keep-Out Constraint

This paper proposes a safe deep reinforcement learning framework for spacecraft reorientation that combines a novel state representation, Soft Actor-Critic training with curriculum learning, and a Control Barrier Function-based safety filter to guarantee adherence to pointing keep-out constraints, demonstrating that such a filter is essential for safety where reward shaping alone is insufficient.

Original authors: Juntang Yang, Mohamed Khalil Ben-Larbi

Published 2026-05-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Juntang Yang, Mohamed Khalil Ben-Larbi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are piloting a very delicate, high-tech camera mounted on a spinning satellite. Your job is to turn the satellite so the camera points at a specific star. However, there is a strict rule: the camera must never look at the Sun. If it does, the sensor will be blinded or damaged.

This is the problem this paper solves: How do you teach a computer to spin a satellite to a new target without accidentally pointing its "eye" at the Sun?

Here is how the authors approached it, broken down into simple concepts:

1. The Problem with "Just Learning"

Usually, when we teach computers to do complex tasks, we use a method called Deep Reinforcement Learning (DRL). Think of this like training a dog. You give the dog a treat (a reward) when it does something right and a scold (a penalty) when it does something wrong. Eventually, the dog learns the best way to get treats.

In this case, the "dog" is the AI agent, the "treat" is reaching the target star, and the "scold" is getting too close to the Sun.

The Catch: The paper found that just giving the AI "scolds" for getting close to the Sun isn't enough. Sometimes, the AI gets greedy for the "treat" (reaching the target fast) and takes a risky shortcut that accidentally points the camera at the Sun. It's like a student who studies hard but cheats on a test because they are desperate to pass; they might pass, but they broke the rules.

2. The New "Training Manual" (State Space & Rewards)

To help the AI learn better, the authors designed a new way for the computer to "see" the situation.

The Map: Instead of just showing the AI where it is, they gave it a special map that clearly highlights the "Sun Zone" (the forbidden area) and the direction it needs to move to stay safe.
The Scorecard: They created a scoring system that heavily penalizes getting close to the Sun, even if the AI is making progress toward the target. This encourages the AI to learn safe paths from the very beginning.

They also used a technique called Curriculum Learning. Imagine teaching a child to ride a bike. You don't start them on a busy highway. You start in an empty driveway, then a quiet street, then a busier road.

Phase 1: The AI learned to spin the satellite without any Sun constraints (just learning to turn).
Phase 2: Once it was good at turning, they added the "Sun" constraint and let it practice avoiding it.

3. The "Safety Bouncer" (The Safety Filter)

Even with a good teacher and a good scorecard, the paper admits that an AI trained this way might still make a mistake once in a while (about 2.6% of the time in their tests).

To fix this, they added a Safety Filter. Think of this as a strict bouncer at a club.

The AI (the guest) suggests a move (e.g., "Turn left quickly!").
The Bouncer (the Safety Filter) checks the move against the rules.
If the move is safe, the Bouncer says, "Go ahead."
If the move looks like it might point the camera at the Sun, the Bouncer overrules the AI and changes the move to a safe one before the satellite actually does it.

This filter uses a mathematical tool called a Control Barrier Function (CBF). In simple terms, it's a mathematical "force field" that physically prevents the satellite from crossing the invisible line into the Sun zone.

4. The Results

The authors ran 10,000 simulations to test their idea.

Without the Bouncer: The AI was fast and usually got the job done, but it broke the "no Sun" rule about 2.6% of the time.
With the Bouncer: The AI still got the job done, but 0% of the time did it break the rule. The safety filter caught every single dangerous move and corrected it.

However, the authors noted a small trade-off: The "bouncer" sometimes made the satellite turn a little slower or take a slightly longer path to be absolutely sure it was safe. Also, in a tiny number of cases, the AI got confused and couldn't finish the turn at all, suggesting it needs a bit more training.

Summary

The paper presents a two-part solution for safely turning spacecraft:

Train the AI well using a smart map and a strict scoring system.
Put a safety net (the filter) in front of the AI to catch any mistakes before they happen.

The result is a spacecraft that can turn itself to look at stars without ever accidentally blinding its own camera on the Sun.

Technical Summary: Safe Deep Reinforcement Learning for Spacecraft Reorientation with Pointing Keep-Out Constraint

Problem Statement
Spacecraft reorientation maneuvers often require strict adherence to pointing constraints to prevent sensitive onboard instruments (e.g., optical cameras) from exposure to bright celestial objects like the Sun. Traditional approaches to constrained attitude control, such as attitude planning and nonlinear Model Predictive Control (MPC), are computationally intensive and challenging for real-time onboard application. Conversely, Artificial Potential Field (APF) methods, while efficient, frequently suffer from local minima. While Deep Reinforcement Learning (DRL) offers a computationally efficient alternative for continuous control, its application to safety-critical, constrained scenarios remains limited. A primary challenge is the lack of formal safety guarantees; standard DRL agents trained with reward shaping alone may fail to strictly adhere to constraints during deployment.

Methodology
The paper proposes a Safe Reinforcement Learning (Safe RL) framework for spacecraft reorientation with a single pointing keep-out zone. The methodology integrates the Soft Actor-Critic (SAC) algorithm with a Control Barrier Function (CBF)-based safety filter.

State Space and Reward Design: A novel state space representation is designed to include a compact and explicit representation of the attitude constraint zone. The state vector includes the relative attitude, angular rate, the payload boresight vector, the angle between the boresight and the avoidance direction, the safety margin angle, and the relative avoidance direction vector. The reward function combines a base reward for attitude convergence and control effort minimization with a specific penalty function ( $P_{f-zone}$ ) that increases as the spacecraft approaches the keep-out zone boundary.
Training Strategy: The agent is trained using a curriculum learning approach in two phases. Phase 1 trains a baseline policy for unconstrained attitude regulation with increasing initial deviation angles. Phase 2 fine-tunes this policy with the keep-out zone penalty active, generating constraints that intersect the agent's likely shortest-path rotation trajectories.
Safety Filter: To guarantee constraint compliance during deployment, a CBF-based safety filter is interposed between the RL agent and the environment. The filter solves a quadratic program (QP) at each time step to find the control torque closest to the agent's proposed action while satisfying the control limits and the keep-out zone constraint. The safe set is defined using a robust inner constraint set that accounts for sampling time and potential disturbances.

Key Results
Extensive simulations, including a Monte Carlo analysis of 10,000 scenarios, were conducted to evaluate the proposed framework:

Standard RL Limitations: When using a standard RL framework (reward shaping only), the agent successfully reached the target in approximately 97% of cases. However, 2.66% of maneuvers resulted in violations of the keep-out zone, demonstrating that reward shaping alone cannot guarantee safety.
Safe RL Performance: When the CBF-based safety filter was integrated, the rate of constraint violations dropped to 0%. The mean reward and settling time remained comparable to the standard RL approach (a slight reduction in reward and a minor increase in settling time), while control effort was slightly reduced.
Parameter Sensitivity: The parameter $\mu$ in the CBF formulation significantly influences performance. While $\mu=0.0025$ provided optimal safety and performance, a smaller, unsuitably set $\mu$ ($0.0001$) resulted in degraded performance (higher settling time and more non-settled cases) compared to the standard RL approach, despite still avoiding violations.
Non-Settled Cases: A small percentage of cases (0.22% with the filter) failed to settle. Analysis indicated these were caused by the agent's policy (resulting in biases or limit cycles) rather than the safety filter, suggesting a need for further agent training.

Significance and Claims
The paper claims that the proposed framework effectively addresses the safety gap in applying DRL to spacecraft attitude control. The primary contribution is the demonstration that while reward shaping improves constraint awareness, it is insufficient for guaranteeing safety in critical maneuvers. In contrast, the integration of a CBF-based safety filter ensures strict compliance with pointing constraints without significantly compromising control performance. The authors conclude that their approach provides a viable, real-time solution for constrained spacecraft reorientation, though they note that future work is required to address non-settled behaviors and extend the algorithm to handle multiple keep-out zones using Long Short-Term Memory (LSTM) networks.

1. The Problem with "Just Learning"

2. The New "Training Manual" (State Space & Rewards)

3. The "Safety Bouncer" (The Safety Filter)

4. The Results

Summary

More like this