💻 computer science

SCoCaT: Success Conditioned Constrained Reinforcement Learning for Spacecraft Docking

This paper identifies and resolves a "feasibility collapse" in termination-based constrained reinforcement learning for spacecraft docking, where agents avoid goal regions to maintain safety, by introducing a dense success signal via an auxiliary value critic that ensures high task completion rates without compromising safety constraints.

Original authors: Aman Arora, Ricard Marsal I Castan, Matteo El-Hariry, Miguel Olivares-Mendez

Published 2026-09-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Aman Arora, Ricard Marsal I Castan, Matteo El-Hariry, Miguel Olivares-Mendez

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Spacecraft docking is one of the most delicate maneuvers in modern engineering. It requires a vehicle to travel through the vacuum of space, navigate toward a moving target, and gently lock onto it without crashing. This task is not just about reaching a destination; it is about doing so while obeying a strict set of safety rules. The spacecraft must not move too fast, must not spin out of control, and must never collide with the target or stray outside its designated flight path. If these rules are broken, the mission fails, and the hardware is lost. For decades, engineers have relied on classical control systems to manage these tasks, but these systems struggle when faced with complex, overlapping safety limits. In recent years, researchers have turned to a type of artificial intelligence called reinforcement learning, where a computer program learns by trial and error. However, standard learning methods are often too reckless for space; they might find a way to reach the goal that involves breaking safety rules, or they might learn to hover just outside the danger zone to avoid punishment, never actually completing the job.

A team of researchers at the University of Luxembourg has developed a new method to solve this specific problem, allowing AI to learn how to dock spacecraft safely and successfully. Their work focuses on a known failure mode in safety-focused learning algorithms. In these systems, the computer is taught that breaking a safety rule is so costly that it effectively ends the learning episode early. This works well for many tasks, but for docking, it creates a paradox. The area where the spacecraft must finally arrive to succeed is also the area where the safety rules are tightest. Because the penalty for entering this zone is so high, the learning algorithm decides it is safer to hover just outside the target, where it can stay alive but never finish the mission. The researchers call this "feasibility collapse," a state where the AI is perfectly safe but completely useless.

To fix this, the team introduced a new approach called Success-Conditioned Constrained Reinforcement Learning. They realized that the AI needed a way to understand that reaching the goal was the primary objective, separate from the fear of breaking safety rules. They added a second layer of feedback to the learning process. While the first layer still punished the spacecraft for violating safety limits, the second layer gave a constant, positive signal whenever the spacecraft was in the correct position and orientation, regardless of whether it had technically "won" the episode yet. This created a dual motivation: the AI learned to avoid the penalties that would stop its progress, but it was also pulled forward by the reward of being in the right place. This simple addition broke the deadlock that caused the AI to hover in place.

The researchers tested this method on two different types of spacecraft simulators. The first was a floating platform in their laboratory that mimics the movement of a satellite in zero gravity, using air bearings to glide on a cushion of air. The second was a digital model of a 6U CubeSat, a small satellite roughly the size of a shoebox, which has a more complex, six-dimensional movement pattern. In the simulations, the standard safety-focused method failed to dock the spacecraft in nearly every attempt, leaving it stuck just outside the target zone. The new method, however, allowed the spacecraft to enter the docking corridor and hold its position successfully. On the floating platform, the new approach achieved a success rate of nearly 100 percent while maintaining a safety compliance rate of over 98 percent. This was a massive improvement over the previous method, which had a success rate of less than one percent.

The team did not stop at computer simulations. They took the software trained in the digital world and deployed it directly onto their physical floating platform without any further adjustments. This "zero-shot" transfer means the AI learned in one environment and worked perfectly in another, a difficult feat for robotic systems. The physical tests confirmed that the spacecraft could approach the target, slow down, and hold its position within a tolerance of 10 millimeters and 0.1 radians of rotation. While the standard safety-only method managed to avoid collisions, it never entered the target zone. The new method entered the zone and stayed there, proving that it is possible to be both safe and successful.

The researchers also examined why the old method failed so dramatically. They showed mathematically that the problem was not a flaw in the reward system itself, but a structural issue in how the safety penalties interacted with the goal. When the penalty for entering the goal zone is high, the AI calculates that staying just outside is the most logical choice to maximize its survival. By separating the signal for "being safe" from the signal for "being successful," the new method allows the AI to take the necessary risks to finish the job without ignoring the safety constraints. The results suggest that for critical tasks like spacecraft docking, where failure is irreversible, AI systems need a clear, distinct signal that tells them when they have succeeded, independent of the fear of failure. This approach offers a path forward for deploying autonomous robots in environments where safety and precision are equally non-negotiable.

Technical Summary: SCoCaT – Success Conditioned Constrained Reinforcement Learning for Spacecraft Docking

1. Problem Statement

The paper addresses a critical failure mode in terminal-navigation tasks (e.g., spacecraft docking, precision landing) when using termination-based Constrained Reinforcement Learning (CRL). Specifically, it targets the Constraints as Terminations (CaT) framework, which enforces safety by stochastically terminating an episode upon constraint violation. While CaT is attractive for safety-critical deployments due to its simplicity and lack of online optimization, the authors identify a structural pathology termed "feasibility collapse."

In terminal-navigation scenarios, the goal lies within a corridor where safety constraints (e.g., velocity limits) become active. Under the standard CaT objective, the survival-weighted return suppresses rewards precisely where the agent must enter to succeed. Consequently, the agent learns a hovering policy: it stays just outside the goal corridor to maximize survival probability and accumulated reward, avoiding the constraint-triggering zone entirely. This results in near-perfect constraint compliance but near-zero task completion.

2. Methodology

2.1 Formalization of Feasibility Collapse

The authors formalize this failure mode using a simplified Markov Decision Process (MDP) with three states: a far state ( $s_{far}$ ), a hover state just outside the goal ( $s_{\partial}$ ), and the goal state ( $s_G$ ).

Proposition 1 demonstrates that under the CaT objective, if entering the goal triggers a non-zero termination probability ( $\delta > 0$ ) while hovering incurs none, the hover policy becomes a strict local optimum. The agent forfeits the remaining horizon to avoid the "cost" of constraint activation, even if the goal offers high reward.

2.2 Proposed Solution: SCoCaT

To resolve this, the authors propose Success-Conditioned Constraints as Terminations (SCoCaT). This method augments the CaT framework with a decoupled success signal that is structurally immune to the survival damping of constraint violations.

Dual Advantage Streams: SCoCaT maintains two independent value critics:
1. $V_r$ (Reward Critic): Trained on the CaT-modified rewards, where rewards are scaled by the survival probability $(1-\delta_t)$ . This stream enforces safety constraints.
2. $V_s$ (Success Critic): Trained on a dense, per-step success indicator ( $I_{succ,t}$ ) using environment-only terminations (time-limits or natural episode ends), explicitly excluding stochastic CaT terminations.
Success Indicator: The signal $I_{succ,t}$ is a binary (or continuous) gate that activates only when the vehicle satisfies all docking tolerances (position, orientation, velocity, and angular rate) simultaneously.
Advantage Scalarization: The policy gradient utilizes a single scalarized advantage, $A_{tot} = \text{standardize}(A_r + A_s)$ $A_{t o t} = standardize (A_{r} + A_{s})$ .
- $A_r$ reflects the survival-weighted return.
- $A_s$ provides a dense gradient toward the goal corridor, independent of the constraint-induced termination probability.
Mechanism: By bootstrapping $V_s$ on environment-only terminations, the success advantage remains positive and strong within the goal corridor, counteracting the negative advantage generated by the survival penalty in $A_r$ . This breaks the local optimum of hovering.

2.3 Implementation

The method is implemented using Proximal Policy Optimization (PPO). The architecture adds only a single auxiliary critic ( $V_s$ ) to standard off-the-shelf RL algorithms. The success tolerances are fixed by operational specifications rather than learned hyperparameters.

3. Key Contributions

Formalization of Feasibility Collapse: The paper derives the theoretical threshold at which a hovering policy becomes strictly preferred over goal entry in termination-based CRL, identifying the constraint-termination channel as the root cause rather than reward shaping.
SCoCaT Algorithm: A minimal extension to CaT that resolves feasibility collapse by introducing a dense per-step success signal decoupled from the survival weighting.
Empirical Validation: The approach is validated on two distinct spacecraft platforms:
- A 3-DoF Floating Platform (air-bearing testbed) used for zero-shot sim-to-real transfer.
- A 6-DoF 6U CubeSat simulator representing the mass and degree-of-freedom envelope of operational proximity operations.

4. Experimental Results

4.1 Simulation Results

Floating Platform (3-DoF):
- CaT: Achieved 99.4% constraint compliance but only 0.4% Task Completion Rate (TCR), confirming feasibility collapse.
- SCoCaT: Achieved 100% TCR with 98.8% constraint compliance, representing a 250 $\times$ improvement in task engagement without safety regression.
- Unconstrained PPO: Achieved high corridor entry (90.8%) and a 49.8% TCR, but failed at controlled docking: without velocity-constraint feedback, the agent passed through the goal at speed without braking. This resulted in significantly higher position errors and docking-velocity violations (70 $\times$ higher than SCoCaT).
6U CubeSat (6-DoF):
- CaT: Entered the goal corridor in only 3.7% of episodes and never completed the task.
- SCoCaT: Achieved a 61.3% declared success rate (meeting terminal velocity and pose criteria) with 94.5% constraint compliance. This represents 66% of the unconstrained PPO's success rate while maintaining near-CaT levels of safety.

4.2 Sim-to-Real Transfer

The authors performed zero-shot transfer of the SCoCaT policy to the physical 3-DoF air-bearing platform.

SCoCaT: Successfully docked in 3 out of 4 real-world runs.
CaT: Failed to dock in 0 out of 4 runs (hovering outside the corridor).
PPO: While the raw "Success Rate" (SR) metric recorded 4 out of 4 runs entering the goal corridor, the paper explicitly notes that PPO "fails at controlled docking." Without velocity-constraint feedback, the agent passed through the goal at speed without braking, failing to meet the controlled docking precision required for a successful mission, unlike SCoCaT which achieved controlled docking.
This demonstrates that the method generalizes beyond simulation to hardware with binary thrusters and discrete control frequencies.

4.3 Ablation Studies

Reward Density: SCoCaT works with both dense and sparse rewards, though dense rewards yield slightly better performance.
Success Signal: A binary success signal fails under sparse rewards; a continuous success signal recovers performance by providing smooth gradients.
Architecture: Adding the success signal as a simple bonus to the reward stream (single-critic) degrades constraint compliance. The dual-critic architecture is essential to preserve the integrity of the safety constraints while enabling task completion.

5. Significance and Claims

The paper claims that SCoCaT provides a principled resolution to the feasibility collapse inherent in termination-based CRL for terminal-navigation tasks. By decoupling the success objective from the safety termination mechanism, the method recovers high task completion rates while maintaining strict safety compliance.

The authors emphasize that:

The solution is minimal, requiring only an auxiliary critic and no complex online solvers or Lagrangian multipliers.
It is deployment-ready, suitable for resource-constrained platforms (SWaP budgets) where inference must be a single neural network forward pass.
It enables zero-shot sim-to-real transfer, a critical capability for space applications where physical testing is limited.

The paper concludes that while the approach significantly improves the safety-success trade-off, it does not claim to be a universal solution for all CRL problems, noting limitations such as the need for mission-specific tuning of constraint severity ( $p^i_{max}$ ) and the presence of sim-to-real gaps in angular velocity control due to hardware discretization.