CALOS: Control-Affine Lyapunov On-manifold Safety Layer for Safe Deep Reinforcement Learning for Quadrotors

Technical Summary: CALOS – Control-Affine Lyapunov On-manifold Safety Layer

Problem Statement
Deep Reinforcement Learning (DRL) has demonstrated significant capability in controlling quadrotors, yet learned policies lack formal guarantees regarding safety constraints during training or deployment. Existing safe DRL approaches face a persistent tension: Constrained Markov Decision Processes (CMDPs) can destabilize learning, while runtime filters (shielding or control barrier functions) often degrade gradient signals if they correct actions too aggressively. Furthermore, current architectural limitations prevent the joint optimization of multiple safety certificates. Methods like ATACOM project actions onto constraint manifolds but lack energy-based certificates for rotational dissipation, while Lyapunov-based methods often treat constraints independently, risking the violation of one constraint while satisfying another. Sequential composition of these methods fails to guarantee joint feasibility, particularly when large tracking errors require maximum control authority, which sequential scaling often nullifies.

Methodology
The paper proposes CALOS (Control-Affine Lyapunov On-manifold Safety), a runtime safety layer that operates transparently on top of a standard DRL policy (specifically Proximal Policy Optimization, PPO) without modifying the underlying learning algorithm. CALOS unifies attitude-tilt constraints and a Lyapunov stability condition into a single Quadratic Program (QP) to compute the minimum-norm correction to the nominal torque output.

1. Tilt Constraints (Predictive Tilt Projection):
   Attitude is represented by the gravity vector in the body frame ($g_b$) to avoid Euler-angle singularities. Using a symplectic Euler discretization, the future gravity vector is modeled as an affine function of the control torque. This formulation allows the imposition of roll and pitch limits ($\phi_{max} = \theta_{max} = 60^\circ$) as four linear inequalities ($A_{PTP}\tau \le b_{PTP}$).

2. Lyapunov Constraint:
   A Lyapunov candidate function $V(x)$ is defined based on tilt error and angular velocity. The layer enforces a decay condition $\dot{V} \le -cV$, which, due to the control-affine nature of rotational dynamics, translates into a single linear inequality ($A_L\tau \le b_L$). This ensures rotational energy dissipation.

3. Unified Formulation:
   Unlike sequential approaches that apply tilt projection followed by Lyapunov scaling (which can lead to torque nullification when errors are large), CALOS stacks the five linear constraints (four tilt, one Lyapunov) into a single system. The safety layer solves:
   $$\tau^\star = \arg \min_{\tau \in \mathbb{R}^3} \frac{1}{2} \|\tau - \tau_0\|^2 \quad \text{s.t.} \quad A(x)\tau \le b(x)$$
   where $\tau_0$ is the nominal torque from the policy.

4. Solvers:

   • CALOS-P: A projected approximation using a damped pseudo-inverse correction. It enforces all constraints jointly without guaranteeing the exact minimum-norm solution, prioritizing speed for large-scale parallel training.
   • CALOS-QP: An exact solver that exploits the three-dimensional torque space. It enumerates all possible active sets (26 candidates) to find the exact minimum-norm solution satisfying the Karush–Kuhn–Tucker (KKT) conditions. If no feasible solution exists, the system falls back to the uncorrected policy action with actuator clamping.

Key Contributions

• Unified Safety Layer: The first runtime layer to jointly optimize tilt constraints and Lyapunov stability in a single QP step, avoiding the feasibility issues of sequential composition.
• Real-Time Scalability: The exact solver (CALOS-QP) is computationally efficient enough to run across thousands of parallel simulation environments, a requirement for modern massively parallel DRL training.
• Zero Violations on Training Trajectories: The method enforces constraints strictly during training, preventing the agent from exploring unsafe regions of the state space.

Experimental Results
Evaluated in NVIDIA Isaac Lab on trajectory-tracking tasks, CALOS was compared against unconstrained PPO, standalone tilt projection (PTP), standalone Lyapunov scaling, and a sequential cascade of both.

• Tracking Performance: CALOS-P and CALOS-QP reduced lateral tracking error by 55–60% relative to the unconstrained PPO baseline on training trajectories. On unseen trajectories with large initial position offsets, CALOS variants maintained errors below 0.08 m, whereas Lyapunov and Cascade methods failed to track due to torque nullification.
• Safety Metrics: On the training trajectory, CALOS-QP achieved zero attitude-constraint violations. Under extreme initial offsets, violations were limited to at most 3 consecutive steps (CALOS-P) or 9 steps (CALOS-QP), compared to up to 27 steps for sequential methods.
• Convergence and Data Efficiency: By restricting exploration to safe regions, CALOS accelerated training convergence.
• Internalized Behavior: Policies trained with CALOS retained safer and more accurate behavior even when the safety layer was disabled at test time, showing 55–74% lower lateral error than PPO-trained policies. This indicates the policy successfully internalized the safety constraints.

Significance
The paper claims that CALOS resolves the trade-off between safety enforcement and learning efficiency. By formulating safety as a single, low-dimensional QP, it ensures that the gradient signal is not degraded by aggressive, sequential corrections. The method allows the policy to learn optimal behaviors within the safe manifold, resulting in policies that are inherently safer and more data-efficient without sacrificing tracking performance. The authors note that the QP feasible set remained non-empty across all tests, confirming the robustness of the joint formulation.

Future Work
The authors identify three directions for future research:

1. Developing learned, task-aware Lyapunov certificates to avoid unnecessary interventions when the drone tracks a non-zero but feasible reference attitude.
2. Extending the framework to non-control-affine dynamics (e.g., models including rotor-speed dynamics or blade-flapping effects).
3. Investigating sim-to-real transfer using domain randomization and data-driven safety filters to handle model uncertainty on physical hardware.