LC-SAC: Lyapunov-Constrained Soft Actor-Critic via Koopman Operator Theory for Trajectory Tracking and Stabilization

Technical Summary: LC-SAC: Lyapunov-Constrained Soft Actor-Critic via Koopman Operator Theory

Problem Statement
Reinforcement Learning (RL) has demonstrated success in complex sequential decision-making but faces significant hurdles when applied to safety-critical physical systems. Standard RL algorithms prioritize reward maximization, often producing policies that induce oscillations or unbounded state divergence due to exploration-driven transients and function approximation errors. While Lyapunov-based methods offer a principled route to stability, existing approaches suffer from recurring limitations: the difficulty of constructing valid Lyapunov functions in unknown, high-dimensional environments; the sample inefficiency of on-policy Lyapunov learning; and the computational burden of solving projections or constrained optimizations at every update step. Furthermore, learned Lyapunov certificates often lack global validity guarantees, and enforcing constraints can destabilize learning if not implemented rigorously.

Methodology
The authors propose LC-SAC (Lyapunov-Constrained Soft Actor-Critic), a framework that integrates Koopman Operator theory with off-policy RL to provide stability guarantees without training auxiliary Lyapunov networks. The methodology consists of three primary components:

1. Offline Model Learning via EDMD:
   Instead of learning dynamics directly in the original state space, the system learns a linear lifted surrogate of the error dynamics ($e_t = x_t - x_{ref,t}$) using Extended Dynamic Mode Decomposition (EDMD). A dictionary of observables (state variables combined with Radial Basis Functions) lifts the nonlinear dynamics into a linear control-affine model: $z_{t+1} \approx A z_t + B u_t$, where $z = g(e)$.

2. Closed-Form CLF Construction:
   Using the identified matrices $A$ and $B$, the authors solve the Discrete Algebraic Riccati Equation (DARE) to obtain a unique stabilizing solution $P \succ 0$. This yields a closed-form quadratic Control Lyapunov Function (CLF): $V(z) = z^\top P z$. To ensure the CLF is well-posed for higher-dimensional lifted models (specifically 3D quadrotors and cartpols), three structural refinements are introduced:

   • Spectral-radius normalization: Scaling the matrix $A$ if its spectral radius $\rho(A) > 1$ to ensure stability before solving DARE.
   • Physically meaningful LQR cost: Setting state costs ($Q$) based on physical error states.
   • Value-bias anchor: Enforcing $V(0)=0$ by subtracting a bias term $V_{bias} = g(0)^\top P g(0)$, correcting for non-zero RBF centers at the origin.

3. Online Policy Optimization with CVaR Constraint:
   The SAC actor update is augmented with a Lagrangian penalty that enforces the Lyapunov decrease condition. Rather than penalizing the average violation, the method aggregates the worst-case tail of violations using a Conditional Value-at-Risk (CVaR) objective (targeting the top 25% of violations). The actor loss becomes:
   $$L_\pi = J_{SAC} + \lambda (L_{CVaR}^v - \zeta)$$
   where $L_{CVaR}^v$ is the mean of the worst-case one-step Lyapunov violations, $\lambda$ is a dual variable updated via projected ascent, and $\zeta$ is a tolerance. This concentrates constraint pressure on rare but severe instability events.

Key Contributions

1. Novel CLF Derivation: A methodology to obtain a closed-form CLF via Koopman/EDMD lifting and DARE, embedded as a Lagrangian constraint in SAC. This avoids training an auxiliary Lyapunov network, reducing computational complexity and improving data efficiency.
2. Theoretical Analysis: Proof that the constrained actor update locally reduces the one-step Lyapunov violation and that satisfying the constraint implies asymptotic (or exponential, under a strengthened margin) stability of the lifted surrogate system.
3. Structural Refinements: The introduction of spectral-radius normalization, physically meaningful LQR costs, and a $V(0)=0$ anchor, which are shown to be necessary for the closed-form CLF to function correctly in high-dimensional lifted models.
4. Comprehensive Empirical Study: Evaluation across six benchmark tasks (2D/3D quadrotor stabilization/tracking, cartpole stabilization/tracking) comparing LC-SAC against vanilla SAC, a mean-aggregated variant (LC-SAC-Mean), and a potential-based reward shaping baseline (Lyap-RS-SAC).

Results
The experiments, conducted on the safe-control-gym suite, reveal a distinct stability-performance trade-off:

• Stabilization Tasks: On tasks with fixed setpoints (e.g., cartpole stabilization), LC-SAC and its variants match or exceed vanilla SAC performance (up to +25% improvement on cartpole) while dramatically reducing trial-to-trial variance. This indicates the constraint prevents catastrophic policy divergence during training.
• Tracking Tasks: On aggressive trajectory tracking tasks, the stability constraint incurs a modest return cost (3–15% lower mean return compared to vanilla SAC) in exchange for significantly reduced variance and more repeatable training.
• Ablation Findings:
  • Hard Constraint vs. Reward Shaping: Replacing the hard Lagrangian constraint with potential-based reward shaping (Lyap-RS-SAC) causes catastrophic failure on quadrotor tasks (up to 94% return collapse), demonstrating that hard constraints are essential for high-dimensional dynamics.
  • CVaR vs. Mean: The CVaR aggregation (LC-SAC) targets worst-case violations, resulting in lower worst-case surrogate losses but slightly higher mean floors compared to the mean-aggregated variant (LC-SAC-Mean).
  • Structural Fixes: Without the proposed EDMD refinements (normalization and bias anchoring), the surrogate Lyapunov loss for 3D tasks was unbounded (868 at step 4000), completely blocking learning. The fixes reduced this to near zero, enabling successful training.

Significance and Claims
The paper claims that LC-SAC provides a computationally efficient and theoretically grounded approach to safe RL by leveraging the global linearization properties of Koopman operators to derive stability certificates offline. The work demonstrates that:

1. Stability constraints can be effectively integrated into off-policy RL without the sample inefficiency of learning a Lyapunov function online.
2. A hard Lagrangian constraint is superior to reward shaping for maintaining stability in complex, high-dimensional systems.
3. Structural refinements in the EDMD pipeline are critical for extending closed-form CLF methods to higher-dimensional systems like 3D quadrotors.
4. The approach offers a favorable trade-off, trading a small amount of peak performance on aggressive tracking tasks for significantly improved training reliability and robustness.

The authors note that the current time-invariant quadratic CLF is a limitation for aggressive time-varying references, suggesting future work should explore time-varying or data-driven CLF designs to further close the performance gap on tracking tasks.