Technical Summary: Stability Margins of Neural Network Controllers

Problem Statement

Standard neural network (NN) controllers trained via reinforcement learning (RL) often lack guarantees regarding closed-loop stability, limiting their deployment in safety-critical domains like aerospace where regulatory bodies require specific robustness metrics. While recent work has addressed verification and synthesis of NN controllers for asymptotic stability, there is a gap in methods that can train NN controllers to maximize performance rewards while strictly adhering to classical robustness requirements, specifically disk margins.

The paper addresses the constrained optimization problem of training a neural network controller $K$ to maximize an expected cumulative reward over a trajectory, subject to the constraint that the closed-loop system remains stable for a class of plants $\mathcal{M}_\alpha$. This class represents a nominal Linear Time-Invariant (LTI) plant interconnected with uncertainties (both LTI and nonlinear) bounded by a disk margin $D(\alpha, \sigma)$. The disk margin is chosen because it accounts for simultaneous variations in gain and phase, offering a more rigorous robustness guarantee than classical gain and phase margins alone.

Methodology

1. System Modeling

The authors model the control architecture as an interconnection of a nominal LTI plant and a neural network controller, both subject to uncertainty blocks.

• Plant: The nominal plant is LTI. Uncertainty is modeled as an input multiplicative block $H_{\Delta p}$, where the uncertainty $\Delta_p$ is $L_2$-norm bounded by $\alpha$.
• Controller: The neural network is modeled as a Recurrent Implicit Neural Network (RINN). It consists of an LTI system $G_k$ interconnected with static, element-wise activation functions $\phi$ (e.g., tanh, ReLU). This formulation allows the controller to have memory (dynamics) and encompasses various architectures, including feedforward and residual networks.
• Uncertainty Characterization: Both the plant uncertainty $\Delta_p$ and the controller's activation functions $\phi$ are characterized using Integral Quadratic Constraints (IQCs). The uncertainty $\Delta_p$ satisfies a sector-bound IQC related to its $L_2$ gain, while the activation functions satisfy sector and slope-restriction IQCs.

2. Stability Certification

The core theoretical contribution is a condition to certify that a specific RINN controller satisfies the disk margin $D(\alpha, \sigma)$.

• The closed-loop system is reformulated as an LTI system interconnected with a composite uncertainty block containing both $\Delta_p$ and $\phi$.
• Using a Lyapunov function approach combined with IQC theory, the authors derive a Linear Matrix Inequality (LMI) condition (Lemma 3.1). This condition involves finding a positive definite matrix $X$ and diagonal multiplier matrices $\Lambda_p$ and $\Lambda_k$ such that a specific matrix inequality holds.
• Direct application of this LMI for training is computationally intractable because the inequality is bilinear in the controller parameters $\theta$ and the Lyapunov matrix $X$.

3. Training Algorithm with Projection

To synthesize a controller that maximizes reward while satisfying the stability constraint, the authors propose an iterative training algorithm (Algorithm 1) that alternates between two steps:

1. Reinforcement Learning Step: A standard RL step (e.g., Proximal Policy Optimization) updates the controller parameters $\theta$ to maximize the reward. The resulting parameters $\theta'$ may violate the stability margin.
2. Stability Margin-Enforcing Step: If $\theta'$ does not satisfy the margin, it is projected back into the set of stabilizing controllers.
   • To make this projection convex, the authors utilize a change of variables (inspired by [8]) to transform the controller parameters $\theta$ and Lyapunov matrix $X$ into new variables $\hat{\theta}$ (including matrices $S, R, N_A, \dots$).
   • In this transformed space, the stability condition becomes affine in $\hat{\theta}$, allowing the projection to be formulated as a Semidefinite Program (SDP).
   • The algorithm solves an SDP to find the closest valid $\hat{\theta}$ to the current $\hat{\theta}'$, then reconstructs the controller parameters $\theta$ from the solution.
   • The algorithm also alternates the multiplier $\Lambda_p$ to reduce conservatism, attempting to certify the margin directly before resorting to the full projection if certification fails.

Key Contributions

• Training Framework: A novel method to train neural network controllers that explicitly enforces disk margin constraints during the reinforcement learning process.
• Mathematical Formulation: The derivation of LMI conditions for disk margin certification of RINN controllers using IQCs for both plant uncertainty and activation functions.
• Convex Projection: The development of a computationally tractable projection step via a change of variables and SDP, enabling the enforcement of non-convex stability constraints within a gradient-based training loop.
• Generalization: The approach extends beyond static controllers to dynamic (recurrent) controllers and handles uncertainties described by IQCs, covering both LTI and nonlinear time-varying dynamics.

Experimental Results

The method was evaluated on a simulation of a flexible rod on a cart, a system with complex dynamics.

• Comparators: The proposed method (SM-RINN) was compared against:
  • An unconstrained Recurrent Implicit Neural Network (U-RINN).
  • A standard Fully Connected Neural Network (FCNN).
  • An LTI controller trained with the same stability constraints.
• Constraints: All constrained controllers (SM-RINN and LTI) were required to satisfy a disk margin with $\alpha = 0.353$ and $\sigma = 0$, corresponding to lower bounds of 3 dB gain margin and 20° phase margin.
• Performance:
  • The unconstrained controllers (U-RINN and FCNN) achieved the highest rewards (~3000) but offered no stability guarantees.
  • The constrained SM-RINN achieved a reward of approximately 2250.
  • The constrained LTI controller achieved the lowest reward (~1500).
• Trade-off: The results demonstrate the typical performance-robustness trade-off. However, the SM-RINN significantly outperformed the constrained LTI controller (a ~50% improvement in reward) while maintaining the exact same robustness guarantees.

Significance and Claims

The paper claims that this method enables the use of neural networks in safety-critical applications where regulatory agencies mandate stability margins. By integrating robustness constraints directly into the training loop via SDP projection, the authors show that NN controllers can achieve superior performance metrics (reward) compared to traditional linear controllers while satisfying the same rigorous robustness requirements. The work bridges the gap between the flexibility of deep learning and the safety guarantees required by classical control theory.