Technical Summary: An Adjoint-based Neural Regulator for Real-Time Optimal Control with State Constraints

1. Problem Statement

Real-time constrained optimal control of nonlinear systems presents a significant challenge in robotics and autonomous systems, requiring controllers to balance performance with strict constraint satisfaction under millisecond-level computational limits.

• Nonlinear Model Predictive Control (NMPC): While effective, NMPC requires solving a constrained nonlinear optimization problem online at every time step. In safety-critical or fast-dynamic settings, the repeated optimization and solver latency render this approach computationally prohibitive.
• Reinforcement Learning (RL): RL offers fast online inference by amortizing computation into offline training. However, standard RL approaches (e.g., Soft Actor-Critic) often struggle with sample efficiency, generalization to out-of-distribution (OOD) conditions, and strict constraint satisfaction. Common safety filters (e.g., Control Barrier Functions or CBFs) applied to RL policies can lead to conservative or aggressive corrections if the nominal policy does not inherently encode the problem's optimality structure.

The core problem addressed is the development of a control framework that achieves the computational speed of learning-based methods while retaining the optimality structure of optimal control and guaranteeing safety constraints without the heavy online optimization burden of NMPC.

2. Methodology: Adjoint-Based Neural Regulator (ANR)

The paper proposes the Adjoint-based Neural Regulator (ANR), a learning-based framework that learns a neural co-state (adjoint) policy rather than a direct control law. This approach is grounded in Pontryagin's Minimum Principle (PMP).

A. Core Architecture

The framework consists of two phases: offline training and online deployment.

1. Co-State Parameterization (CoNN):

   • A neural network, termed the Co-State Neural Network (CoNN), maps the current system state $x_i$ to a finite-horizon co-state (adjoint) trajectory $\hat{\Lambda}_i = \{\hat{\lambda}_0, \dots, \hat{\lambda}_{N-1}\}$.
   • Unlike RL, which learns a policy $\pi(u|x)$, the CoNN learns the mapping from state to adjoint variables, which represent the sensitivity of the value function with respect to the state.

2. Offline Training:

   • Rollout: Given a predicted co-state sequence, the control input sequence is derived by minimizing the Hamiltonian pointwise. For quadratic costs, this yields an explicit control law: $\hat{u}^*_k = -\frac{1}{2}R^{-1}g^\top(x_k)\hat{\lambda}_k$.
   • Loss Function: The network is trained end-to-end using a three-term loss:
     • Stage Cost: Sum of running costs ($x^\top Q x + u^\top R u$) over the horizon.
     • Terminal Cost: Cost at the final state.
     • Co-State Regularization: An $\ell_1$ norm penalty on the co-state trajectory to encourage minimum-norm predictions.
   • The loss is backpropagated through the horizon rollout to update network parameters.

3. Online Deployment (Real-Time Control):

   • At each time step $t_k$, the CoNN predicts a co-state sequence from the current state $x_k$.
   • The first predicted co-state $\hat{\lambda}_{0|k}$ is used to compute the control input.
   • Safety and Feasibility: Instead of a simple projection, the control input is determined by solving a constrained Hamiltonian minimization problem in real-time. This optimization incorporates:
     • Actuator limits ($u \in \mathcal{U}$).
     • State safety constraints enforced via High-Order Control Barrier Functions (HOCBFs).
   • The resulting control law is:
     $$u^*_k = \arg \min_{u} (u^\top R u + \hat{\lambda}^\top_{0|k} g(x_k) u)$$
     subject to input constraints and HOCBF conditions ensuring safety.

3. Key Contributions

• Novel Framework: Introduction of the ANR, which decouples optimality learning (via the adjoint) from constraint enforcement (via real-time constrained minimization).
• Optimality Preservation: By learning the adjoint rather than the action, the framework explicitly encodes the indirect optimality structure of the constrained control problem, avoiding the "nominal-then-project" pitfalls of standard RL-CBF methods.
• Unified Safety and Feasibility: The method handles hard state constraints (via HOCBFs) and input constraints simultaneously within the Hamiltonian minimization step, ensuring feasibility without requiring complex reward engineering.
• Computational Efficiency: The approach achieves performance comparable to NMPC but with significantly reduced computational cost by replacing online nonlinear optimization with a single neural network inference and a small Quadratic Program (QP).

4. Experimental Results

The framework was evaluated on a unicycle robot model with dynamics $\dot{x} = v \cos \theta, \dot{y} = v \sin \theta, \dot{\theta} = \omega, \dot{v} = a$, tasked with navigating to a reference while avoiding circular obstacles. Comparisons were made against NMPC (using IPOPT) and RL (Soft Actor-Critic with HOCBF-QP safety filter).

Performance Metrics

• Computational Speed:
  • RL: Fastest (~0.5 ms).
  • ANR: ~1.2–1.4 ms (approx. 250x faster than NMPC).
  • NMPC: ~350–1200 ms (depending on scenario complexity).
• Generalization (Out-of-Distribution):
  • RL: Failed to generalize reliably to unseen initial states or references, even within the training domain distribution, achieving low success rates (e.g., 30% in in-distribution tests, failing completely in OOD tests).
  • ANR: Exhibited near-invariant performance across unseen scenarios, including OOD initial states and non-zero references. Success rates were comparable to NMPC (e.g., 98% success in OOD obstacle scenarios).
• Accuracy:
  • ANR achieved Absolute Convergence Errors (ACE) and Mean Squared Derivatives (MSD) comparable to NMPC, though slightly higher than RL in in-distribution cases where RL overfitted.
  • In OOD scenarios, ANR significantly outperformed RL, which failed to reach goals or satisfy constraints.

Specific Findings

• In obstacle-free OOD scenarios, ANR matched NMPC performance with a computational cost two orders of magnitude lower.
• In obstacle-laden scenarios, ANR maintained high success rates (98% in randomized dense obstacle tests) where RL failed.
• The ANR successfully handled non-zero reference tracking tasks in OOD settings, a scenario where RL baselines were not evaluated due to prior poor performance.

5. Significance and Claims

The paper claims that the ANR framework successfully bridges the gap between the computational efficiency of learning-based control and the rigorous optimality and safety guarantees of optimal control theory.

• Significance: The method offers a viable alternative to NMPC for applications requiring millisecond-level control decisions, such as fast dynamical systems and safety-critical robotics, where repeated online optimization is prohibitive.
• Robustness: Unlike standard RL, which often requires extensive reward engineering and struggles with generalization, the ANR leverages the structural properties of PMP to generalize effectively to unseen initial conditions and references.
• Safety: By integrating HOCBFs directly into the Hamiltonian minimization, the framework ensures strict satisfaction of state and input constraints without the structural limitations of post-hoc safety filters applied to non-optimal nominal policies.

The authors conclude that the proposed approach achieves performance at par with NMPC at a fraction of the computational cost, while significantly outperforming RL in out-of-training-distribution regimes. Future work is identified as extending the framework to dynamic obstacles and higher-dimensional systems.