Safety-Constrained Optimal Control for Unknown System Dynamics

1. Problem Statement

The paper addresses the challenge of solving continuous finite-horizon optimal control problems when the true system dynamics (the "plant") are unknown or too complex to model precisely.

• The Core Issue: In many applications (e.g., autonomous driving), controllers rely on approximate models (digital twins). However, model mismatch typically leads to suboptimal performance or safety violations.
• The Specific Challenge: The authors focus on problems involving safety constraints that depend on both the system state and the control input. The goal is to determine under what conditions a control strategy derived from an imperfect model remains optimal for the true system, despite the dynamics mismatch.

2. Methodology

The authors propose a framework that combines Pontryagin's Minimum Principle (PMP) with a penalized cost formulation.

A. Problem Formulation

1. True System (Plant): Governed by unknown dynamics $\dot{\hat{x}} = \hat{f}(t, \hat{x}, u)$ with safety constraints $c(t, \hat{x}, u) \leq 0$. The objective is to minimize a cost functional $J_{act}$.
2. Surrogate Model: A known approximate model $\dot{x} = f(t, x, u)$ is used for control synthesis.
3. Penalized Cost: To bridge the gap between the model and the plant, the authors augment the model-based cost function with a penalty term proportional to the squared error between the model state $x$ and the observed plant state $\hat{x}$:
   $$J_{mod} = \int_0^T \left[ \ell(t, x, u) + \beta(t)\|x - \hat{x}\|^2 \right] dt + \phi(x(T))$$
   Here, $\beta(t)$ is a time-varying weighting function.

B. Hamiltonian Analysis

The authors derive the optimality conditions for both the true plant and the penalized model using PMP.

• Plant Hamiltonian ($\hat{H}$): Includes the true dynamics and constraints.
• Model Hamiltonian ($H$): Includes the approximate dynamics, constraints, and the penalty term.
• Key Insight: The penalty term $\beta(t)\|x - \hat{x}\|^2$ depends on the states but not explicitly on the control input $u$. Consequently, this term does not appear in the gradient of the Hamiltonian with respect to $u$ ($\nabla_u H$).

C. Equivalence Conditions

The paper establishes structural conditions under which the optimal control derived from the model ($u^\circ$) is identical to the optimal control for the true plant ($u^*$), even if the dynamics differ.

• General Case (Convex Analysis): Using subdifferentials and normal cones, the authors prove that if the subgradients of the Hamiltonians with respect to $u$ coincide at a specific point, the minimizers over the feasible set $C$ are identical.
• Quadratic Control Case: For systems with quadratic control effort (common in robotics), the authors derive explicit, verifiable conditions:
  1. State Alignment: The model state must match the plant state ($x(t) = \hat{x}(t)$).
  2. Gradient Matching: The gradients of the dynamics and cost terms with respect to $u$ must match:
     $$\nabla_u \hat{f} = \nabla_u f, \quad \lambda^\circ = \hat{\lambda}^*, \quad \mu^\circ = \hat{\mu}^*$$
     Crucially, the penalty term influences the evolution of the states and costates (via the costate dynamics $\dot{\lambda}$) but does not alter the structure of the pointwise minimization problem for $u$.

3. Key Contributions

1. Theoretical Framework for Model Mismatch: The paper provides a rigorous mathematical proof that optimal control strategies can be equivalent between a model and a plant even with unknown dynamics, provided specific structural conditions on the Hamiltonian minimizers are met.
2. Decoupling of Accuracy and Optimality: A major insight is that the penalty term corrects the trajectory (state/costate evolution) to align with reality without changing the control law structure (the minimization of the Hamiltonian). This explains why approximate models often work well in practice.
3. Safety Constraints Integration: Unlike previous works that focused on unconstrained systems, this framework explicitly handles safety constraints involving both state and control inputs.
4. Hardware Validation: The theory is validated on a real robotic testbed (LIMO robots) for a cruise control application with rear-end collision avoidance constraints.

4. Experimental Results

The authors implemented the framework on a fleet of LIMO robots:

• Setup: An "ego" robot (controlled) follows a "front" robot (uncontrolled). The controller uses a model with incorrect parameters (e.g., different actuation lag $\tau$ and gain $k$) compared to the real robot.
• Comparison: Two runs were performed:
  1. Original Optimal Control: Using the true (unknown) parameters (simulated for comparison).
  2. Model-Based Penalized Control: Using the approximate model with the proposed penalty term.
• Findings:
  • Despite significant parameter mismatch, the control inputs generated by the model-based strategy ($u_{mb}$) were identical to the true optimal control ($u_{opt}$) whenever the safety constraints were active or the system was in a specific regime.
  • The penalty term successfully drove the model state to align with the plant state, ensuring the Hamiltonian minimizers coincided.
  • The results confirmed that $\nabla_u \hat{H} \neq \nabla_u H$ (gradients differ), yet $\arg \min_u \hat{H} = \arg \min_u H$ (minimizers are the same) due to the dominance of the control constraints and the specific structure of the penalty.

5. Significance and Impact

• Shift in Learning Paradigm: The paper suggests a paradigm shift for learning-based control (e.g., Reinforcement Learning). Instead of striving for perfect system identification (exact model matching), learning efforts should focus on preserving the structural properties that ensure Hamiltonian minimization equivalence.
• Robustness of Digital Twins: It provides a theoretical justification for the success of "digital twins" in control applications, showing that as long as the penalty term corrects the state trajectory, the control policy derived from the twin remains optimal.
• Practical Applicability: The framework offers a principled way to design controllers for systems where exact modeling is impossible (e.g., complex robotics, biological systems) while guaranteeing safety and optimality under mild convexity assumptions.

In summary, this work bridges the gap between theoretical optimal control and practical implementation with imperfect models, proving that with the right penalization strategy, model-based controllers can achieve true optimality even in the presence of unknown dynamics and safety constraints.