Policy Optimization with Differentiable MPC: Convergence Analysis under Uncertainty

1. Problem Statement

The paper addresses the challenge of hyperparameter tuning for Model Predictive Control (MPC) when the system dynamics are partially unknown and subject to stochastic noise.

• Context: MPC relies on an internal model of the system to compute optimal control actions. Traditionally, MPC parameters (cost weights, constraints, and model parameters) are tuned manually or via trial-and-error.
• The Gap: While "Differentiable MPC" (using implicit differentiation to compute gradients of the MPC solution map) has emerged as a powerful tool for automatic tuning, existing convergence guarantees typically assume the system model is perfectly known.
• The Core Problem: In real-world applications, the true system dynamics ($\theta$) are unknown. The paper seeks to answer: Can we simultaneously learn the system model and optimize the MPC policy using gradient-based methods, and does this joint process converge to an optimal solution?

2. Methodology

The authors propose a closed-loop optimization framework that interleaves system identification with policy optimization. The method operates in iterative "epochs" (or iterations $k$).

A. System Model and Formulation

• System Dynamics: The system is modeled as a discrete-time linear-in-parameters system with additive noise:
  $$x_{t+1} = \phi(x_t, u_t)^\top \theta + \varphi(x_t, u_t) + w_t$$
  where $\theta$ is the unknown constant parameter vector, and $w_t$ is stochastic noise.
• MPC Policy: The control input $u_t$ is generated by an MPC controller that solves a Quadratic Program (QP) over a finite horizon. The MPC uses a prediction model based on a nominal parameter estimate $\tilde{\theta}$.
• Objective: Minimize an upper-level cost function $C$ (e.g., tracking error, energy) over the closed-loop trajectory, subject to state and input constraints.

B. The Proposed Algorithm (Algorithm 2)

The core algorithm combines three distinct steps in each iteration $k$:

1. Data Collection & System Identification (Section IV-A):

   • The system is deployed with the current policy $p_k$.
   • State-input trajectories are collected.
   • A Recursive Least Squares (RLS) estimator is used to update the nominal model estimate $\theta_k$ and construct a confidence set $\Theta_k$. This set is an ellipsoid guaranteed to contain the true parameter $\theta$ with high probability ($1-\delta$).
   • Key Assumption: The system satisfies a "Persistency of Excitation" (PE) condition over the iteration horizon, ensuring the uncertainty in $\theta_k$ shrinks as $k \to \infty$.

2. Gradient Estimation via Backpropagation (Section IV-B):

   • Using the current nominal model $\theta_k$, the authors compute the gradient of the cost function with respect to the design parameters $p$.
   • They utilize Differentiable Optimization (specifically, the implicit function theorem applied to the KKT conditions of the MPC QP) to compute the sensitivity of the MPC solution map.
   • The gradient is computed via the chain rule, linking the cost function $\to$ MPC solution $\to$ system dynamics.

3. Parameter Update (Section IV-C):

   • The design parameters $p_k$ (which include both MPC cost weights and potentially the prediction model $\tilde{\theta}$) are updated using a Projected Gradient Descent step:
     $$p_{k+1} = \Pi_{Y_k} [p_k - \alpha_k J_k^{\bar{C}}]$$
   • The projection $\Pi_{Y_k}$ ensures that the updated prediction model $\tilde{\theta}$ remains within the confidence set $\Theta_k$ derived from system identification.

C. Special Variants

• Certainty Equivalence (Algorithm 3): A simplified variant where the MPC prediction model is forced to be exactly the current best estimate $\theta_k$ at every step. This reduces the dimensionality of the optimization problem.
• Imperfect Model Learning (Section VI-B): A variant for cases where the PE condition is not met (model does not converge to the true $\theta$). The authors derive a probabilistic upper bound on the gradient norm to guarantee convergence to a suboptimal solution within a known error bound.

3. Key Contributions

1. Convergence Guarantees under Uncertainty:

   • The paper provides the first rigorous convergence analysis for differentiable MPC policy optimization when the system dynamics are unknown and stochastic.
   • Theorem 2: Proves that under asymptotically exact system identification (PE condition holds), the algorithm converges to a critical point of the expected cost function with arbitrarily high confidence.
   • Theorem 3: Establishes convergence to a suboptimal solution with a bounded error when the model is not learned perfectly, providing a probabilistic bound on the gradient norm.

2. Integration of System ID and Policy Optimization:

   • The work formalizes a framework where system identification and policy tuning occur simultaneously. It demonstrates that the confidence set from system identification can be used to constrain the policy search space, ensuring stability and feasibility.

3. Handling Non-Smoothness and Constraints:

   • The analysis relies on path-differentiability and o-minimal structures, allowing the framework to handle non-smooth cost functions and constraints (via slack variables) common in MPC, which traditional smooth optimization theory cannot easily address.

4. Simulation Results

The authors validate their approach on three distinct benchmarks:

• Random Linear Systems:
  • Compared the proposed algorithms (with and without Certainty Equivalence) against an untrained nominal MPC and an "omniscient" controller (which knows the true model).
  • Result: The tuned MPC significantly reduced suboptimality (from ~8% to ~0.3% relative to the omniscient controller). The Certainty Equivalence (CE) variant performed slightly better due to a smaller search space.
• Nonlinear Quadcopter:
  • Applied to a 12-dimensional nonlinear drone model.
  • Result: The algorithm successfully learned the model parameters (estimation error < 1% in ~40 iterations) and tuned the MPC. The closed-loop trajectories of the trained controller converged to those of the omniscient controller.
  • Robustness: The method successfully optimized a controller even when the MPC cost function did not perfectly match the upper-level objective (e.g., adding an energy penalty term not present in the MPC formulation).
• Autonomous Vehicle Lateral Control:
  • Tested on a bicycle model with unknown parameters and a non-stochastic disturbance (path curvature).
  • Result: Using the "imperfect model" variant (Section VI-B), the trained controller reduced the closed-loop cost by 35% compared to the untrained version, achieving 15% suboptimality relative to the best possible trajectory.

5. Significance and Impact

• Bridging the Gap: This work bridges the gap between data-driven control (learning models) and optimization-based control (tuning MPC). It moves beyond the assumption that a perfect model is available a priori.
• Theoretical Rigor: By providing convergence guarantees in the presence of uncertainty, the paper moves differentiable MPC from a heuristic "black-box" tuning tool to a theoretically grounded framework.
• Practical Applicability: The inclusion of the Certainty Equivalence variant and the handling of imperfect model learning makes the approach highly relevant for real-world applications where data is noisy and perfect identification is impossible.
• Safety: While the paper focuses on convergence, it notes that the learned models can be integrated with robust MPC frameworks (like those in their previous work [11]) to ensure safety, paving the way for safe, self-tuning autonomous systems.

In summary, the paper establishes that iterative system identification combined with differentiable policy optimization is a viable and convergent strategy for designing high-performance controllers for systems with unknown dynamics and noise.