Policy Optimization for Unknown Systems using Differentiable Model Predictive Control

1. Problem Statement

The paper addresses the challenge of policy optimization for Model Predictive Control (MPC) when the system dynamics are unknown or inaccurately modeled.

• Context: MPC policies rely on a system model to predict future trajectories and optimize control inputs. Standard policy optimization methods often assume the dynamics model is exact.
• The Gap: When the model is inaccurate (inexact), purely model-based gradient descent can lead to suboptimal performance or divergence. Conversely, purely data-driven (model-free) zeroth-order methods are robust but often suffer from slow convergence and high sample complexity.
• Objective: The goal is to find optimal MPC parameters ($\theta$) that minimize a closed-loop cost function $C(x, u)$ over a finite horizon, despite the true dynamics $f$ being unknown and only an approximate model $g$ being available. The problem is inherently non-smooth due to the MPC optimization layer and the presence of constraints.

2. Methodology

The authors propose a hybrid policy optimization framework that blends model-based gradient estimation with zeroth-order (model-free) optimization. This approach is inspired by "gray-box" optimization techniques but adapted for the non-smooth landscape of MPC.

A. Mathematical Foundations

• Non-smooth Analysis: Since MPC solutions are non-smooth functions of parameters, the authors utilize the theory of definable functions and conservative Jacobians (Bolte & Pauwels, 2021). This allows for the definition of generalized derivatives that satisfy the chain rule, enabling sensitivity analysis of the MPC solution map.
• Stationarity: Due to the lack of full information, the algorithm targets Goldstein $\delta$-critical points rather than standard Clarke stationary points. This is a weaker optimality condition suitable for non-smooth, non-convex problems where exact gradients are unavailable.

B. The Hybrid Algorithm

The core update rule at iteration $k$ combines two gradient estimates:
$$\theta_{k+1} = \Pi_\Theta [\theta_k - \alpha_k d_k]$$
where the search direction $d_k$ is a weighted sum:
$$d_k = \eta_k d_{k,1} + (1 - \eta_k) d_{k,2}$$

1. Model-Based Direction ($d_{k,1}$):

   • Uses the approximate model $g$ to compute a conservative Jacobian of the cost function via the chain rule.
   • Recursively computes Jacobians of the state and control trajectories ($J_x, J_u$) using the model's Jacobians ($J_g$) and the MPC policy's Jacobians.
   • Pros: Fast convergence if the model is accurate. Cons: No convergence guarantees if the model is biased.

2. Zeroth-Order Direction ($d_{k,2}$):

   • Uses a one-point gradient estimator based on randomized smoothing (Flaxman et al., 2004; Lin et al., 2022).
   • Evaluates the cost function at a perturbed parameter $\theta_k + \delta v_k$ (where $v_k$ is random noise) to estimate the gradient of a smoothed version of the cost function, $C_\delta$.
   • Pros: Guaranteed convergence under mild assumptions regardless of model accuracy. Cons: Slower convergence rate.

3. Adaptive Weighting ($\eta_k$):

   • The scalar $\eta_k \in [0, 1]$ balances the two directions.
   • The paper proposes a schedule where $\eta_k \to 0$ as $k \to \infty$. This allows the algorithm to leverage the model for fast initial transients and gradually rely on data-driven correction to ensure asymptotic convergence to a critical point.

C. Convergence Guarantees

The authors prove that under assumptions of local Lipschitz continuity and definability (Assumptions 1 & 2), and with appropriate step-size ($\alpha_k$) and weighting ($\eta_k$) schedules, the sequence $\theta_k$ converges to a Goldstein Fritz-John $\delta$-critical point of the constrained optimization problem.

3. Key Contributions

1. Novel Hybrid Framework: Introduction of a policy optimization algorithm that seamlessly integrates differentiable MPC (model-based) with zeroth-order optimization (model-free) for unknown systems.
2. Theoretical Guarantees under Uncertainty: Proving convergence to a critical point even when the dynamics model is inaccurate, a significant advancement over existing MPC tuning methods that require exact models.
3. Handling Non-smoothness: Application of conservative Jacobian theory to MPC policy optimization, allowing for rigorous sensitivity analysis of the non-smooth MPC solution map.
4. Adaptive Trust Mechanism: A flexible weighting scheme ($\eta_k$) that allows practitioners to trade off between model speed and data robustness dynamically.

4. Simulation Results

The method was validated on a 12-dimensional nonlinear quadcopter control task (Abdulkareem et al., 2022).

• Setup: The true system is a nonlinear quadcopter. The controller uses a linearized model $g$ identified via least-squares from 100 trajectories. The goal is to track a reference trajectory while respecting state and input constraints.
• Baselines:
  • Pure Model-Based: Fast initial improvement but became unstable and failed to converge due to model mismatch.
  • Pure Zeroth-Order: Stable and consistent but converged very slowly.
  • Proposed Hybrid: Achieved the best of both worlds—fast initial convergence (driven by the model) and stable long-term convergence (driven by zeroth-order correction).
• Performance:
  • The hybrid method achieved a final cost of 1084.91, compared to 1081.3 for a controller tuned with the exact nonlinear model (less than 1% suboptimality).
  • The trajectories of the learned controller were qualitatively and quantitatively nearly identical to the "perfect model" baseline.
  • Varying the decay rate of $\eta_k$ showed that higher reliance on the model ($\gamma=0.25$) sped up initial learning but increased instability, while lower reliance ($\gamma=0.75$) was more robust but slower.

5. Significance

This work bridges the gap between model-based control (which offers safety and interpretability) and model-free learning (which offers robustness to uncertainty).

• Practical Impact: It provides a viable path for deploying MPC in real-world systems where obtaining a perfect physics-based model is impossible, yet purely data-driven reinforcement learning is too sample-inefficient or unsafe.
• Safety: By retaining the MPC structure, the method maintains constraint satisfaction guarantees during the learning process, unlike many black-box policy gradient methods.
• Future Direction: The paper suggests future work on strengthening safety guarantees and extending the framework to handle nonconvex upper-level constraints more broadly.

In summary, the paper presents a robust, theoretically grounded, and empirically effective method for tuning MPC policies in the presence of model uncertainty, demonstrating that a "gray-box" approach significantly outperforms purely model-based or purely model-free alternatives.