On Building Myopic MPC Policies using Supervised Learning

1. Problem Statement

Model Predictive Control (MPC) is a powerful control strategy that solves an optimization problem online at every sampling instant. However, this imposes a heavy computational burden, making it difficult to implement on embedded systems with limited resources or requiring fast sampling times.

To mitigate this, Approximate Explicit MPC uses supervised learning to train a neural network to mimic the optimal MPC policy (state-to-action mapping) offline. While this reduces online computation, it suffers from two main drawbacks:

1. Performance Loss: The learned policy is an approximation; changes in controller parameters (e.g., constraints, sampling time) during online operation can lead to performance degradation or safety violations.
2. Lack of Guarantees: Simply minimizing the error between the learned policy and the optimal policy does not guarantee that the closed-loop system remains stable or performs well.

Existing approaches to learning the cost-to-go function (value function) typically rely on Reinforcement Learning (RL) or Approximate Dynamic Programming (ADP) using closed-loop trajectory data. This paper addresses the gap in using offline supervised learning to learn the value function directly from optimal state-action-value tuples, aiming to create a "myopic" (short-horizon) MPC that retains the performance guarantees of long-horizon MPC.

2. Methodology

The proposed method shifts the focus from learning the policy ($\pi(x) \to u$) to learning the optimal value function ($V^*(x)$), which represents the future cost from a given state.

A. Myopic MPC Formulation

Instead of solving a long-horizon optimization online, the controller solves a one-step (myopic) optimization problem:
$$\min_{u \in U} \ell(x(t), u) + V(x^+)$$
where $x^+ = f(x(t), u)$ is the successor state, $\ell$ is the stage cost, and $V(x^+)$ is the learned approximation of the optimal cost-to-go. If $V$ is the exact optimal value function, Bellman's Principle of Optimality ensures that this greedy, one-step approach yields the same performance as the full-horizon MPC.

B. Supervised Learning with Descent Property Constraint

The authors propose training a parametric function approximator $V(x; \theta)$ using a dataset $D$ of optimal tuples $\{(x_i, u^*_i, V^*(x_i))\}$ generated offline by solving long-horizon MPC problems.

The core innovation lies in the loss function and constraints:

1. Standard Approach (MSE): Minimizing the Mean Squared Error (MSE) between the predicted value and the optimal value:
   $$\min_\theta \sum \|V(x_i; \theta) - V^*(x_i)\|^2$$
   The authors argue that a low MSE does not guarantee good control performance. A function can fit the data well but have incorrect gradients or relative differences, leading to poor closed-loop behavior.

2. Proposed Approach (Descent Property): The authors introduce a constraint ensuring that the learned value function satisfies the descent property. Specifically, for any training point $x_i$ and optimal action $u^*_i$, the cost-to-go of the successor state must be lower than the current state's cost, adjusted by the immediate stage cost:
   $$V(f(x_i, u^*_i); \theta) - V(x_i; \theta) \leq \ell(x_i, u^*_i)$$
   The training problem becomes a constrained optimization:
   $$\min_\theta \text{MSE} \quad \text{s.t.} \quad V(f(x_i, u^*_i); \theta) - V(x_i; \theta) \leq \ell(x_i, u^*_i)$$
   This constraint steers the learning process to ensure the value function behaves like a Control Lyapunov Function (CLF), guaranteeing that the system moves toward lower costs.

C. Data Generation

The training data is generated by solving the full-horizon MPC offline for various states. To handle the computational cost of generating large datasets, the authors leverage parametric sensitivity analysis. Once an optimal solution is found for a specific state, sensitivities allow for the rapid approximation of solutions for neighboring states, effectively augmenting the dataset without re-solving the full optimization problem for every point.

3. Key Contributions

• Value Function over Policy: Proposes learning the cost-to-go function via supervised learning rather than the policy itself, enabling online adaptability.
• Descent Property Constraint: Introduces a novel constraint in the supervised learning objective that enforces the physical reality of cost reduction, which is more critical for stability than raw MSE minimization.
• Offline Learning with Online Adaptability: Demonstrates that while the value function is learned offline, the resulting myopic MPC can adapt online to changes in constraints, sampling times, and model parameters because it solves an optimization problem (Eq. 2) rather than executing a fixed neural network policy.
• Sensitivity-Based Data Augmentation: Utilizes parametric sensitivities to efficiently generate the large volume of state-action-value tuples required for training.

4. Results (CSTR Benchmark Example)

The method was validated on a Continuous Stirred Tank Reactor (CSTR) control problem with a long-horizon expert MPC ($N=140$) as the baseline.

• MSE-Only Training: A Radial Basis Function (RBF) network trained only on MSE achieved a very low training error (MSE = 0.0515). However, when used in the myopic MPC, the closed-loop trajectories failed to reach the setpoint, exhibiting cyclic behavior and poor performance.
• Descent-Constraint Training: An RBF network trained with the descent property constraint had a significantly higher training error (MSE = 0.8199). Despite the "worse" fit to the training data, the resulting myopic MPC successfully tracked the setpoint, matching the performance of the full-horizon expert MPC.
• Robustness to Parameter Changes: The authors tested the myopic MPC with the descent-constrained value function under changed conditions:
  • Sampling time reduced from 3s to 0.1s.
  • State constraints tightened (lower bound on temperature increased).
  • Result: The myopic MPC adapted successfully to these changes, whereas a direct policy approximation would have failed because it is fixed to the training distribution.

5. Significance

This paper fundamentally challenges the standard paradigm of "approximate explicit MPC" where the goal is to minimize the error between the learned policy and the optimal policy. It demonstrates that minimizing prediction error (MSE) is insufficient for control stability.

By enforcing the descent property, the authors ensure that the learned value function preserves the structural properties required for stability, even if the absolute values are less accurate. This approach offers a middle ground:

1. Low Online Computation: Solves a simple one-step optimization instead of a long-horizon one.
2. High Performance: Maintains the stability and optimality guarantees of standard MPC.
3. Flexibility: Unlike fixed policy networks, the myopic MPC can handle real-time changes in system constraints and parameters, making it highly suitable for practical embedded control applications.