Technical Summary: Sampling-based Model Predictive Control Using Trust Regions

Problem Statement

Sampling-based Model Predictive Control (MPC) algorithms, such as the Cross-Entropy Method (CEM) and Model Predictive Path Integral (MPPI), solve optimal control problems by iteratively refining a proposal probability density function (PDF) over control sequences. These methods draw samples, evaluate trajectory costs, and update proposal parameters. However, current approaches rely heavily on heuristic strategies for hyperparameter adaptation, such as:

• Temperature parameters: Used in MPPI to control the sharpness of importance weighting, often held constant or adjusted heuristically.
• Momentum/Smoothing: Used in CEM and iterative MPPI to prevent premature collapse of the proposal distribution.

These heuristics are difficult to tune and can compromise convergence or performance across different problem settings. Furthermore, standard random sampling can be inefficient, requiring large sample sizes to achieve smooth control trajectories and reliable convergence, particularly in low-sample and low-iteration regimes.

Methodology

The authors propose a principled framework that replaces heuristic hyperparameter tuning with a Trust Region formulation derived from dual optimization.

1. Trust Region Formulation

The core of the method constrains the update of the proposal distribution $f_{j+1}$ relative to the previous distribution $f_j$ using a Kullback–Leibler (KL) divergence bound. The optimization problem is defined as:
$$\min_{f_{j+1}} \mathbb{E}_{f_{j+1}}[J(\bar{\xi})]$$
Subject to:

• $D_{KL}(f_{j+1} \| f_j) \leq \epsilon$ (Trust region constraint)
• $H(f_{j+1}) \geq H_{min}$ (Optional entropy lower bound to prevent collapse)
• Normalization constraints

Here, $J(\bar{\xi})$ is the cumulative cost, $\epsilon$ is the maximum allowed divergence, and $H_{min}$ is the minimum required entropy.

2. Dual Optimization and Weight Derivation

The constrained problem is solved using Lagrange multipliers ($\eta$ for KL, $\alpha$ for entropy, $\nu$ for normalization). By taking the functional derivative of the Lagrangian, the authors derive the optimal proposal distribution:
$$f^*_{j+1}(\bar{\xi}) = \frac{1}{Z_{\eta,\alpha}} f_j(\bar{\xi})^{\frac{\eta}{\eta+\alpha}} \exp\left(-\frac{J(\bar{\xi})}{\eta + \alpha}\right)$$
The optimal Lagrange multipliers $\eta^*$ and $\alpha^*$ are found by numerically solving the dual maximization problem. This process yields optimal weights for the sampled trajectories without manual tuning. When the entropy constraint is inactive ($\alpha^*=0$), the formulation coincides with standard MPPI weighting but is derived optimally rather than heuristically.

3. Projection and Sampling

Since the optimal $f^*_{j+1}$ is generally non-Gaussian (unless $\alpha^*=0$), it is projected back onto the original parametric family (e.g., Gaussian) via moment matching. The weighted empirical mean and covariance are used to update the proposal parameters for the next iteration.

4. Algorithmic Enhancements

To further improve efficiency, the authors integrate Deterministic Localized Cumulative Distribution (LCD) sampling:

• LCD Sampling: Precomputed deterministic samples are used to represent the standard normal distribution optimally.
• Transformation: These samples are mapped to the current proposal parameters via scaling and a random rotation matrix (drawn from $SO(d_\xi)$) to preserve optimality while introducing necessary stochasticity.
• Buffering: A buffer retains the best trajectories from previous iterations to warm-start the optimization.
• Time-Correlated Covariance: A fixed correlation structure (colored noise) is applied to encourage smooth control trajectories, with only marginal variances updated during optimization.

Key Contributions

1. Principled Trust Region Update: The paper introduces a KL-constrained update rule for sampling-based MPC that eliminates the need for heuristic temperature or momentum tuning. The hyperparameters are derived as optimal solutions to a dual optimization problem.
2. Integration with Deterministic Sampling: The trust region update is combined with deterministic LCD-based sampling. This combination is shown to improve sample efficiency and convergence, particularly in regimes with limited computational budgets (low sample counts and few iterations).
3. Comprehensive Evaluation: The method is evaluated against heuristic baselines (MPPI and CEM) using random, low-discrepancy (Sobol, Halton), and deterministic LCD sampling strategies across two benchmark environments.

Experimental Results

The proposed method was tested on two environments: Cart-Pole Swing-Up and Truck Backer-Upper.

• Sample Efficiency: In low-sample regimes (e.g., $N < 100$), the trust region variants consistently achieved lower cumulative costs than heuristic variants. The combination of Trust Region and LCD sampling (dsTR) outperformed all other methods, including dsCEM and heuristic MPPI.
• Convergence: Trust region variants demonstrated faster convergence, reaching lower median cumulative costs with fewer iterations. In the Truck Backer-Upper task, heuristic variants failed to reach the cost achieved by trust region variants even after ten iterations, whereas trust region variants achieved this with only three.
• Control Smoothness: The dsTR method generally achieved the best balance of low cost and high smoothness. While some heuristic variants with random sampling showed slightly smoother controls in specific cart-pole scenarios, dsTR maintained superior overall performance.
• Runtime: Trust region updates incur a higher computational cost per step than heuristic methods due to the additional dual optimization step. However, the improved sample efficiency means fewer samples are required to achieve a target performance, which can offset runtime costs, especially on parallel hardware.

Significance and Claims

The authors claim that the proposed trust region formulation provides a principled alternative to heuristic parameter adaptation in sampling-based MPC. By deriving update weights from a constrained optimization problem, the method ensures stable learning and prevents aggressive updates that could degrade performance.

The paper emphasizes that the combination of trust region updates with deterministic LCD sampling is particularly effective for resource-constrained applications. It achieves faster convergence and better sample efficiency in low-sample and low-iteration regimes without sacrificing control smoothness. The authors suggest this approach is well-suited for real-world applications where reliable convergence and sample efficiency are critical, noting that reducing the required sample size directly translates to reduced computation time on parallel hardware.