Technical Summary: Model-Driven Policy Optimization in Differentiable Simulators via Stochastic Exploration

1. Problem Statement

Differentiable planning enables gradient-based optimization of decision-making problems by leveraging differentiable models of system dynamics. While effective in deterministic settings, this approach faces significant challenges when applied to highly nonlinear and hybrid discrete–continuous domains.

The core issue lies in the optimization landscape. To enable gradient propagation in hybrid domains, discrete or logical structures must be approximated via smooth relaxations (e.g., t-norms). This creates a fundamental trade-off:

• High Fidelity Relaxations: Closely approximate the original problem but result in ill-conditioned landscapes characterized by flat regions, sharp transitions, and discontinuities. In these regions, gradients are either undefined or uninformative, causing optimizers to get trapped in poor local minima.
• Aggressive Smoothing: Improves gradient behavior and optimization tractability but distorts the underlying problem geometry, potentially leading to solutions that are suboptimal or infeasible under the true dynamics.

Consequently, standard deterministic differentiable planning often struggles with slow convergence, sensitivity to initialization, and failure to escape poor local optima, even when the underlying model is accurate. Existing model-free reinforcement learning (RL) methods, while incorporating exploration, fail to explicitly exploit known system structure, leading to poor sample efficiency in domains where models are available.

2. Methodology: Model-Driven Policy Optimization (MDPO)

The authors propose Model-Driven Policy Optimization (MDPO), a framework that integrates stochastic exploration directly into the differentiable planning process. Unlike standard RL which explores via policy parameter noise or environment interaction, MDPO injects noise into the action space during the optimization of the policy parameters.

2.1 Stochastic Action Perturbation

Instead of executing actions deterministically ($a_t = \pi_\theta(s_t)$), MDPO defines a stochastic policy:
$$\tilde{a}_t = \pi_\theta(s_t) + \epsilon_t, \quad \epsilon_t \sim \mathcal{N}(0, \Sigma_t)$$
The system evolves according to the dynamics $s_{t+1} = f(s_t, \tilde{a}_t, \xi_t)$. The optimization objective becomes the expected cumulative reward under this perturbed policy. Using the reparameterization trick, the noise $\epsilon_t$ is decoupled from the policy parameters $\theta$, allowing gradients to be computed via standard backpropagation through the stochastic computation graph. This enables the optimizer to probe regions of the objective landscape that are inaccessible to deterministic gradients, particularly in flat or saturated regions.

2.2 Adaptive Noise Scaling

A key innovation of MDPO is the adaptive, model-driven scaling of the noise magnitude. Since the planning model is fully available, the method leverages gradient-derived sensitivity to construct a time-dependent exploration profile $\{\sigma_t\}_{t=0}^{H-1}$.

The process operates in a two-pass procedure per iteration:

1. Analysis Rollout: A trajectory is generated using the current policy without action noise. The sensitivity of the objective with respect to actions at each timestep is estimated via the gradient magnitude: $g_t = \nabla_{a_t} \sum r(s_k, a_k)$. A scalar sensitivity score $s_t = \|g_t\|^2$ is computed.
2. Noise Profile Construction: The sensitivity scores are normalized and mapped to a noise scale $\sigma_t$ using a function such that:
   $$\sigma_t = \sigma_{\max} + (\sigma_{\min} - \sigma_{\max})(1 - \hat{s}_t)^\alpha$$
   This mapping assigns larger noise to timesteps with higher sensitivity (where actions have a greater impact on the trajectory objective) and lower noise to insensitive regions.
3. Update Rollout: A second rollout is performed using the derived noise profile to update the policy parameters.

This approach ensures that exploration is structure-aware, respecting system dynamics and targeting regions of high objective sensitivity, rather than applying isotropic noise blindly.

3. Key Contributions

1. Identification of Optimization Pathologies: The paper formally identifies and analyzes the ill-conditioned nature of optimization landscapes in differentiable planning, specifically highlighting the trade-off between relaxation fidelity and gradient tractability in hybrid domains.
2. Stochastic Exploration Mechanism: The proposal of MDPO, which introduces structured, model-driven stochasticity into the action space. Unlike standard stochastic optimization, the perturbations are applied at the trajectory level within the planning model.
3. Adaptive Noise Scaling: The development of a mechanism to dynamically adjust noise magnitude based on local trajectory sensitivity, enabling a time-dependent exploration profile that adapts to both the optimization landscape and the specific timestep within the horizon.
4. Empirical Validation: Demonstration of substantial improvements in solution quality and convergence speed across benchmark problems compared to deterministic variants and state-of-the-art baselines.

4. Experimental Results

The method was evaluated on three benchmark domains from the 2023 International Planning Competition (IPC): PowerGen (unit commitment), HVAC (nonlinear control), and Reservoir Control. These domains feature nonlinear and hybrid discrete–continuous dynamics.

Baselines:

• Deterministic Differentiable Planning: The noise-free variant of MDPO.
• JaxPlan: A state-of-the-art implementation of differentiable planning using relaxed hybrid dynamics.
• Constant Noise MDPO: Variants with fixed Gaussian noise ($\sigma \in \{1, 3\}$).
• PPO (Proximal Policy Optimization): A model-free RL baseline.

Findings:

• Performance: MDPO with adaptive noise consistently outperformed all baselines across all instances. It achieved significantly higher solution quality (lower costs/higher rewards) compared to deterministic planning, JaxPlan, and constant noise variants.
• Convergence: While deterministic methods and JaxPlan quickly plateaued in poor local optima, and PPO showed minimal improvement due to the lack of a model, adaptive MDPO demonstrated sustained improvement and faster convergence.
• Variance: MDPO exhibited low variance across random seeds, maintaining high solution quality.
• Adaptive Behavior: Analysis of the noise profiles revealed that the adaptive mechanism dynamically allocates exploration. For example, in the Reservoir domain, noise was concentrated in early timesteps, while in HVAC, it varied non-monotonically, reflecting the specific sensitivity structure of each problem.

5. Significance and Claims

The paper posits that the primary limitation of differentiable planning in complex domains is not merely the modeling or the relaxation technique, but the optimization dynamics used to solve the resulting problems.

• Optimization Dynamics: The authors argue that deterministic gradient updates are insufficient for navigating the ill-conditioned landscapes of hybrid domains. By introducing stochastic exploration, MDPO allows the optimizer to escape poor local minima and traverse sharp transitions.
• Model-Driven vs. Model-Free: The work highlights that when models are available, they should not only be used for simulation but also to guide the exploration process. MDPO bridges the gap between model-based gradient optimization and exploration-driven learning by using the model to derive informative exploration signals.
• General Approach: The paper claims that MDPO provides a simple, general approach to addressing optimization pathologies in model-based planning without requiring modifications to the underlying model or its relaxation. It suggests that tailoring exploration to the local structure of the optimization landscape is crucial for robust decision-making in nonlinear and hybrid settings.