Does "Do Differentiable Simulators Give Better Policy Gradients?'' Give Better Policy Gradients?

1. Problem Statement

In Policy Gradient (PG) reinforcement learning, access to a differentiable simulator allows for the use of 1st-order gradient estimators (via the reparameterization trick), which typically offer lower variance and faster convergence than 0th-order estimators (like REINFORCE). However, real-world robotics tasks often involve discontinuous dynamics (e.g., contact, friction, collisions).

• The Core Conflict: In the presence of discontinuities, 1st-order estimators become biased.
• The "Empirical Bias" Phenomenon: Prior work (Suh et al., 2022) identified that in finite samples, a biased 1st-order gradient can exhibit spuriously low empirical variance. Standard Inverse Variance Weighting (IVW) methods, which mix 0th and 1st-order estimators based on estimated variance, fail in these scenarios. They incorrectly trust the biased 1st-order estimator because its empirical variance appears low, leading to poor performance.
• Limitations of Existing Solutions: The state-of-the-art solution, Alpha-order Batched Gradient (AoBG), attempts to detect these biases by constructing confidence intervals around the noisy REINFORCE (0th-order) estimator. However, AoBG suffers from:
  1. High Noise: Relying on REINFORCE for the confidence interval makes the detection mechanism itself noisy.
  2. Poor Sample Efficiency: The intervals are broad, requiring large sample sizes.
  3. Hyperparameter Sensitivity: It requires extensive, task-specific tuning of a threshold parameter ($\gamma$).

The paper asks: Is empirical bias the primary obstacle to using differentiable simulators in practice, and what are the minimal fixes required?

2. Methodology

The authors propose two distinct methods to address these issues, targeting different regimes of the problem.

A. Discontinuity Detection Composite Gradient (DDCG)

Goal: To provide a robust, low-hyperparameter method for detecting discontinuities and switching estimators, specifically addressing the "empirical bias" problem in controlled settings.

• Mechanism: DDCG retains the IVW mixing scheme but introduces a lightweight statistical gate to decide whether to trust the 1st-order estimator.
• The Gate Logic: The method checks two assumptions required for IVW to be reliable:
  1. Reliable Variance: The empirical variance of the 1st-order gradient is close to its true variance.
  2. Local Smoothness: The function is locally well-behaved (near-quadratic).
• The Test: Instead of using the noisy REINFORCE estimator to build confidence intervals (as AoBG does), DDCG compares the empirical gradient variance against a theoretical bound derived from the function value variance (assuming a locally quadratic model).
  • If the empirical gradient variance is significantly lower than what the function value variance implies (suggesting heavy-tailed errors or discontinuities), the test fails.
  • Action: If the test passes, IVW is applied. If it fails, the method falls back to the pure 0th-order estimator ($\alpha = 0$).
• Advantages:
  • Efficiency: The test statistic is $O(d)$ times less noisy than AoBG's statistic (where $d$ is dimension), as it avoids the score-function term.
  • Robustness: It uses a single unified hyperparameter ($c$) and remains effective even with small sample sizes.

B. Stepwise Inverse Variance Weighting (IVW-H)

Goal: To determine if explicit discontinuity detection is necessary for practical robotics control, or if variance control alone suffices.

• Mechanism: IVW-H implements a per-step, per-action inverse variance weighting.
  • Instead of mixing gradients at the episode or batch level, it computes empirical variances across parallel actors (rollouts) for each time step and each action dimension.
  • It calculates a weight $\hat{\alpha}_{t,a}$ for every $(t, a)$ pair: $\hat{\alpha}_{t,a} = \frac{\hat{V}_0}{\hat{V}_0 + \hat{V}_1}$.
• Key Innovation: This approach isolates the role of variance control. It does not include an explicit discontinuity detection gate.
• Rationale: The authors hypothesize that in many standard robotics benchmarks, the primary bottleneck is high variance, not empirical bias. By stabilizing variance at the finest granularity (step-level), the method may succeed without needing to detect bias.

3. Key Contributions

1. Re-evaluation of Empirical Bias: The paper confirms that empirical bias is a real phenomenon that causes IVW to fail in explicitly discontinuous settings (e.g., "Ball with Wall," "Tennis").
2. DDCG Algorithm: Introduces a statistically rigorous, low-noise discontinuity detector that outperforms AoBG in robustness and sample efficiency, requiring no task-specific tuning.
3. IVW-H Implementation: Provides a clear, efficient, per-step implementation of IVW that matches or exceeds state-of-the-art baselines (like GIPPO) on standard MuJoCo-style tasks.
4. Empirical Insight: Demonstrates that while bias detection is crucial for explicitly discontinuous toy problems, variance control is the dominant factor in practical, high-dimensional robotics tasks. Explicit bias detection (like AoBG) often fails to outperform simple variance weighting in these practical settings.

4. Experimental Results

Part I: Explicit Empirical Bias Regimes (Toy Tasks)

• Tasks: "Ball with Wall," "Momentum Transfer," "Pushing" (soft/stiff collisions), "Friction," and "Tennis."
• Findings:
  • IVW and 1st-order only fail near discontinuities due to bias.
  • AoBG works but is highly sensitive to the hyperparameter $\gamma$. In low-sample regimes, it becomes overly conservative (reverting to 0th-order too often).
  • DDCG achieves comparable or better performance than AoBG with a single fixed parameter ($c=0.3$). It maintains robustness even with very small sample sizes (e.g., $N=3$ or $N=10$) where AoBG degrades.

Part II: Practical Continuous Control (MuJoCo Benchmarks)

• Tasks: CartPole, Hopper, Ant (with modified high-contact stiffness).
• Findings:
  • IVW-H achieves the best or near-best performance across all tasks.
  • AoBG does not outperform standard IVW, even with extensive hyperparameter tuning.
  • GIPPO (a complex state-of-the-art baseline) performs similarly to IVW but is computationally heavier.
  • Conclusion: In these practical settings, the "empirical bias" is not the primary bottleneck. The variance reduction provided by step-wise IVW is sufficient to handle the non-smoothness. The explicit bias detection mechanism of AoBG is unnecessary and adds no value in these domains.

5. Significance and Conclusion

• Theoretical Clarification: The paper resolves a debate on whether differentiable simulators are useful in non-smooth environments. The answer is yes, provided variance is managed correctly.
• Practical Guidance:
  • For highly discontinuous, low-sample problems (e.g., specific contact-rich optimization), use DDCG to safely switch estimators.
  • For standard robotics control (MuJoCo), use IVW-H. It is computationally efficient, requires no bias detection logic, and yields superior results by focusing on variance stabilization.
• Methodological Impact: The work highlights that "empirical bias" (low variance but high error) is a specific failure mode of variance-based mixing, but it is not the only or dominant failure mode in practical RL. Careful variance control at the step level often suffices to overcome the challenges of differentiable simulation in robotics.

In summary, the paper argues that while bias detection is theoretically important for specific pathological cases, variance control is the practical key to unlocking the potential of differentiable simulators in reinforcement learning.