Robust Adversarial Policy Optimization Under Dynamics Uncertainty

1. Problem Statement

Reinforcement Learning (RL) policies often fail when deployed in real-world environments due to dynamics mismatch (the gap between training simulation and reality). While standard RL maximizes nominal returns, it is brittle to aleatoric (stochastic) and epistemic (model uncertainty) variations.

Existing approaches have limitations:

• Domain Randomization (DR): Samples environments uniformly, treating all perturbations equally. This leads to inefficient training and overly conservative policies.
• Adversarial RL: Often uses explicit adversarial agents or hard-minimum sampling. This can cause training instability, oscillatory dynamics, and fails to provide distributional guarantees (blind spots in the ambiguity set).
• Distributionally Robust RL (DR-RL): Formulates the problem as a minimax game over a set of transition kernels (e.g., a KL-divergence ball). However, solving the primal problem directly is computationally intractable. Existing dual formulations often rely on surrogate adversaries that approximate the worst-case distribution, leading to over-conservatism or instability.

The Core Challenge: How to design an RL algorithm that is robust to dynamics uncertainty without collapsing into extreme conservatism or suffering from training instability, while maintaining computational tractability.

2. Methodology: Robust Adversarial Policy Optimization (RAPO)

The authors propose RAPO, a framework that realizes distributionally robust RL by combining a dual formulation with two complementary components: trajectory-level exponential tilting and model-level Boltzmann reweighting.

A. Theoretical Foundation: Dual Formulation

Instead of solving the primal minimax problem directly ($\inf_p \mathbb{E}_p[V]$), RAPO utilizes the convex dual of the KL-constrained robust expectation.

• Primal: $\inf_{p: D_{KL}(p\|\hat{p}) \leq \epsilon} \mathbb{E}_p[V]$
• Dual: $\sup_{\eta \geq 0} \left\{ -\frac{1}{\eta} \log \mathbb{E}_{\hat{p}}[e^{-\eta V}] - \frac{\epsilon}{\eta} \right\}$
  Here, $\hat{p}$ is the nominal transition kernel, and $\eta$ is a dual temperature parameter. This transforms the infinite-dimensional optimization over distributions into a tractable scalar optimization. The term $-\frac{1}{\eta} \log \mathbb{E}[e^{-\eta V}]$ represents an entropic risk (exponential tilting), which naturally emphasizes low-return (adverse) outcomes without requiring hard sampling.

B. Component 1: AdvNet (Trajectory-Level Robustness)

Directly solving for the optimal $\eta^*$ via root-finding at every step is computationally prohibitive.

• Mechanism: RAPO introduces an Adversarial Network (AdvNet), a feedforward neural network that takes the current state (and optionally action) as input and predicts the dual temperature $\eta_t$.
• Function: AdvNet amortizes the search for $\eta^*$. It steers rollouts toward low-return regions consistent with the KL budget $\epsilon$.
• Stability: The network is trained to satisfy the KL constraint via a penalty term. Crucially, the paper proves that even with approximate $\eta$, the robust Bellman operator remains a $\gamma$-contraction, ensuring convergence to a unique fixed point. The approximation error in value scales quadratically with the error in $\eta$.

C. Component 2: Boltzmann Reweighting (Model-Level Robustness)

AdvNet handles local trajectory fragility, but global model misspecification requires addressing the ensemble of dynamics models.

• Mechanism: RAPO maintains an ensemble of $K$ deterministic simulators with varied physical parameters (e.g., mass, friction). Instead of uniform sampling, it employs Boltzmann reweighting over this ensemble.
• Function: The weights $w$ of the ensemble models are updated to minimize the expected return subject to a KL budget $\kappa$ relative to a prior distribution. This results in a soft-minimum distribution:
  $$w^*_\beta(k) \propto \rho(k) \exp(-\beta h_k)$$
  where $h_k$ is a vulnerability score (e.g., low value estimates) for model $k$, and $\beta$ is a temperature adjusted to satisfy the KL constraint.
• Benefit: This focuses training on "adverse" models where the policy currently underperforms, providing policy-sensitive coverage of the dynamics space.

D. Unified Algorithm

RAPO integrates these components into a PPO-compatible loop:

1. Rollout: Collect data under the nominal environment.
2. AdvNet Update: Predict $\eta$ to compute robust Bellman targets (exponential tilting).
3. Actor-Critic Update: Update the policy using PPO with the robust targets.
4. Ensemble Update: Update the mixture weights $w$ via Boltzmann reweighting based on current policy performance across the ensemble.

3. Key Contributions

1. Dual Formulation for Robust RL: The paper establishes a practical realization of distributionally robust RL using a dual formulation that explicitly exposes the robustness-performance trade-off via the temperature parameter $\eta$.
2. Two-Layer Architecture: It introduces a novel two-layer structure:
   • Trajectory-level: AdvNet for local, state-action dependent exponential tilting.
   • Model-level: Boltzmann reweighting for global coverage of adverse dynamics in an ensemble.
     These components act independently yet complementarily to guard against both local fragility and global misspecification.
3. Theoretical Guarantees:
   • Proves that the robust Bellman operator remains a contraction even with approximate $\eta$.
   • Shows the value gap between nominal and robust policies scales as $O(\sqrt{\epsilon})$, justifying $\epsilon$ as a tunable knob.
   • Demonstrates convergence of the finite-ensemble approximation to the continuous case at a Monte Carlo rate ($O(K^{-1/2})$).
4. Practical Scalability: The method amortizes the dual optimization via neural networks, making it scalable and compatible with standard actor-critic algorithms like PPO.

4. Experimental Results

The authors evaluated RAPO on Walker2d (locomotion) and a Quadrotor Payload Tracking task (coupled dynamics).

• Robustness to Dynamics Uncertainty:
  • RAPO was tested under significant parameter shifts (mass, inertia, friction, torque) ranging from 0.5x to 1.5x of nominal values.
  • Baselines: Compared against PPO (nominal), PPO+DR (Domain Randomization), RARL (Adversarial RL), and EPOpt (Risk-averse).
  • Outcome: RAPO maintained high returns in Out-of-Distribution (OOD) regimes where PPO collapsed and PPO+DR degraded significantly. RARL was unstable, and EPOpt was overly conservative. RAPO achieved the best balance, sustaining performance even at extreme scales (e.g., 0.5x or 1.5x mass).
• Ablation Studies:
  • Removing AdvNet (relying only on root-finding) reduced robustness and increased variance.
  • Removing Boltzmann Reweighting (uniform sampling) led to brittle tails in OOD regions.
  • Removing both resulted in the steepest performance drop, confirming the necessity of both components.
• Quadrotor Payload Task:
  • In a complex task involving a suspended payload with random disturbances, RAPO achieved 0% crash rate in OOD environments, whereas PPO-DR had a 40% crash rate.
  • RAPO significantly reduced trajectory oscillations and steady-state error compared to all baselines.

5. Significance

RAPO bridges the gap between theoretical distributional robustness and practical RL deployment.

• Solves the "Blind Spot" Problem: Unlike domain randomization (which treats all perturbations equally) or pure adversarial training (which may miss the true worst-case distribution), RAPO mathematically guarantees coverage of the ambiguity set via the dual formulation.
• Stability vs. Conservatism: It avoids the over-conservatism of worst-case optimization by using soft-minima (exponential tilting) and avoids the instability of hard adversarial agents.
• Generalization: By focusing on policy-sensitive adverse dynamics, RAPO produces policies that generalize effectively to unseen environments without retraining, a critical requirement for safety-critical robotics applications.

In summary, RAPO provides a theoretically grounded, scalable, and empirically superior framework for training RL agents that are resilient to dynamics uncertainty.