Technical Summary: Stabilizing Policy Gradient Methods via Reward Profiling

Problem Statement

Policy Gradient (PG) methods are a cornerstone of reinforcement learning (RL), offering a direct framework for optimizing parameterized policies in high-dimensional and continuous-control settings. However, these methods frequently suffer from high variance in gradient estimations. Small stochastic fluctuations in early-episode rewards can propagate through Monte Carlo estimators (e.g., in REINFORCE), leading to erratic updates, slow convergence, and occasional catastrophic performance collapse.

Existing variance-reduction techniques often introduce significant drawbacks:

• Algorithm-specific tuning: Methods like baselines or advantage normalization require problem-specific adjustments.
• Computational overhead: Trust-region methods (e.g., TRPO) rely on expensive second-order solvers or Hessian computations.
• Complexity: Off-policy actor-critic algorithms (e.g., DDPG, TD3) often require careful regularization to prevent divergent updates or spiky Q-value estimates.

The authors pose a central question: Is it possible to reduce PG variance and stabilize learning across arbitrary policy-gradient methods without relying on specialized tuning or heavy second-order machinery?

Methodology: The Reward Profiling Framework

The paper proposes a universal reward-profiling framework that acts as a "wrapper" around any existing PG algorithm. The core philosophy is straightforward: selectively update the policy only if the updated version implies a better performance.

Since the true value function $J(\theta)$ is unknown, the framework relies on high-confidence comparisons using empirical return estimates. The framework operates by sampling a small number of additional rollouts ($E$) to evaluate candidate policies before committing to an update. It introduces three specific variants:

1. Lookback (LB):

   • Compares the current policy $\pi_{old}$ with the newly proposed policy $\pi_{new}$.
   • Updates the policy only if the empirical return $\hat{J}(\pi_{new}) \geq \hat{J}(\pi_{old})$.
   • Otherwise, the update is rejected, and the agent retains the old policy. This acts as a backtracking line search in policy space.

2. Mix-up (MU):

   • Addresses the risk of getting stuck due to noisy estimates masking genuine improvements.
   • Introduces an intermediate policy $\pi_{mix}$ defined by a convex combination of parameters: $\theta_{mix} = \lambda \theta_{new} + (1-\lambda) \theta_{old}$.
   • Selects the best performing policy between $\pi_{old}$ and $\pi_{mix}$. This functions as a "cheap trust region" without requiring Hessian computations or KL constraints.

3. Three-Points (TP):

   • Unifies the previous approaches by evaluating three candidates: $\theta_{old}$, $\theta_{new}$, and $\theta_{mix}$.
   • Selects the parameter set yielding the highest empirical return.
   • Designed to accelerate convergence while maintaining stability and providing room for exploration.

Algorithmic Integration:
The framework (Algorithm 1) wraps any first-order PG method. It requires $O(\epsilon^{-2} \ln(T/\delta))$ additional rollouts per iteration to ensure high-confidence comparisons but does not alter the core update logic of the underlying optimizer (e.g., DDPG, TRPO, PPO).

Theoretical Contributions

The authors provide theoretical guarantees regarding the convergence and stability of the framework:

• Monotonicity: Under the Lookback rule, with a sufficient number of evaluation samples ($E$), the framework ensures that accepted updates yield true performance improvements with high probability ($1-\delta$). Specifically, $J(\pi_{t+1}) \geq J(\pi_t) - 2\epsilon$.
• Convergence Rate: When applied to REINFORCE, the framework converges to a near-optimal policy at a rate of $O(T^{-1/4})$. This rate is comparable to stochastic actor-critic algorithms, demonstrating that the profiling overhead does not significantly slow down convergence.
• Global Optimality: Under standard smoothness assumptions and gradient dominance properties, the framework converges to a stationary point, implying global optimality for the learned policy.
• Bias Handling: The analysis extends to biased cases (common in actor-critic methods where critics are approximated), showing convergence as long as the bias in value estimation is bounded and decreases sufficiently over time.

Empirical Results

The framework was evaluated on eight continuous-control benchmarks (Box2D and MuJoCo/PyBullet suites) using three representative algorithms: DDPG, TRPO, and PPO.

• Convergence Speed: The profiling approach yielded up to 1.5× faster convergence to near-optimal returns in several setups.
• Variance Reduction: It achieved up to a 1.75× reduction in return variance on specific configurations, significantly stabilizing training curves.
• Specific Environment Performance:
  • CarRacing: Transformed catastrophic failures (negative returns due to crashing) into viable policies with positive returns for both TRPO and PPO.
  • BipedalWalker: Improved DDPG performance from $-116.1$ to $-47.0$ (average return).
  • Ant: DDPG with Mix-up showed earlier convergence, while the Three-Points variant reduced variance by 11%.
  • Unity ML-Agents (Reacher): In a multi-robot task with 20 parallel arms, the Three-Points variant demonstrated substantially more stable learning and faster convergence compared to vanilla DDPG, validating the approach in a realistic, complex simulation.

Significance and Claims

The paper claims that Reward Profiling offers a general, theoretically grounded path to more reliable and efficient policy learning. Its primary significance lies in:

1. Universality: It is a plug-in wrapper applicable to any PG method (on-policy or off-policy) without requiring problem-specific baselines or hyperparameter tuning.
2. Simplicity vs. Stability: It achieves stability comparable to complex trust-region methods (like TRPO) but uses only first-order updates and minimal computational overhead (reusing samples for evaluation).
3. Robustness: It effectively mitigates the "catastrophic collapse" often seen in vanilla PG methods, particularly in high-dimensional or sparse-reward environments.

The authors conclude that while the framework incurs a modest computational cost (roughly 20% wall-clock time increase in their experiments), the trade-off is favorable for achieving stable, monotonic improvements, making it suitable for physical deployment where predictable learning behavior is critical.