Evolutionary Bilevel Reward Shaping for Generalization in Reinforcement Learning

Technical Summary: Evolutionary Bilevel Reward Shaping for Generalization in Reinforcement Learning

Problem Formulation
This paper addresses the generalization gap in Deep Reinforcement Learning (RL), where policies trained on limited environments fail to perform in unseen test environments with different dynamics (e.g., varying transition probabilities). The authors focus on a specific, constrained scenario common in privacy-preserving or restricted settings:

• Training Environments ($M_{train}$): A small, fixed set of environments where full state-action trajectories are observable.
• Validation Environments ($M_{val}$): A larger set of environments where only scalar performance metrics (cumulative undiscounted rewards) are available. Full trajectories are inaccessible, preventing the computation of policy gradients or standard domain randomization (DR) techniques that rely on diverse trajectory data.
• Objective: Maximize generalization performance ($J_{gen}$) on unseen test environments by utilizing the limited scalar feedback from validation environments to guide training on the accessible training environments.

Methodology: Generalization via Evolutionary Reward Shaping (GERS)
The authors propose GERS, a bilevel optimization framework designed to automatically shape the reward function to improve generalization without access to validation trajectories.

1. Bilevel Formulation:

   • Lower Level (Policy Optimization): An RL agent (using Proximal Policy Optimization, PPO) learns a policy $\pi_\theta$ on the training environments ($M_{train}$) using a shaped reward function $\hat{r}_\nu$. The objective is to maximize the discounted return on training data: $\theta^*(\nu) = \arg\max_\theta \hat{J}_{train}(\pi_\theta; \nu)$.
   • Upper Level (Reward Shaping Optimization): An evolutionary strategy (Covariance Matrix Adaptation Evolution Strategy, CMA-ES) optimizes the reward shaping parameters $\nu$. The objective is to maximize the cumulative unshaped reward obtained by the lower-level policy on the validation environments: $\max_\nu J_{val}(\pi_{\theta^*(\nu)})$.
   • The shaped reward is defined linearly: $\hat{r}_\nu(s, a) = r(s, a) + \langle \nu, \phi(s, a) \rangle$, where $\phi(s, a)$ concatenates state and action vectors.

2. Algorithmic Implementation:

   • Pretraining: A PPO agent is first pretrained on the original reward function to provide a robust initialization.
   • Evolutionary Search: CMA-ES samples candidate reward parameters $\nu$. For each candidate, the PPO agent is fine-tuned (using the pre-trained policy as a starting point) on the training environments with the shaped reward.
   • Evaluation: The resulting policies are evaluated on the validation environments to obtain scalar scores. These scores guide the CMA-ES update.
   • Efficiency: To reduce computational cost, the method employs parameter sharing (inheriting both Actor and Critic networks) from the pre-trained model and utilizes parallel evaluation of CMA-ES candidates.

Key Contributions

• Novel Setting: The paper formalizes and addresses the challenge of RL generalization where validation environments are "black boxes" providing only scalar returns, a constraint often ignored by standard Domain Randomization (DR) or Procedural Content Generation (PCG) methods.
• Bilevel Framework: It introduces a bilevel optimization approach that decouples policy learning (lower level) from reward function design (upper level), using evolutionary strategies to bridge the gap between training and validation performance.
• Data Efficiency: The method demonstrates that high generalization can be achieved without accessing trajectories in validation environments, relying solely on scalar feedback to steer reward shaping.

Experimental Results
The authors evaluated GERS on four MuJoCo continuous control tasks (InvertedPendulum, Hopper, HalfCheetah, Ant) with varying physical parameters (e.g., mass, friction, gravity).

• Baseline Comparison: Standard PPO trained on a single environment showed significant overfitting, achieving high training rewards but poor performance on unseen test environments.
• GERS Performance: GERS significantly outperformed the standard PPO baseline across all tasks, achieving generalization performance comparable to training performance in several cases (e.g., InvertedPendulum).
• Comparison with Domain Randomization (DR):
  • GERS was compared against a DR baseline that utilized the same set of environments as training data (with full trajectory access).
  • While DR performed best on complex tasks (HalfCheetah, Ant), GERS outperformed DR on simpler tasks (InvertedPendulum, Hopper).
  • Crucially, GERS achieved competitive robustness to DR despite operating under much stricter information constraints (no trajectory access in validation environments).
• Out-of-Distribution (OOD) Analysis: Heatmap evaluations showed GERS maintained strong robustness when one physical parameter was pushed into extreme OOD regions, though performance dropped when multiple parameters were simultaneously extreme.

Significance and Claims
The paper claims that GERS effectively enhances RL generalization under restricted data access constraints. Its primary significance lies in demonstrating that reward shaping optimized via evolutionary strategies on scalar validation feedback can serve as a viable alternative to Domain Randomization when trajectory data is unavailable. The authors posit that this approach is particularly relevant for real-world applications like automated red teaming or privacy-sensitive deployments, where developers can observe performance outcomes but cannot access the internal state transitions of the target systems.

Limitations Acknowledged
The authors modestly note several limitations:

• The evaluation is restricted to dense-reward continuous control tasks; applicability to sparse-reward problems is unverified.
• High computational costs limited the number of independent trials, affecting statistical power.
• The performance gain is a combination of reward shaping and the selection of the best policy from many stochastic runs; an ablation study to isolate the specific contribution of reward shaping is needed.
• The upper-level CMA-ES did not fully converge in all experiments, suggesting a need for noise-aware variants in future work.