RoAd-RL: A Unified Library and Benchmark for Robust Adversarial Reinforcement Learning

Technical Summary: RoAd-RL

Problem Statement
Deep Reinforcement Learning (DRL) has achieved significant success in robotics and autonomous systems but remains vulnerable to adversarial perturbations that can induce unsafe behavior and degrade performance. While numerous adversarial attacks (e.g., FGSM, PGD, JSMA) and defense mechanisms (e.g., adversarial training, temporal smoothing) have been proposed, the field of adversarial reinforcement learning suffers from fragmentation. Research is currently hindered by custom, non-reproducible implementations, inconsistent evaluation protocols, and a lack of standardized benchmarks. Unlike the supervised learning community, which benefits from unified frameworks like RobustBench and Foolbox, adversarial RL lacks a widely adopted infrastructure that offers comparable extensibility and reproducibility.

Methodology and Framework Design
To address these limitations, the authors present RoAd-RL, an open-source, unified benchmarking framework designed for Robust Adversarial Reinforcement Learning. The framework is built on three core design principles:

1. Composability: Attacks, defenses, policies, and metrics function as independent components within a common evaluation pipeline.
2. Extensibility: All major components inherit from abstract base classes, allowing researchers to integrate new methods via subclassing without modifying the framework core.
3. Reproducibility: Deterministic seed management, structured logging, and standardized procedures ensure consistent results across environments.

The framework architecture consists of four primary abstractions:

• Policy: A unified interface for RL agents (currently supporting Stable-Baselines3 implementations of DQN, PPO, and SAC) that decouples learning algorithms from evaluation. It includes a DifferentiablePolicy interface for gradient-based attacks.
• Attack: Encapsulates adversarial perturbation generation (e.g., FGSM, PGD, JSMA) operating in observation space.
• Defense: Represents inference-time mechanisms (e.g., temporal smoothing, feature squeezing, outlier clipping) that process observations before policy inference.
• Metric: Standardizes robustness reporting, including Return Metrics (mean/median return), Risk Metrics (CVaR), Safety Metrics (termination rates), and Robustness Metrics (Area Under the Curve, AUC–$\epsilon$).

The evaluation workflow follows three stages: (1) Policy training within a selected environment, (2) Composition of the trained policy with specific attack and defense modules, and (3) Robustness evaluation across multiple perturbation budgets ($\epsilon$), random seeds, and episodes. The framework integrates seamlessly with Stable-Baselines3 and Gymnasium, offering both Python API and Command-Line Interface (CLI) execution.

Experimental Evaluation
The authors validated RoAd-RL by evaluating DQN, PPO, and SAC agents in two distinct environments: LunarLander-v2 (compact, sensitive dynamics) and Highway-v0 (structured, kinematic features). The benchmark covered a full factorial grid of 192 configurations, combining:

• Algorithms: DQN, PPO, SAC.
• Attacks: FGSM, PGD, JSMA, and a clean baseline.
• Defenses: No defense, temporal smoothing, median smoothing, Gaussian noise, feature squeezing, outlier clipping (MAD), and adversarial detection.
• Perturbation Budgets: $\epsilon \in \{0.0, 0.005, 0.01, 0.025, 0.05\}$.

Performance was measured using normalized AUC–$\epsilon$ (Area Under the Return–$\epsilon$ curve), where 1.0 represents the clean baseline performance.

Key Results

• Environment Dependency: Robustness varies significantly across environments. LunarLander agents were highly vulnerable to perturbations, whereas Highway-v0 agents remained largely robust to attacks.
• Algorithm Sensitivity: SAC agents on LunarLander were particularly vulnerable to iterative PGD attacks (AUC dropping to 0.590), likely due to the exploitation of smooth policy gradients. DQN showed clear degradation under single-step FGSM.
• Defense Effectiveness:
  • Temporal Smoothing: Consistently achieved the strongest performance, particularly for SAC under PGD on LunarLander, recovering approximately 27% of performance relative to the undefended baseline.
  • Feature Squeezing: Often proved detrimental in low-dimensional control tasks (e.g., reducing DQN's AUC from 0.793 to 0.547 under FGSM), suggesting it is better suited for high-dimensional image observations.
  • Defense-Induced Failure: Several defenses (Outlier Clip, Adversarial Detector) degraded performance more than the attacks they aimed to mitigate, particularly for continuous-action agents. In some cases, the defense alone caused greater performance loss than the attack.
• PPO Anomalies: PPO on LunarLander exhibited AUC values >1.0 under certain attacks, which the authors attribute to the policy operating near a weak baseline where small perturbations acted as implicit noise, temporarily improving returns before eventual collapse.

Significance and Claims
The paper claims that RoAd-RL establishes a standardized benchmark for adversarial reinforcement learning, filling a critical gap in reproducibility and extensibility. By providing a unified framework that standardizes the interaction between policies, attacks, defenses, and metrics, RoAd-RL enables fair comparisons and systematic analysis of attack-defense interactions.

The authors emphasize that their results highlight the necessity of evaluating defenses not only by their ability to mitigate attacks but also by their impact on nominal policy performance, as some common defenses can be more harmful than the threats they address. RoAd-RL is publicly available as a Python package to facilitate community-driven development of robustness evaluation methodologies. Future work is outlined to include support for black-box attacks, adversarial training, certified robustness, and multi-agent environments.