Evaluating Fuzz Testing for Reinforcement Learning Agents

Technical Summary: Evaluating Fuzz Testing for Reinforcement Learning Agents

Problem Statement
Reinforcement Learning (RL) agents are increasingly deployed in safety-critical domains such as autonomous driving, robotics, and drone control. While these agents learn decision-making policies through interaction, they remain susceptible to unexpected behaviors and crashes when exposed to rare or complex environmental conditions. Fuzz testing has emerged as a primary method for exploring the vast state spaces of RL agents to expose such failures. However, the field lacks a comprehensive, standardized empirical evaluation. Existing studies often differ in baselines, evaluation metrics, and experimental configurations, making it difficult to determine the relative effectiveness, efficiency, and practical utility of different fuzzing strategies. Furthermore, prior work often focuses solely on crash discovery counts, neglecting how detected crashes contribute to downstream tasks like agent robustness improvement and safety monitoring.

Methodology
To address these gaps, the authors conducted the first large-scale, systematic empirical study evaluating RL fuzzing methods. The study design involved:

• Benchmarks: Three environments of increasing complexity were selected: MountainCar (low-dimensional control), BipedalWalker (high-difficulty continuous control), and CARLA (high-fidelity autonomous driving). Robust RL agents were selected for each to ensure that performance differences stemmed from the fuzzing methods rather than intrinsic agent vulnerabilities.
• Subjects: Five state-of-the-art fuzzing methods were benchmarked: MDPFuzz, CureFuzz, Generative Model-Based Testing (G-Model), SeqDivFuzz, and QDFuzz. A fundamental Random Testing strategy was included as a baseline to provide a stochastic lower bound.
• Implementation: All methods were re-implemented or updated to their latest versions under unified configurations to ensure fairness. Experiments were repeated five times with distinct random seeds.
• Evaluation Dimensions: The study evaluated methods across four complementary perspectives:
  1. Effectiveness: Measured by the number of valid unique crashes (UC) detected within a fixed 12-hour time budget.
  2. Diversity: Assessed via Input Diversity (clustering of environmental configurations) and Output Diversity (clustering of execution trajectories).
  3. Efficiency: Measured by UD-AUC (temporal efficiency of crash discovery) and AG/UC (average generations/mutations required per unique crash).
  4. Practical Utility: Evaluated through two downstream tasks: Agent Robustness Improvement (fine-tuning agents on crash data to reduce future crash rates) and Safety Monitoring (training classifiers to predict crash-prone episodes).

Key Results
The study yielded several distinct findings regarding the strengths and trade-offs of current fuzzing strategies:

• Effectiveness and Efficiency: MDPFuzz consistently outperformed all other methods across all benchmarks in both crash discovery volume and time efficiency. Its lightweight, throughput-oriented design allowed for rapid exploration. Conversely, SeqDivFuzz and CureFuzz exhibited lower efficiency, often terminating exploration prematurely due to aggressive seed pruning or high computational overhead.
• Diversity: Methods explicitly designed to encourage exploration, such as SeqDivFuzz and QDFuzz, uncovered more diverse crash behaviors (higher Output Diversity) compared to throughput-oriented methods. MDPFuzz, while highly effective at finding crashes, often triggered highly similar crash modes, resulting in lower diversity scores.
• The Role of Random Testing: Random testing performed surprisingly well, particularly in low-dimensional tasks, often ranking second only to MDPFuzz. This highlights that complex guidance mechanisms are not always necessary for simple tasks and that random testing is a critical baseline often overlooked in prior literature.
• Complementarity: Different fuzzing methods explored largely non-overlapping regions of the state space. The intersection of crash-inducing initial seeds between methods was nearly zero, indicating that no single method covers all testing blind spots.
• Downstream Utility:
  • Robustness: Fine-tuning agents on fuzzing-generated crashes improved robustness for most methods. QDFuzz yielded the highest robustness gains (up to 41.5% reduction in crashes) due to its focus on diverse samples. However, fine-tuning on crashes from certain methods (e.g., CureFuzz) sometimes led to regression faults or reduced robustness against other fuzzers, suggesting overfitting.
  • Safety Monitoring: Models trained on crash-triggering inputs from a single fuzzer achieved high accuracy (>95%) even when tested on crashes generated by other fuzzing methods. This indicates that crash behaviors share common underlying characteristics that generalize well across different testing strategies.

Significance and Contributions
The paper claims three primary contributions:

1. Systematic Empirical Study: It provides the first comprehensive benchmarking of RL fuzzing methods across effectiveness, diversity, efficiency, and practical utility, establishing a unified evaluation protocol.
2. Actionable Insights: The study reveals inherent trade-offs (e.g., between crash quantity and diversity) and demonstrates that combining complementary strategies (e.g., high-throughput methods with diversity-driven methods) yields more comprehensive coverage. It also validates the practical value of fuzzing data for downstream safety tasks.
3. Reproducibility: The authors re-implemented and standardized the compared methods, constructing a reproducible evaluation pipeline and publicly releasing all code, datasets, and artifacts.

Actionable Takeaways
The authors distill specific guidance for researchers and practitioners:

• Select Fuzzers by Goal: Use MDPFuzz for maximum crash discovery in time-limited scenarios; use QDFuzz or SeqDivFuzz when diverse crash samples are needed for robustness training; use G-Model for mining long-tail risks in complex simulators.
• Combine Strategies: Due to low overlap in discovered crashes, practitioners should integrate multiple fuzzing pipelines to expand coverage.
• Adopt Multi-Level Metrics: Diversity should be evaluated from both input and output perspectives, as high input diversity does not guarantee diverse agent behaviors.
• Standardize Evaluation: Future research must include strong baselines (including random testing), test on tasks of varying complexity (including realistic scenarios like CARLA), and assess practical utility beyond simple crash counts.
• Design for Scalability: Future fuzzers should prioritize lightweight, task-agnostic designs to avoid the high adaptation costs associated with current methods that rely on handcrafted features or specific environment constraints.