Reinforcement Learning for Testing Interdependent Requirements in Autonomous Vehicles: An Empirical Study

1. Problem Statement

Autonomous Vehicles (AVs) operate in complex, dynamic environments where safety and functional requirements are often interdependent (e.g., violating a functional requirement like route completion might compromise safety by causing a collision).

• The Challenge: Current testing methods often treat requirements as independent or combine them into a single weighted reward function using Single-Objective Reinforcement Learning (SORL). It remains unclear whether SORL is sufficient for testing interdependent requirements or if Multi-Objective Reinforcement Learning (MORL), which explicitly optimizes multiple criteria simultaneously without pre-defined weights, is necessary.
• The Gap: There is a lack of systematic empirical evidence comparing SORL and MORL in the context of generating critical AV scenarios that violate multiple, interdependent requirements simultaneously.

2. Methodology

The authors conducted a large-scale empirical study comparing SORL and MORL using a high-fidelity simulation environment.

Experimental Setup

• Subject System: Interfuser, an advanced end-to-end AV controller based on the Transformer architecture, running on the CARLA high-fidelity simulator.
• Environment: Six distinct road structures (highways, two-way roads, intersections, and roads with damage) covering diverse traffic scenarios.
• Requirements & Objectives: Five interdependent requirements were selected and converted into objective functions:
  1. Distance (Dis): Safe distance from obstacles.
  2. Time to Collision (TTC): Temporal safety margin.
  3. Route Completion (RC): Functional ability to reach the destination.
  4. Jerk: Passenger comfort (acceleration stability).
  5. Speed Difference (SD): Adherence to surrounding traffic speed.
• Algorithms Compared:
  • MORL: Implemented using Envelope Q-learning (EQL). EQL learns a single policy network that can generalize across a space of weight vectors, dynamically adjusting the relative importance of objectives during training.
  • SORL: Implemented using Deep Q-learning (DQN) with a fixed, equal-weighted aggregation of all objective functions into a single scalar reward.
• Evaluation Metrics:
  • Effectiveness: Objective Value (OV), Single/Multi-requirement Violation Counts (#SV, #MV), Violation Severity (SVS, MVS), and Collision Count (#C).
  • Diversity: Unique Behavior Count (#UB), Behavior Diversity (UBD, WBD), and Scenario Diversity (SCD) using Dynamic Time Warping (DTW).

3. Key Contributions

1. Systematic Empirical Evaluation: The first comprehensive comparison of SORL and MORL specifically for testing interdependent AV requirements, moving beyond independent requirement testing.
2. MOMDP Formulation: Demonstrated how to formulate AV testing as a Multi-Objective Markov Decision Process (MOMDP) and solved it using EQL, providing a blueprint for similar software engineering problems.
3. Trade-off Analysis: Identified specific conditions (objective combinations and road structures) where one algorithm outperforms the other, offering preliminary guidelines for practitioners.

4. Key Results

Overall Performance (RQ1)

• Effectiveness: SORL and MORL show comparable overall effectiveness.
  • MORL generates a higher number of violation scenarios (both single and multi-requirement).
  • SORL tends to produce violations of higher severity (more extreme values in metrics like collision proximity or route failure).
• Diversity: MORL significantly outperforms SORL in scenario diversity. It generates a broader range of unique behaviors and physical scenario configurations, covering the search space more effectively.

Impact of Objective Combinations (RQ2)

• TTC Constraint: When the Time to Collision (TTC) objective is included, the optimization space is often strictly constrained by the AV controller. In these cases, the problem behaves like a single-objective task, where SORL performs slightly better.
• Interdependencies: When objectives have strong correlations or trade-offs (e.g., Jerk vs. Speed Difference), MORL leverages these relationships better, often finding solutions that SORL misses.
• Independence: For independent objectives (e.g., Distance vs. Route Completion), SORL showed a slight edge in specific combinations, likely due to the simplicity of the weighted sum.

Impact of Road Structures (RQ3)

• Diversity: MORL consistently maintains superior diversity across all road types, including complex intersections and damaged roads.
• Effectiveness: Neither algorithm shows a consistent advantage across all road structures. Performance varies based on specific traffic interactions (e.g., SORL performed better on roads with poor conditions in some metrics, while MORL excelled in complex intersections).

5. Significance and Implications

• The Exploration vs. Exploitation Trade-off: The study highlights a fundamental trade-off. MORL is superior for exploration, ensuring broad coverage of diverse failure modes and interdependencies. SORL is superior for exploitation, often finding "worst-case" scenarios with higher severity under specific constraints.
• Guidelines for Practitioners:
  • Use MORL when the goal is to maximize scenario diversity, uncover a wide variety of failure modes, and understand complex trade-offs between requirements.
  • Use SORL when the primary goal is to find severe, critical violations (e.g., near-collisions) and the requirements are known to be tightly constrained or effectively single-objective in nature.
• Requirement Dependencies: The study proves that treating interdependent requirements as a simple weighted sum (SORL) may miss nuanced trade-offs that MORL captures, but SORL can still be effective for finding extreme edge cases.

Conclusion

The paper concludes that neither SORL nor MORL is universally superior. The choice depends on the testing objective: MORL is preferable for comprehensive coverage and diversity, while SORL may be more efficient for exposing severe violations in constrained environments. The findings underscore the necessity of explicitly modeling requirement dependencies in RL-based AV testing.