Technical Summary: Realistic Adversarial Scenario Generation via Human-Like Pedestrian Model for AV Control Optimisation

1. Problem Statement

The safe deployment of Autonomous Vehicles (AVs) requires reliable interaction with vulnerable road users, particularly pedestrians. While simulation-based testing is a practical alternative to costly and unsafe real-world trials, current methodologies face two primary limitations:

1. Lack of Realism in Pedestrian Models: Most simulation frameworks rely on rule-based or deterministic pedestrian models (e.g., standard CARLA walkers) that oversimplify human behavior. These models fail to capture the natural inter-individual (differences between people) and intra-individual (variability within the same person) variability, leading to unrealistic interaction dynamics.
2. Disconnect Between Adversarial Testing and Realism: Adversarial scenario generation, used to probe safety limits, often prioritizes difficulty over plausibility. Existing methods (e.g., "suicidal pedestrian" or aggressive jaywalker models) generate exaggerated behaviors that may force AV controllers to become overly conservative, resulting in controllers that do not generalize well to real-world human interactions.

There is a methodological gap: human-like pedestrian models and adversarial testing have largely developed in parallel. This study addresses the need to integrate cognitively inspired, human-like pedestrian models into adversarial testing to generate behaviorally plausible yet challenging scenarios for AV control optimisation.

2. Methodology

The study proposes a framework implemented within the CARLA simulation environment, integrating the COMMOTIONS pedestrian model (Markkula et al., 2023) with AV control optimisation. The methodology consists of three core components:

2.1. Evaluation of Pedestrian Model Impact

The authors conducted a comparative study using a 2×2 design:

• Pedestrian Models: The standard rule-based CARLA pedestrian (WalkerAIController) vs. the cognitively inspired COMMOTIONS model (integrating sensory, motor, and cognitive mechanisms with stochastic variability).
• AV Controllers: The built-in CARLA rule-based planner vs. the open-source Autoware autonomous driving stack.
• Scenario: A simplified unsignalised zebra crossing where a pedestrian waits at the curb and initiates crossing based on Time-to-Arrival (TTA) values (6, 10, 14, 18 s).
• Metrics: Collision rate, gap acceptance rate, Post-Encroachment Time (PET), and sudden speed change rate (deceleration > 2.5 m/s²).

2.2. Generation of Adversarial Scenarios

To identify safety-critical interactions without resorting to unrealistic behaviors, the authors employed a parameter-TTA search using Bayesian optimisation:

• Objective: Minimize PET for specific pedestrian parameter sets (representing distinct individuals) by varying the TTA trigger time within [6, 18] s.
• Process: 100 distinct pedestrian parameter sets were sampled from the COMMOTIONS model. For each individual, Bayesian optimisation identified the TTA that produced the lowest PET.
• Adversarial Definition: Scenarios where the minimum PET fell below 1.5 s were classified as adversarial. These represent safety-critical interactions arising naturally from human variability rather than forced "suicidal" behavior.

2.3. AV Control Optimisation

The generated adversarial scenarios were used to optimise a single AV control parameter: braking distance (the threshold for initiating an emergency stop).

• Optimisation Goal: Minimise vehicle time lost (efficiency) while satisfying safety constraints: PET ≥ 1.5 s and maximum deceleration ≤ 2.5 m/s².
• Comparison Baselines: The optimisation was tested against three scenario sets:
  1. COMMOTIONS Low-PET: The identified safety-critical, human-like scenarios.
  2. COMMOTIONS Random: Randomly sampled TTA values for the same individuals (untargeted realism).
  3. Jaywalker Model: A rule-based adversarial baseline (Muktadir and Whitehead, 2022) known for aggressive, less plausible behaviors.

3. Key Results

3.1. Impact of Human-Like Models on Interaction

• Safety: The COMMOTIONS model produced zero collisions across all tested conditions with both AV controllers, whereas the rule-based CARLA pedestrian resulted in multiple collisions, particularly at intermediate TTAs.
• Behavioral Realism: COMMOTIONS pedestrians exhibited gap acceptance patterns that increased with TTA, consistent with human data. In contrast, CARLA pedestrians showed no clear trend and accepted gaps unrealistically early.
• Comfort: Interactions with COMMOTIONS resulted in significantly fewer sudden speed changes (<20% vs. >80% for CARLA pedestrians) and smoother vehicle decelerations.
• Variability: The COMMOTIONS model demonstrated that unsafe interactions (low PET) occur only for specific individuals under specific timing conditions, highlighting the importance of both inter- and intra-individual variability.

3.2. Effectiveness of Adversarial Optimisation

• Low-PET Optimisation: Tuning the braking distance using the human-like adversarial scenarios (Low-PET set) significantly improved performance:
  • Mean PET increased from 2.48 s to 4.96 s.
  • Minimum PET increased from 1.05 s to 1.80 s (eliminating safety-critical interactions).
  • Sudden speed changes dropped from 83% to 17%.
  • Average vehicle time lost decreased from 0.12 s to 0.05 s.
• Random vs. Low-PET: While random sampling improved efficiency, it failed to eliminate safety-critical interactions (Min PET remained at 1.23 s), demonstrating that targeting specific safety-critical cases is necessary for robust optimisation.
• Jaywalker Comparison: Optimisation against the aggressive Jaywalker model increased mean PET (5.52 s) but at a high cost: sudden speed changes remained high (38%), and time lost more than doubled (0.25 s). This indicates that the AV became overly defensive due to the unrealistic nature of the adversarial agent.

4. Significance and Claims

The paper claims to provide a proof of concept demonstrating that integrating human-like road user models into simulation-based adversarial testing enhances the credibility of AV evaluation and provides a practical basis for controller optimisation.

• Bridging Realism and Stress Testing: The study demonstrates that safety-critical scenarios can be generated through natural human variability (inter- and intra-individual) rather than artificial exaggeration. This allows for stress testing that is both challenging and behaviorally plausible.
• Optimisation Value: Unlike adversarial agents that induce overly conservative (inefficient) AV behaviors, human-like adversarial scenarios enable the optimisation of AV control parameters to achieve a better trade-off between safety, comfort, and efficiency.
• Methodological Contribution: The work unifies cognitively inspired pedestrian modeling with adversarial scenario generation, offering an end-to-end evaluation pipeline where pedestrian models and AV controllers are jointly simulated.

5. Limitations and Scope

The authors explicitly note the following limitations:

• Scenario Complexity: The study is restricted to a simplified one-to-one zebra crossing, lacking the complexity of multi-agent urban environments, occlusions, and traffic signals.
• Controller Diversity: The evaluation was limited to two controllers (CARLA and Autoware) and did not test deep RL-based or commercial proprietary stacks.
• Parameter Scope: Optimisation focused on a single parameter (braking distance) rather than a multi-parameter approach involving acceleration, jerk, and lateral control.
• Model Scope: While COMMOTIONS captures perceptual, motor, and cognitive variability, it omits social factors like group dynamics, cultural norms, and explicit communication (e.g., eye contact).

The paper concludes that while the framework is a foundational step, future work is required to extend these methods to more complex environments and diverse control architectures to ensure generalisability.