Generating Realistic Safety-Critical Scenarios for Vehicle-Pedestrian Interactions

Technical Summary: Generating Realistic Safety-Critical Scenarios for Vehicle-Pedestrian Interactions

Problem Statement
The deployment of Automated Driving Systems (ADS) requires rigorous validation against safety-critical vehicle-pedestrian interactions. However, a critical gap exists in current data resources: real-world datasets (e.g., NGSIM, INTERACTION, HDI) capture authentic human behaviors but rarely contain sufficient high-risk near-miss or collision-avoidance scenarios due to their rarity. Conversely, simulation-based platforms (e.g., CARLA, Virtual KITTI) offer scalable scenario generation but rely on rule-based controllers and simplified kinematic models that fail to reproduce the adaptive, interactive, and evasive decision-making characteristic of real human drivers and pedestrians. Existing hybrid approaches often treat simulation merely as a deployment environment for pre-trained models, leaving agents unable to adapt to new scenarios or interaction patterns, resulting in "frozen" policies with limited diversity and brittle decision-making.

Methodology
To address these limitations, the authors propose a three-stage framework that integrates real-world behavioral grounding with simulation-driven online learning to generate the Vehicle–Pedestrian Safety-Critical Interaction (VPSCI) dataset. The core of this framework is a Multi-Agent State-Space Transformer-enhanced Deep Deterministic Policy Gradient (MA-SST-DDPG) model.

• Stage 1: Pre-training with Real-World Data: The MA-SST-DDPG agents (vehicle and pedestrian) are pre-trained on 336 safety-critical interactions extracted from the HDI dataset (9,443 time steps). Safety-criticality is defined using a Curvilinear Time-to-Collision (CurvTTC) threshold of 5 seconds, which captures the full behavioral response window of curved intersection trajectories. The model learns human-like evasive behaviors through data-driven learning, utilizing a Practical Replay Buffer and a reward function that penalizes deviations from typical driving and walking speeds.
• Stage 2: Online Learning in Simulation: The pre-trained agents are deployed in the CARLA simulator (Town10 map) for online reinforcement learning. A monitoring module triggers the MA-SST-DDPG policy only when CurvTTC < 5 s, overriding the rule-based CARLA autopilot to execute learned evasive maneuvers. During these critical events, the model updates its policy using a Simulation Replay Buffer. This stage employs adaptive exploration, knowledge preservation mechanisms (to prevent catastrophic forgetting of real-world behaviors), and an adaptive reward weighting scheme that dynamically balances collision penalties, goal achievement, and speed consistency. This process refines the model into a "Refined MA-SST-DDPG" capable of generalizing across diverse intersection layouts.
• Stage 3: Large-Scale Data Generation: The Refined MA-SST-DDPG model is used to generate 198,157 high-resolution interaction episodes across eight distinct scenarios (two locations, four crossing directions each). The resulting VPSCI dataset includes 3,603 km of vehicle and 2,625 km of pedestrian trajectories, capturing diverse kinematic conditions and yielding behaviors.

Key Contributions

1. Three-Stage Hybrid Framework: A novel architecture that combines data-driven pre-training with simulation-driven online learning, addressing the scarcity of real-world safety-critical data and the lack of behavioral realism in simulation-only methods.
2. VPSCI Dataset: A large-scale, high-resolution dataset of over 198,000 safety-critical vehicle-pedestrian interaction episodes, validated for behavioral realism.
3. Behavioral Analysis: Demonstration of the framework's utility in performing safety analysis, quantifying conflict rates and behavioral risk patterns (e.g., yielding probabilities) that align with real-world observations.

Results

• Model Performance: The Refined MA-SST-DDPG model achieved the lowest trajectory errors among all compared baselines (Multi-Agent RL, Single-Agent RL, Trajectory Prediction, and Supervised Learning), with an Average Displacement Error (ADE) of 0.072 m and Final Displacement Error (FDE) of 0.142 m. This outperformed the pre-trained-only model and all other architectures, confirming the benefit of the hybrid learning approach.
• Distributional Equivalence: Statistical comparisons (Kolmogorov–Smirnov test, Wasserstein distance) confirmed that the CurvTTC distributions of the generated VPSCI data are statistically equivalent to the real-world HDI dataset. Furthermore, the generated data reproduced the behavioral logic of pedestrian yielding relative to vehicle proximity and speed.
• Turing Test Validation: A perceptual study with 44 transportation engineering experts evaluated the realism of the generated interactions against real-world projections and default CARLA behaviors. The three-stage framework's outputs received high realism scores (3.96–4.08 on a 5-point scale) and were found to be statistically indistinguishable from real-world data (p = 0.92), whereas default CARLA behaviors were rated significantly lower.
• Safety Analysis: Analysis of the generated data revealed realistic trends: conflict rates increased monotonically with vehicle and pedestrian speeds, and pedestrian yielding probability increased with vehicle proximity and speed. In contrast, CARLA's default autopilot failed to reproduce these speed-sensitive dynamics.

Significance
The paper claims that this framework successfully bridges the gap between the scarcity of real-world safety-critical data and the lack of behavioral realism in simulations. By producing a dataset that retains human-like evasive behaviors while offering large-scale diversity, the VPSCI dataset serves as a valuable resource for the development and evaluation of ADS. Specifically, it enables rigorous corner-case testing and behavior-conditioned training where pedestrian agents act as adaptive, realistic surrogates rather than scripted entities. The study concludes that this approach offers a scalable strategy to improve model performance in domains where real data is scarce but high-risk scenarios are critical for safety validation.