🤖 AI

Scenario Generation for Testing of Autonomous Driving Systems Using Real-World Failure Records

This paper proposes a modular, LLM-based pipeline that converts natural language historical failure records (specifically NHTSA crash data) into diverse, realistic test scenarios for autonomous driving systems, successfully identifying system failures within a limited testing budget on the Metadrive simulator.

Original authors: Anjali Parashar, Chuchu Fan

Published 2026-07-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Anjali Parashar, Chuchu Fan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a self-driving car how to survive a chaotic city. The old way of doing this is like a teacher handing a student a list of 20 specific math problems to solve. The teacher knows these problems are common, but they might miss the weird, tricky situations that actually cause accidents in real life. This process is slow, expensive, and relies on humans guessing what might go wrong.

This paper proposes a smarter, faster way to build a "test drive" for self-driving cars by looking at the car's past mistakes.

The Problem: The "Static Test" Trap

Right now, engineers design test scenarios manually. They say, "Okay, let's test what happens if a car goes straight and another car turns left." They then tweak the speed slightly to see if the car crashes.

The Flaw: This is like only practicing for a soccer game by kicking the ball against a wall. You might get good at that one specific move, but you haven't practiced what happens when it starts raining, the ball bounces weirdly, or a player trips unexpectedly.
The Gap: Real-world crash reports are full of these weird, specific details (like a construction zone or a driver making a sudden, weird turn), but current testing methods often ignore them because they are hard to turn into a math formula.

The Solution: The "AI Detective" Pipeline

The authors, Anjali Parashar and Chuchu Fan from MIT, built a system that acts like an AI Detective. Instead of guessing what to test, the system reads real police crash reports and turns them into video game levels.

Here is how their "three-step detective" works:

The Summarizer (Paraphrasing Agent):
Real crash reports are messy. They say things like, "It was raining on 5th Avenue at 5 PM and the police weren't called." The AI reads this and says, "Okay, ignore the time and the police. The important part is: A car went straight, and another car tried to turn left in front of it." It strips away the noise to find the core story.
The Architect (Scenario Generation Agent):
Now, the AI takes that core story and builds a specific test case for the simulator (a video game environment called Metadrive). It translates the story into rules: "Set the road type to 'Intersection,' make the first car go straight, and make the second car try a left turn." It's like turning a written recipe into a fully cooked meal.
The Refiner (Fine-Tuning Agent):
Sometimes the first version of the test isn't dangerous enough to really stress-test the car. The AI looks at the result and says, "The cars were too far apart. Let's move them closer together to see if the self-driving car actually brakes in time." It tweaks the numbers until the test is just right.

The Experiment: 20 Scenarios, Big Results

The team tested this on 20 different scenarios generated from real NHTSA (National Highway Traffic Safety Administration) crash records. They didn't just make random tests; they used a "clustering" method (like sorting laundry into piles of socks, shirts, and pants) to make sure they covered a wide variety of different types of crashes.

What did they find?

It Works: The system successfully turned text descriptions of crashes into working simulations.
It Finds Hidden Flaws: When they ran these 20 tests on a standard self-driving policy, they found interesting failures that humans might have missed.
- Example 1: The car was too cautious. When a car in front turned, the self-driving car braked too early and started "shaking" or oscillating back and forth, which is annoying and unsafe in real traffic.
- Example 2: The car was great at avoiding big cars but terrible at avoiding small things like construction cones. If there was a "work zone," the car would crash into the cones.
- Example 3: The car handled straight-on crashes well but failed when cars tried to cut in from the side.

The Bottom Line

This paper introduces a tool that automates the creation of test drives for self-driving cars. Instead of humans spending weeks designing scenarios, an AI reads real-world crash stories, cleans them up, and builds a diverse set of "stress tests" that reveal exactly where the self-driving software is weak.

It's like giving the self-driving car a "final exam" based on the actual mistakes other drivers have made, ensuring it's ready for the messy, unpredictable reality of the road, not just the clean, perfect scenarios we usually imagine.

Technical Summary: Scenario Generation for Testing of Autonomous Driving Systems Using Real-World Failure Records

Problem Statement
The primary challenge addressed is the inefficiency and rigidity of current testing methods for Autonomous Driving Systems (ADS). Existing approaches rely heavily on static, manually designed scenario templates (e.g., Euro NCAP protocols) or simulation-based searches that assume fixed scenario representations and focus on optimizing numerical parameters within those templates. These methods often fail to capture the rich, contextual nuances of real-world failures found in historical crash data, such as specific road types, work zones, or complex interaction dynamics. Furthermore, there is a lack of a principled framework to extract scenario definitions directly from natural language crash narratives and adapt them for system-specific simulation testing.

Methodology
The authors propose an automated, modular scenario generation pipeline that leverages historical ADS crash records (specifically NHTSA data) to construct diverse test suites. The approach integrates categorical data and natural language narratives through a three-stage Large Language Model (LLM) pipeline:

Meta-Variable Extraction: The system identifies categorical features from crash records (e.g., Road Type, Pre-crash movement of Self Vehicle and Crash Partner, Work Zone presence) to define high-level scenario templates.
LLM-Based Scenario Generation: The pipeline employs three sequential LLM agents:
- Paraphrasing Agent: Processes raw crash narratives to filter out irrelevant details (e.g., location, time, injury reports) and retains only topological and contextual details necessary for reconstruction. It also filters out scenarios incompatible with the target simulator (e.g., rear-end collisions caused by non-ego negligence that cannot be replicated by the specific ego policy being tested) and generates plausible alternative narratives.
- Scenario Generation Agent: Takes the paraphrased narrative and the defined meta-variable template to generate a specific scenario instance ( $z_s$ ). This includes encoding specific parameters such as agent trajectories, speeds, and map types compatible with the simulator (in this case, Metadrive).
- Fine-Tuning Agent: Refines the generated scenario based on performance metrics (e.g., minimum gap) collected during trajectory rollouts. This step iteratively adjusts numerical parameters to stress-test the system, potentially reducing safety margins to reveal failure modes.
Diversity via Clustering: To ensure broad coverage within a limited testing budget, the authors cluster the historical crash records using k-means. The clustering features include one-hot encoded meta-variables and 6-dimensional vector embeddings of the narratives (generated via all-MiniLM-L6-v2). One scenario is generated per cluster to represent distinct failure modes.

Key Contributions

Automated Pipeline: The authors present a novel procedure that automates the transition from real-world crash reports to simulation-ready scenarios, utilizing both categorical features and natural language narratives.
System-Specific Adaptation: The method introduces a modular LLM-based approach to adapt historical records to specific system constraints and policies, moving beyond static template design.
Validation Framework: The approach is validated by generating a test suite for an IDM policy on the Metadrive simulator using 2021 NHTSA ADS crash records.

Experimental Results
The authors evaluated the pipeline using 2021 NHTSA data, resulting in 20 distinct scenario clusters.

Accuracy: 17 out of 20 generated scenarios perfectly matched the intended meta-variable templates. The three mismatches involved minor errors in crash partner movement direction or work zone presence, which the authors note can be further optimized via the fine-tuning agent.
Diversity and Stress Testing: The clustering successfully captured 11 out of the 14 scenario templates naturally present in the filtered historical data. The fine-tuning process significantly reduced the minimum gap between vehicles in 14 out of 16 scenarios requiring adjustment, effectively creating "near-miss" or collision scenarios.
System Insights: The generated tests revealed specific behavioral characteristics of the IDM policy:
- The policy successfully avoided head-on collisions but exhibited undesirable oscillatory movements and early braking when avoiding straight-moving vehicles.
- The policy was less reactive to small obstacles (e.g., cones in work zones) compared to large vehicles, leading to collisions in work zone scenarios.
- Lateral motion scenarios resulted in side collisions, highlighting specific failure modes not always captured by standard templates.

Significance and Claims
The paper claims that its approach provides a principled way to bridge the gap between unstructured real-world failure data and structured simulation testing. By utilizing historical records, the method alleviates the manual effort required to design scenario templates and ensures that testing covers both common and uncommon failure conditions. The authors emphasize that their generated scenarios are compatible with existing low-level testing methods (such as Bayesian Optimization or MCMC sampling), allowing for further iteration on numerical parameters. The work demonstrates that LLMs can effectively transform natural language failure descriptions into executable, system-specific test cases, revealing system vulnerabilities within a limited testing budget (20 scenarios).

Limitations and Future Work
The authors acknowledge limitations, including the lack of a unified definition for "coverage" in testing, which complicates the pursuit of diverse generation. They also note that fixing a specific policy limits the ability to fully replicate ego vehicle movements in all desired scenarios. Future work directions include extending the pipeline to generate visual conditions for vision-based policies and using LLMs to create synthetic failure narratives where real-world data is absent.

The Problem: The "Static Test" Trap

The Solution: The "AI Detective" Pipeline

The Experiment: 20 Scenarios, Big Results

The Bottom Line

Technical Summary: Scenario Generation for Testing of Autonomous Driving Systems Using Real-World Failure Records

More like this