SIREN-Bench: Behavior-Driven Generation and Evaluation of Emergency-Vehicle Interactions
The paper introduces SIREN, a behavior-driven SUMO-CARLA co-simulation platform and benchmark (SIREN-Bench-v1) that generates synchronized emergency-vehicle interaction scenarios to evaluate autonomous driving systems, revealing significant behavior-dependent failure modes in current perception, prediction, and risk-understanding models.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine a city street where an ambulance, its siren wailing, needs to reach a hospital. The cars around it do not simply sit still; they react. Some brake hard, others swerve into empty lanes, and some pull over to the curb to create a clear path. This moment of shared movement is a complex dance of cause and effect: the emergency vehicle changes its rules of the road, and the civilian drivers must instantly adapt their own behavior to keep everyone safe. For the computers that will one day drive our cars, understanding this split-second negotiation is critical. If an autonomous vehicle cannot predict how a human driver will react to a siren, or if it fails to see a car squeezing into a tight gap, the consequences can be severe. Yet, testing these systems is difficult. Real-world data is rare and unpredictable, while computer simulations often lack the realistic, messy details of how people actually behave when their routine is interrupted.
Researchers have built a new tool to solve this problem, creating a virtual laboratory where they can study exactly how emergency vehicles and regular traffic interact. They call their system SIREN. Instead of just recording what happened in the past, this platform allows scientists to design specific situations and watch how a digital world responds. They set the rules for the emergency vehicle—telling it to ignore a red light or squeeze between two lanes of stopped cars—and then program the surrounding cars to react in different ways, from simply slowing down to forming a dedicated rescue corridor. By running these scenarios in a high-fidelity simulation that mimics real sensors and road physics, the team generated a massive dataset of these interactions. They then used this data to test three different types of artificial intelligence: systems that identify objects, systems that predict where cars will go next, and systems that try to understand the overall risk of a scene.
The results revealed that not all emergency situations are equally difficult for these machines. When the goal was simply to spot cars on the road, the AI struggled the most during "clearance" scenarios. In these moments, civilian drivers are actively moving out of the way, creating a chaotic, shifting pattern of vehicles that are no longer neatly aligned in their lanes. The computers, trained on orderly traffic, found it hard to keep track of cars that were suddenly changing positions to make room. Conversely, predicting where a car would move next proved hardest when the emergency vehicle was crossing an intersection against a red light. In these moments, the civilian drivers were still trying to navigate the intersection while reacting to the emergency vehicle, creating a confusing mix of movements that the prediction models could not accurately foresee. In fact, on average, the advanced computer models were no better at guessing the future path of these cars than a simple rule that assumes a car will keep moving at the same speed.
The study also tested how well artificial intelligence could understand the severity of a situation. The researchers asked the systems to look at video footage and decide if the scene was normal, if it was a near miss, or if a collision was about to happen. The results were stark: the models were heavily biased. Some systems were so cautious that they labeled almost everything as "normal," missing the danger entirely. Others were so sensitive that they saw danger where there was none, or they failed to recognize the most critical moments. No single model was able to correctly identify all three types of situations. This suggests that while current technology is good at seeing the world, it still struggles to understand the complex, human logic behind why drivers move the way they do when an emergency vehicle approaches.
The researchers emphasize that the difficulty does not come from the emergency vehicle itself, but from the behavior it triggers in everyone else. A car driving on the shoulder is easy to predict; a car that suddenly brakes and swerves because a siren is blaring is much harder to understand. By creating a benchmark that focuses on these specific interactions, the team hopes to help engineers build safer autonomous vehicles. Their work shows that to make self-driving cars truly safe, we must teach them not just to see the road, but to understand the social rules of the road, especially when those rules are being rewritten by an emergency. The platform they built is now available for other scientists to use, offering a way to test new ideas against a wide variety of realistic, behavior-driven scenarios that were previously impossible to study in such detail.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.