ROAD-Waymo: A Large-Scale Action Awareness Dataset for Autonomous Driving

Technical Summary: ROAD-Waymo: A Large-Scale Action Awareness Dataset for Autonomous Driving

Problem Statement
Autonomous Vehicle (AV) perception systems currently rely heavily on object detection and scene segmentation. However, safe interaction with other road users requires a holistic understanding of the scene, specifically the ability to comprehend the actions and events occurring within it. While previous datasets like the original ROAD (UK) introduced multi-label annotations for agents, actions, and locations, they remain limited in scale (only 22 videos from a single city) and diversity. Existing large-scale AV datasets (e.g., Waymo, nuScenes) focus primarily on object detection, tracking, or trajectory forecasting, lacking dense, high-level annotations for action and event detection. There is a critical need for a dataset that bridges this gap, offering large-scale, diverse, and verified annotations for action-aware perception, while also enabling research into domain adaptation between different real-world driving environments.

Methodology
The authors introduce ROAD-Waymo, a dataset constructed by adding dense spatiotemporal annotations to the existing Waymo Open Dataset. The methodology involves:

1. Annotation Protocol: Following the road event = {Agent, Actions, Locations} framework established in the original ROAD dataset, the authors annotated 198k video frames. Each relevant agent is assigned a bounding box (inherited or added via YOLOv6 for traffic lights) and labeled with:

   • Agent: 12 classes (e.g., car, pedestrian, emergency vehicle).
   • Action: 30 classes (e.g., moving, turning, braking, indicating).
   • Location: 16 classes (e.g., junction, incoming lane, pavement).
     Annotations are updated only when semantic changes occur, resulting in 54k agent tubes and a total of 12.4M labels.

2. Automated Verification Pipeline: A novel contribution is the integration of a logical verification process. The authors elicited 251 domain-dependent logical requirements (e.g., "a traffic light cannot be red and green simultaneously") expressed in propositional logic. An automated pipeline checks all annotations against these constraints, flagging and correcting 6,276 tube annotations to ensure the dataset's integrity and commonsense compliance.

3. Domain Adaptation Framework (ROAD++): By ensuring compatibility with the original ROAD (UK) dataset, the authors created ROAD++, a benchmark for unsupervised domain adaptation between real-world scenarios in the UK and the US (involving 4 cities total).

4. Baseline Models: The paper establishes baselines for various tasks using:

   • 3D-RetinaNet: Utilizing I3D and SlowFast backbones for spatiotemporal feature extraction.
   • YOLOv8: For single-frame agent detection.
   • RevGrad: A reverse gradient approach for unsupervised domain adaptation.
   • Neuro-Symbolic Models: Integrating the logical constraints directly into the loss function using t-norms (Gödel and Łukasiewicz) to guide neural predictions.

Key Contributions

1. ROAD-Waymo Dataset: A large-scale, multi-label dataset for action and event detection. It is 7 times larger than the original ROAD dataset (4.3M action labels) and covers 4 US cities with 6 weather conditions. It includes significantly more complex scenarios, such as emergency vehicles and dense traffic, compared to prior datasets.
2. Verified Annotations: It is the first AV dataset guaranteed to comply with a set of commonsense logical constraints via an automated verification pipeline.
3. ROAD++ Benchmark: A framework enabling domain adaptation research between real-world datasets from different countries (UK vs. US) and cities, addressing the "real-to-real" adaptation challenge.
4. Comprehensive Baselines: The paper provides strong baselines for agent, action, location, duplex, and event detection, as well as neuro-symbolic prediction and unsupervised domain adaptation tasks.

Experimental Results

• Performance: Experiments on ROAD-Waymo show that SlowFast backbones generally outperform I3D for action and duplex detection, while I3D performs well for location and event detection. YOLOv8 achieved the highest performance for single-frame agent detection (38.1% f-mAP on validation).
• Domain Adaptation: Cross-dataset evaluation reveals a significant performance drop (over three-fold) when training on the UK ROAD dataset and testing on US ROAD-Waymo, highlighting the difficulty of the domain shift. The ROAD++ approach (training on combined data) yields the best results. Unsupervised domain adaptation (RevGrad) results fall between the "Lower Bound" (source-only training) and "Upper Bound" (target-only training), confirming the benchmark's utility for developing adaptation algorithms.
• Neuro-Symbolic Gains: Integrating logical constraints into the loss function improved performance, particularly for location detection using the Gödel t-norm (increasing f-mAP from 15.31% to 16.03% on the validation set).
• Error Analysis: The authors note that errors are concentrated in long-tail action classes (e.g., lane changes) and short-duration behaviors, reflecting the inherent difficulty of the benchmark.

Significance
The paper positions ROAD-Waymo as a necessary evolution in AV perception research, moving beyond simple object detection toward "situation awareness." Its significance lies in:

• Scale and Diversity: Providing the largest known dataset for action/event detection, covering multiple cities and complex scenarios (e.g., emergency vehicles) previously absent in smaller datasets.
• Data Integrity: Establishing a new standard for dataset quality through automated logical verification, ensuring annotations adhere to physical and commonsense rules.
• Real-World Adaptation: Offering the first real-to-real domain adaptation benchmark for event awareness across different countries and driving cultures (left-hand vs. right-hand traffic), a critical step for deploying AVs globally.
• Neuro-Symbolic Integration: Demonstrating how background knowledge (logical constraints) can be effectively leveraged to improve neural network performance in safety-critical driving scenarios.

The authors conclude that ROAD-Waymo and the ROAD++ framework provide a challenging playground for developing advanced perception systems and tackling novel problems in autonomous driving, specifically regarding event awareness and cross-domain generalization.