HetroD: A High-Fidelity Drone Dataset and Benchmark for Autonomous Driving in Heterogeneous Traffic

This paper introduces HetroD, a large-scale, high-fidelity drone-based dataset and benchmark designed to address the challenges of autonomous driving in heterogeneous traffic by providing centimeter-accurate annotations of complex interactions between vehicles and vulnerable road users, while demonstrating that current state-of-the-art models struggle to handle these unstructured and dense scenarios.

The Gist

Imagine you are teaching a robot to drive a car.

Most of the time, we teach these robots using "textbooks" that look like perfect, organized highways. In these textbooks, everyone stays in their lane, follows the rules strictly, and drives like a well-oiled machine. This is what most existing datasets (like NuScenes or Waymo) look like. They are great for learning the basics, but they are a bit like learning to swim in a calm, chlorinated pool.

The Problem:
Real life isn't a swimming pool. Real life is a chaotic, crowded river where boats, kayaks, swimmers, and ducks are all jostling for space. In many parts of the world (like the busy streets of Taiwan where this study happened), the traffic is a mix of cars, scooters, pedestrians, and bicycles all weaving, cutting in, and negotiating right-of-way without clear lanes. This is called heterogeneous traffic.

Current self-driving cars struggle here because they've only been trained on the "pool." When they hit the "river," they get confused. They don't know how to predict a scooter that might suddenly weave through a gap, or how to handle a pedestrian who doesn't follow the crosswalk.

The Solution: HetroD
The researchers behind this paper built a new, massive "training manual" called HetroD.

Instead of filming from inside a car (which is like trying to see the whole river while sitting in a single boat), they used drones.

• The Drone View: Imagine a bird flying high above the traffic. It can see everything at once—no blind spots, no cars blocking the view. It captures the entire chaotic dance of the street.
• The Data: They filmed 17.5 hours of this chaos, capturing over 65,000 individual paths of cars, scooters, and people. Crucially, 70% of these "agents" are Vulnerable Road Users (VRUs) like scooters and pedestrians, which are usually ignored in other datasets.
• The Precision: They didn't just guess where people were; they mapped the streets with centimeter-level accuracy, creating a perfect digital twin of the real world.

The "Stress Test"
The researchers didn't just collect the data; they put the world's best self-driving AI models to the test using HetroD. It was like throwing a swimmer who only practiced in a pool into a raging river to see if they could survive.

What Happened?
The results were a wake-up call:

1. The Prediction Failure: The AI models were terrible at guessing where scooters and pedestrians would go next. They couldn't predict the "weaving" or the sudden turns. It's like trying to predict where a swarm of bees will go; the AI just froze or guessed wrong.
2. The Planning Failure: Even when the AI knew where everyone was, it didn't know how to drive safely among them. The rule-based planners (the "brain" of the car) kept trying to stay perfectly in the center of a lane, even when the lane didn't exist or was blocked. This led to dangerous situations, especially side-swipes with scooters and pedestrians, which the AI completely missed.

The Takeaway
The paper argues that to build truly safe self-driving cars for the real world, we can't just teach them on perfect highways anymore. We need to train them on the messy, unpredictable, "heterogeneous" traffic that actually exists.

HetroD is the new, high-fidelity "river" simulation that forces these robots to learn how to navigate the chaos, ensuring that when they finally hit the streets, they don't just drive like robots—they drive like humans who understand the flow of the crowd.

Technical Summary

1. Problem Statement

Autonomous driving (AD) systems currently struggle to navigate heterogeneous traffic environments dominated by Vulnerable Road Users (VRUs) such as pedestrians, cyclists, and motorcyclists.

• Data Gap: Existing large-scale datasets (e.g., NuScenes, Waymo) are primarily vehicle-centric, collected via on-board sensors, and focus on structured, lane-disciplined traffic. They lack sufficient data on complex VRU interactions, unstructured maneuvers (e.g., hook turns, lane splitting), and dense mixed-traffic scenarios.
• Sensor Limitations: On-board sensors suffer from occlusions and limited fields of view (FOV), making it difficult to track non-ego agents in dense crowds.
• Model Failure: Current state-of-the-art (SOTA) motion prediction and planning models, trained on structured data, fail to generalize to real-world heterogeneous scenarios, particularly in predicting lateral VRU movements and handling unstructured right-of-way negotiations.

2. Methodology: The HetroD Dataset

The authors introduce HetroD, a large-scale, high-fidelity dataset captured via drones to provide a holistic, occlusion-free view of traffic.

• Data Collection:

  • Source: 17.5 hours of ultra-high-resolution (5.4K) video recorded across six diverse urban locations in Taiwan (including busy signalized intersections, unsignalized intersections, and straight roads).
  • Scale: Contains over 65,400 unique agent trajectories, with 69.9% being VRUs (significantly higher than existing datasets).
  • Annotations: Provides centimeter-accurate ground truth, HD maps (Lanelet2 and OpenDRIVE formats), and synchronized traffic signal states.
  • Processing: An automated pipeline involving distortion removal, deep learning-based detection/classification, Kalman filtering for gap filling, and rigorous manual quality assurance to ensure tracking accuracy.

• Key Metrics & Properties:

  • Interaction Scale: HetroD exhibits the highest "Heterogeneous Interaction Scale" among existing drone-view datasets, capturing complex cross-type interactions (Vehicle-VRU, VRU-VRU).
  • Risk Indicators: The dataset shows higher latent conflict rates, characterized by shorter Time-to-Collision (TTC) and higher Deceleration Rate to Avoid Crash (DRAC) compared to datasets like SinD or inD.
  • Behavioral Diversity: Captures unstructured maneuvers such as hook turns, aggressive overtaking, queue cutting, and informal lane splitting, which are rare in lane-disciplined datasets.

• Unified Toolkit:

  • The authors developed a modular toolkit to convert HetroD data into standardized formats compatible with popular AD frameworks (ScenarioNet, GPUDrive, trajdata, ScenarioMax), facilitating immediate use for motion prediction, planning, and simulation.

3. Key Contributions

1. First VRU-Rich Drone Dataset: A centimeter-accurate, large-scale dataset specifically designed for heterogeneous traffic, with 70% VRU content and diverse topological scenarios.
2. Standardized Benchmarks: Established evaluation protocols for motion prediction, motion planning, and cross-dataset generalization.
3. Comprehensive Analysis: A thorough investigation revealing the specific failure modes of SOTA models in heterogeneous environments.
4. Open Source: The dataset, maps, and toolkit are released to the community to accelerate research in VRU modeling.

4. Evaluation Results

The authors evaluated SOTA models on HetroD to benchmark their performance in heterogeneous traffic.

A. Motion Prediction (Forecasting)

• Models Tested: MTR and Wayformer.
• Cross-Dataset Generalization: Models trained on on-board datasets (NuScenes, Waymo) performed significantly worse on HetroD than models trained on drone data (SinD), highlighting the "viewpoint gap."
• Failure Modes:
  • Lateral Prediction: Models struggle to predict lateral movements of VRUs (scooters/pedestrians).
  • Density Sensitivity: Performance degrades sharply in high-density, high-heterogeneity scenes.
  • Map Sensitivity: Prediction accuracy drops significantly (e.g., +166% error for MTR) when the map topology changes, indicating a heavy reliance on map priors rather than dynamic interaction modeling.
  • Agent Difficulty: Two-wheelers (scooters/motorcycles) are the most difficult to predict, followed by pedestrians, while vehicles are the easiest.

B. Motion Planning

• Models Tested: Intelligent Driver Model (IDM) and PDM-Closed (NuPlan benchmark).
• Evaluation Protocol: Closed-loop, non-reactive scoring using the V-Max framework.
• Results:
  • Performance Drop: Both planners showed significant score degradation on HetroD compared to NuPlan (e.g., PDM-Closed score dropped from 0.83 to 0.70).
  • Collision Analysis: While lane-keeping remained relatively stable, lateral collisions with VRUs increased drastically.
    • PDM-Closed: Lateral VRU collision rate was 0.022 (vs. 0.004 frontal).
    • IDM: Lateral VRU collision rate was 0.031 (vs. 0.008 frontal).
  • Qualitative Failures: Planners failed in scenarios requiring continuous multi-agent reasoning, unprotected left turns with crossing VRUs, and narrow road negotiations, often due to a lack of awareness of unstructured lateral interactions.

5. Significance and Conclusion

• Bridging the Gap: HetroD addresses the critical lack of data for modeling complex, unstructured interactions between vehicles and VRUs, which are prevalent in many parts of the world but underrepresented in current AD research.
• Revealing Limitations: The benchmark demonstrates that current AD systems are not robust enough for heterogeneous traffic. They rely too heavily on structured lane assumptions and fail to anticipate the erratic, lateral behaviors of VRUs.
• Future Directions: The paper calls for new approaches in:
  • VRU Modeling: Developing models that explicitly handle lateral interactions and unstructured maneuvers.
  • Simulation: Creating high-fidelity simulators capable of reactive VRU behaviors.
  • Planning: Designing planners that account for multi-agent density and heterogeneous risk, moving beyond simple lane-following logic.

In summary, HetroD serves as a critical testbed to expose the weaknesses of current autonomous driving technologies in real-world, mixed-traffic environments and provides the necessary data and tools to develop safer, more robust systems.