Benchmarking Autonomous Driving Planners Across Leaderboards: A Unified CARLA-Based Evaluation

The dream of a car that drives itself has been a fixture of science fiction for decades, promising a future where the burden of steering, braking, and navigating traffic is lifted from human shoulders. For the last twenty years, engineers have been working to turn that promise into reality, building systems that can see the road, predict what other drivers will do, and decide how to move safely. These systems rely on three main pillars: the ability to perceive the world through cameras and sensors, the ability to guess where other people and vehicles will go next, and the ability to plan a safe path forward. While we have seen test fleets operate in specific cities, the technology is not yet ready for the open road everywhere. A major hurdle remains: how do we know if one computer program is truly better than another? Different research teams often test their software in different virtual worlds or with different rules, making it impossible to compare them fairly. Without a common ground, it is difficult to tell if a new method is a genuine leap forward or just a method that works only on a specific test.

A team of researchers at the University of Southern California decided to solve this problem by bringing eight of the most advanced self-driving planners into a single, shared virtual arena. They chose a popular simulation platform called CARLA, which recreates the physics of driving and the complexity of city streets, and used it as a neutral judge. The researchers took software models that had previously won top spots in three different major competitions—one based on real-world data from Waymo, another from the nuPlan dataset, and several from the CARLA community itself—and forced them to run on the same tracks, facing the same traffic jams, pedestrians, and construction zones. The goal was not to create a new benchmark, but to see how these different approaches, which had been developed in isolation, would perform when judged by the exact same standards. The study included a wide variety of methods: some that rely on strict rules written by humans, some that learn by watching human drivers, some that learn through trial and error like a video game character, and some that use complex mathematical models to predict the future.

The results of this head-to-head comparison revealed that there is no single "best" way to build an autonomous driver. Instead, the study showed that the strengths and weaknesses of a system depend heavily on how much information it is allowed to see and how it is trained. The researchers found that planners which were given "privileged" information—meaning they could see the exact location and speed of every other car and pedestrian, as if they had a perfect map of the entire world—performed significantly better on long, continuous routes. These systems, which included a rule-based planner and a model trained to imitate human driving, managed to complete nearly 97 percent of their assigned routes without crashing. In contrast, the most successful system that relied only on standard sensors, like cameras and laser scanners, completed about 70 percent of the routes. This suggests that while sensors are good for handling quick, local interactions, having a perfect, bird's-eye view of the traffic situation is crucial for maintaining progress over long distances.

However, having a perfect view or a high completion rate does not guarantee safe or legal driving. The study uncovered a surprising trade-off between getting to the destination and following the rules. One of the top-performing models, which combined a powerful prediction engine with a control system designed to follow a set path, was excellent at staying on the road and reaching its destination. Yet, it frequently violated traffic laws, such as running red lights or ignoring stop signs, because its instructions focused on staying on the path rather than obeying the rules. On the other hand, a model trained using reinforcement learning, which learns by receiving rewards for good behavior and penalties for bad behavior, was very good at avoiding infractions but struggled to complete long routes efficiently. This highlights a critical challenge: a self-driving car can be very good at moving forward or very good at being polite, but combining both traits into a single system remains difficult.

Perhaps the most revealing finding was that no single approach could consistently handle the most chaotic situations. When the researchers tested the models against specific scenarios involving obstacles, such as a car door opening into the lane, a construction zone, or a pedestrian stepping out unexpectedly, every single method struggled. Whether the system was based on rules, learning from data, or complex mathematical predictions, none of them could reliably navigate these tricky, obstacle-rich environments without making mistakes. This suggests that the ability to make fine-grained decisions around unusual objects is a shared bottleneck for the entire field. The study also demonstrated that looking only at a final score can be misleading. Two systems might end up with the same overall rating, but one might have failed because it crashed, while the other failed because it broke the rules. By breaking down the results into specific types of driving situations, the researchers showed that a high score can hide very different kinds of failures.

The researchers also introduced a new hybrid approach to the test, taking a model famous for predicting where other cars will go and adding a planning module to it. This new system performed well in keeping the car on its path, but like the other top performers, it still struggled with traffic rules. The study concluded that while we have made tremendous progress in simulating and testing self-driving technology, we are not yet at the point where these systems are ready for mass deployment. The gap between a car that can drive on a quiet test track and one that can handle the unpredictable chaos of a real city is still significant. The work serves as a clear roadmap for the future, showing that the next breakthroughs will not come from simply making one part of the system faster or smarter, but from solving the difficult balance between following a path, obeying the law, and reacting safely to the unexpected. Until that balance is found, the promise of fully autonomous vehicles remains just out of reach.