Planning-Oriented End-to-End Autonomous Driving: Architectures, Evaluation, and Emerging Paradigms
This survey reviews the evolution of end-to-end autonomous driving from simple regression to planning-oriented systems, synthesizing architectures and evaluation protocols across four key axes while arguing that the critical distinction lies in learning representations that ensure safe, feasible, and human-aligned planning rather than merely using intermediate features.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
For decades, the dream of the self-driving car relied on a simple idea: break the problem into small, manageable pieces. First, a computer would look at the road and identify objects like cars and pedestrians. Next, it would guess where those objects were going. Finally, a separate computer would decide how to steer and accelerate to avoid them. This modular approach worked well enough to get us started, but it created fragile hand-offs between steps. If the first step made a small mistake, the next step might not know how to fix it, leading to confusion or danger. To solve this, researchers began building systems that learned to drive in one continuous flow, taking in camera images and directly outputting steering commands. These early versions were impressive, but they often struggled when the car encountered a situation it hadn't seen before, or when the driver needed to make a complex decision like merging into heavy traffic. The question shifted from simply copying human steering to understanding how a machine could plan a safe and smooth path through a chaotic world.
A new survey by a team of researchers offers a clear map of where this field has gone and where it is heading. They argue that the most important progress in self-driving technology is no longer about removing structure from the system, but about how that structure is learned and tested. The researchers reviewed hundreds of studies to show that the best modern systems do not just mimic human movements; they learn to build internal representations of the world that help them plan for the future. Instead of just reacting to what is immediately in front of the car, these systems learn to imagine different possible futures, weigh the risks, and choose a path that keeps everyone safe while following the route. The team found that the old way of testing these cars—simply checking if the computer's path matched a human's recorded path on a video—is no longer enough. A car might perfectly copy a human's route on a quiet day but fail completely when a child runs into the street. The researchers emphasize that true success requires testing the car in a way that forces it to react to its own mistakes and the actions of others, just as a real driver would.
The survey traces a journey from simple imitation to sophisticated planning. Early systems were like students who memorized a specific route; if the road changed slightly, they got lost. Newer methods have learned to understand the geometry of the road and the intentions of other drivers. They use structured data, such as vectorized maps that represent lanes and traffic rules as clear lines and shapes, rather than just blurry pictures. This allows the car to reason about space and safety more effectively. The researchers highlight that the most advanced systems now act like a team of experts working together. One part of the system might focus on predicting where a pedestrian will step, while another part plans the car's trajectory, and a third part checks if the plan follows traffic laws. Crucially, these parts are not separate modules but are trained together to support the final goal: a safe drive. The study shows that when these systems are trained with better supervision—such as learning from experts who know the hidden details of the road or by practicing in simulated environments where they can make mistakes safely—they become much more robust.
However, the researchers also point out a significant gap between what these systems can do in a computer simulation and what they can do on a real road. Many studies claim success based on open-loop tests, where the car follows a recorded video of a human driver without actually interacting with the world. The survey argues that this is a misleading measure of safety. A car that looks perfect on a video might freeze or crash when it is actually driving and has to deal with unpredictable drivers. The authors suggest that the field must move toward closed-loop evaluation, where the car drives in a simulation that reacts to its actions, or in real-world tests that capture long, difficult scenarios. They also note the rise of systems that use language to help the car understand the world. These systems can read a sign or understand a hand gesture from a police officer, but the researchers caution that the ability to explain a decision in words does not guarantee the decision itself is safe. The language must be tightly connected to the actual driving action, not just a fancy explanation added afterward.
Looking ahead, the researchers identify several challenges that must be solved before these cars are ready for widespread use. One major issue is uncertainty. A good driver knows when they are unsure and slows down; the survey suggests that self-driving systems need to be better at recognizing when they are confused and asking for help or switching to a safer mode. Another challenge is the "long tail" of rare events. Most driving happens on familiar roads, but accidents often happen in unusual situations, like a car driving the wrong way or a storm blocking the road. The survey calls for better ways to test these rare scenarios and to train systems to handle them without panicking. Finally, the researchers stress the need for transparency and reproducibility. Too many studies are hard to verify because the code or data is not shared, or because the testing methods are hidden. They argue that for the technology to advance, the community must agree on standard ways to test and compare these systems, ensuring that claims of safety are backed by solid, repeatable evidence.
The ultimate conclusion of this work is that the future of self-driving cars lies not in making the system look more like a human brain, but in making it more reliable and accountable. The researchers propose that we should judge these systems not by how many layers of neural networks they have, but by how well they can plan a safe journey under difficult conditions. They suggest that the next generation of technology will combine the ability to learn from vast amounts of data with strict safety checks that ensure the car never takes a risk it cannot handle. By focusing on planning, rigorous testing, and clear standards, the field is moving closer to a future where self-driving cars are not just a novelty, but a trusted part of our daily lives. The path forward requires patience and a commitment to truth, ensuring that every step taken is measured not just by how fast the car goes, but by how safely it gets there.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.