Imitation Learning for Autonomous Driving in CARLA
This paper presents a compact multimodal behavioral cloning policy trained on 236,882 windows of expert demonstrations in CARLA that successfully drives autonomously for hours without collisions on both training and unseen routes, demonstrating effective closed-loop performance and qualitative transfer to new environments.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
The dream of teaching a car to drive itself has long relied on a simple, intuitive idea: if a machine can watch an expert driver and copy their every move, it should eventually learn to drive on its own. This approach, known as imitation learning, treats the complex challenge of steering, braking, and accelerating as a pattern-matching exercise. A computer watches hours of video and sensor data from a human or a perfect computer program, learning to predict what action to take next based on what it sees. The difficulty lies in the gap between watching and doing. When a human learns to drive, they practice in a loop; if they drift slightly off course, they correct it and learn from that correction. A machine trained only on perfect examples, however, has never seen what happens when it makes a mistake. If it drifts even a little, it enters a situation the expert never demonstrated, and without a guide, that small error can spiral into a crash. The question researchers face is whether a system can learn enough from a static library of perfect drives to handle the messy, unpredictable reality of driving alone.
In a recent study using a sophisticated driving simulator called CARLA, a researcher named Jordy Kieto explored exactly this question. The goal was not to invent a new type of computer brain, but to see how much real driving skill a relatively simple, compact system could acquire from a carefully curated library of expert drives. The researcher built a policy—a set of instructions for the car—that could look at a stream of images from a camera, a bird's-eye view map created by a laser scanner, and a list of upcoming road directions. By feeding this system five seconds of history at a time, the team trained it to predict the throttle, brake, and steering wheel movements of an expert driver. The training data came from a unique source: a systematic process that generated thousands of short driving clips, ensuring that the car saw a balanced mix of straight roads, left turns, and right turns, rather than just the easy, straight paths that often dominate driving logs.
The results of this experiment were surprisingly robust. Once trained on roughly three and a half hours of these verified driving clips, the policy was placed in the simulator to drive on its own, with no human and no safety net to intervene. In this closed-loop test, where every turn the car made determined what it would see next, the system drove for hours on the very roads it had been trained on, and also on completely different roads it had never seen before. It navigated curves, managed intersections, and stayed within its lane without a single collision in the author's runs. Perhaps most notably, the system showed a capacity to recover from large mistakes. When the car was placed far off the road or pointed in the wrong direction—situations it had never been explicitly taught to fix—it often managed to steer itself back onto the drivable surface and resume its journey. This recovery happened without the system having ever seen a "recovery" demonstration in its training data; it simply generalized from the small corrections it had learned during the training process.
However, the study is careful to distinguish between what was observed and what was proven. The driving feats described here took place entirely within a computer simulation, a controlled environment free of pedestrians, traffic lights, or unpredictable weather. The system's success relied heavily on a "privileged" piece of information: a precise, pre-calculated path of lane markers provided by the simulator's internal map, which the car could see but a real driver would not have access to in the same way. The researchers explicitly noted that while the car performed well, they did not claim it had solved the problem of driving in the real world. They also pointed out that the system's ability to handle turns was still the most challenging part, with errors in turning being more frequent than errors in driving straight. The study serves as a powerful demonstration that a simple system, fed with high-quality, diverse data, can learn to drive autonomously in a simulation for extended periods, but it stops short of declaring this a finished product for the open road.
The significance of this work lies less in the specific computer architecture used and more in the method of preparation. The researcher treated the data itself as the primary variable, moving from a small, messy set of manual drives to a rigorous, automated process that generated and verified every single training clip. By ensuring the training data covered specific maneuvers and included slight, randomized variations in starting positions, the system learned to handle a wider range of situations than it would have from a standard dataset. The study concludes that while offline training—learning from a static library—can produce a driver that works well in a loop, the true test of competence remains the ability to handle the unexpected. The researchers have released all their code, data, and the trained model itself, inviting others to test these findings, measure the recovery rates with greater precision, and explore how much of this success depends on the privileged map information versus the visual understanding of the road.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.