LightEMMA: A Longitudinal Evaluation of Vision-Language Models for Autonomous Driving

Self-driving cars have long relied on rigid, rule-based systems that follow a strict script, or on learning-based frameworks that map raw sensor data directly to steering commands. While these approaches have made significant strides, they often struggle with the unexpected: rare, complex situations that fall outside their training data, such as a construction worker waving a hand or a sudden, ambiguous traffic pattern. To bridge this gap, researchers have turned to vision-language models. These are powerful artificial intelligence systems trained to understand both images and human language, capable of reasoning about a scene much like a human driver might—describing what they see, interpreting intent, and making decisions based on context. The prevailing hope has been that as these models become more advanced and capable of general reasoning, they will naturally become better at driving, eventually surpassing specialized systems designed solely for the road.

A team of researchers at the University of Michigan set out to test this assumption with a rigorous, long-term look at how these models actually perform. They introduced a new evaluation framework called LightEMMA, designed to measure the driving skills of these models without giving them any special training or tweaking their internal settings. Instead of teaching the models how to drive, the researchers simply asked them to look at a street scene and predict where the car should go next. They tested fifteen different models from five major families, spanning three years of rapid development, using a challenging dataset of real-world urban driving scenes. The goal was to see if the newest, most powerful models were truly better drivers than their predecessors, or if the leap in general intelligence had failed to translate into safer, more accurate driving.

The results of this longitudinal study reveal a surprising disconnect. Despite the models becoming larger, faster at general reasoning, and more capable of understanding complex language, they did not consistently get better at predicting vehicle trajectories. In fact, some of the newest generations performed worse than older versions of the same family. When the researchers measured the accuracy of the predicted paths against the actual paths taken by real cars, they found that newer models often made larger errors. For instance, while one of the top-performing models from the GPT family achieved an average error of just over one meter over a three-second prediction window, other newer models from the same family or different families showed higher error rates, sometimes exceeding two meters. This suggests that simply making a model "smarter" in a general sense does not automatically make it a better driver.

The study also uncovered specific ways these models fail when faced with real driving scenarios. One common error involved an overreliance on the car's recent history. If a vehicle had just turned right at an intersection, the models often continued to predict a rightward curve, even when the car had straightened out and the road ahead was clear. They struggled to update their mental model based on the current view, instead projecting the previous action forward. In other cases, the models failed to reconcile conflicting visual cues. When a traffic light turned green but a vehicle was stopped directly ahead, some models correctly predicted the car should wait, while others ignored the obstacle and predicted the car would speed through the intersection. Similarly, when approaching a red light, some models predicted a sudden, harsh stop rather than a smooth deceleration, while others failed to slow down at all.

Beyond accuracy, the researchers examined the practical costs of using these models for driving. They found that inference time—the time it takes for the model to process an image and generate a prediction—varied widely. The fastest model took about 4.5 seconds to process a single frame, while the slowest took over 40 seconds. For a car traveling at highway speeds, waiting even a few seconds to make a decision is far too long; real-time driving requires decisions in milliseconds. Furthermore, the cost of using commercial models for this task was significant, with some costing several cents per frame, which would add up to a prohibitive expense for continuous operation. The study also noted that even the best models occasionally failed to follow instructions, producing outputs that were difficult to read or parse, though the researchers developed a method to correct most of these formatting errors.

Ultimately, the work suggests that the path to safe, autonomous driving using vision-language models requires more than just waiting for the next generation of general-purpose AI. The models are capable of describing a scene and understanding language, but they lack the specific, domain-trained intuition needed to navigate the physical world safely and reliably. The researchers conclude that to make these systems viable, they will need specialized adaptation to learn the specific rules and dynamics of driving, rather than relying on their broad, general reasoning abilities alone. The LightEMMA framework now stands as a tool for the community to continue testing and refining these models, ensuring that future advancements in artificial intelligence translate into tangible improvements on the road.