Geo-VLA: Geometry-Aware Vision-Language-Action Planning via Internalization of Map Semantics

Driving a car requires more than just seeing the road ahead; it demands an understanding of the invisible rules that govern movement. A vehicle must know where the lane ends, how a curve will bend, and how different roads connect at an intersection. For decades, engineers have tried to teach computers to drive by showing them cameras, but cameras only capture what is visible in the moment. They struggle to grasp the permanent structure of the road, such as the exact path a lane will take around a bend or the specific rules of a complex junction. This gap between seeing the world and understanding its geometry has kept fully autonomous driving from becoming a reliable reality in complex environments. Researchers are now turning to a new approach that combines vision with language, using large models that can "read" and "reason" about images to make driving decisions. However, even these advanced systems often fail when the road curves sharply or when an intersection is tricky, because they rely too heavily on the visual appearance of the road rather than its underlying shape and connectivity.

A team of researchers from the University of Science and Technology Beijing has developed a new method called Geo-VLA to solve this specific problem. Their goal was to teach these driving models to understand the geometry of the road without forcing them to carry heavy, pre-made digital maps that require constant updates and complex hardware to read. Instead of giving the car a map to look at while it drives, the researchers taught the car to "internalize" the map during its training phase. They created a special dataset called Geo-QA, which pairs driving camera images with questions and answers derived from offline map data. These questions focus on the static structure of the road, asking things like how a lane curves or where a drivable area ends. By training the model to answer these questions, the system learns to recognize road shapes and connections as if it were reading a map, even though it only sees a camera image during actual driving.

The process works in two distinct stages. First, the model is exposed to thousands of image-question-answer pairs. It learns to connect the visual details of a road scene with the precise geometric facts found in the map data. This step acts like a deep lesson in road structure, allowing the model to build a mental representation of how roads are laid out. Once this learning is complete, the researchers lock in this new understanding and use it to teach the model how to steer the car. Crucially, when the car is put to the test on the road, it does not need any map data, no extra sensors, and no complex software to look up road information. It simply looks through its front-facing camera, just as it did before, but now its internal reasoning is guided by the geometric knowledge it absorbed during training. This means the system remains simple and fast, avoiding the delays and errors that come with trying to match live video to a digital map in real time.

The results of this approach were tested on a standard driving simulation platform known as NAVSIM v1. The researchers compared their new method against existing systems that rely on high-definition maps or other complex inputs. The Geo-VLA system consistently outperformed its predecessors, achieving a score of 92.1 on a comprehensive metric that measures safety, comfort, and progress. This score represents a new high point for single-camera driving planners, surpassing previous records by a small but significant margin. The system proved particularly effective at navigating turns and intersections, where older models often drifted off course or failed to follow the correct path. By learning the rules of the road through language and questions rather than raw map data, the model generated trajectories that stayed closer to the intended lane and respected the physical boundaries of the road.

The study also revealed that teaching the model about static road structures was far more important than teaching it about moving objects like other cars or pedestrians. When the researchers tested a version of the system that focused only on dynamic elements, the performance did not improve. This suggests that while the model can already see moving things quite well, it struggles to understand the fixed geometry of the road without specific guidance. The Geo-VLA method fills this gap by providing that guidance during training, allowing the system to make safer and more efficient decisions. The researchers note that their approach relies on the quality of the map data used to create the training questions, and that future work will need to test how the system behaves in interactive traffic with other human drivers. For now, however, the study demonstrates that a car can learn the shape of the road through conversation and questions, carrying that knowledge with it as a silent, invisible guide.