💻 computer science

GeoWAM: Visual Geometry World Action Models for Autonomous Driving

This paper introduces GeoWAM, a world action model for autonomous driving that replaces pixel-based future prediction with future geometry forecasting using point clouds to better capture spatial structure and motion, resulting in significantly stronger driving policies compared to image-based alternatives.

Original authors: Yiren Lu, Xin Ye, Jiaming Liu, Jin Yao, Yi-chung Chen, Liam Merino, Dhruva Dixith Kurra, Min Cai, Tom Lampo, Yu Yin, Danhua Guo, Burhan Yaman

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yiren Lu, Xin Ye, Jiaming Liu, Jin Yao, Yi-chung Chen, Liam Merino, Dhruva Dixith Kurra, Min Cai, Tom Lampo, Yu Yin, Danhua Guo, Burhan Yaman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Autonomous driving is fundamentally a game of anticipation. To navigate safely, a vehicle must do more than simply see the road in front of it; it must understand how that road and the objects upon it will change in the seconds to come. It needs to predict where a pedestrian might step, how a car ahead will slow down, and how its own path will shift through the environment. For years, researchers have tried to teach computers this skill by showing them endless streams of video, hoping the machines would learn to guess the next frame of a movie. These systems, known as world models, are powerful because they can simulate the future, but they often struggle with a specific flaw: they see the world as a flat picture. They focus on colors, textures, and lighting, which can obscure the true three-dimensional shape and movement of objects. When a computer tries to figure out how a car is turning based only on changing pixels, it is like trying to understand the mechanics of a clock by watching the shadows it casts on a wall; the information is there, but it is indirect and difficult to untangle.

A team of researchers at Uber AV Labs and Case Western Reserve University has proposed a different approach, one that skips the flat picture entirely and looks directly at the shape of the world. They introduced a new system called GeoWAM, which treats the driving environment not as a video to be predicted, but as a collection of 3D points that move and transform over time. Instead of trying to generate a realistic future image of a street, the system is trained to forecast the future geometry of that street—the precise location of every curb, tree, and vehicle in three-dimensional space. By focusing on this spatial structure, the researchers found they could give the autonomous vehicle a much clearer understanding of how the world evolves. Their work suggests that when a driving agent learns to predict the physical shape of the future rather than its visual appearance, it becomes significantly better at planning safe and smooth trajectories.

The core idea behind this new method is that pixels are a poor language for describing motion. A standard video world model learns to predict how light and color will shift from one moment to the next. While this can produce visually convincing results, the underlying physical movements—the actual rotation of a wheel or the translation of a car through space—remain hidden inside those changing colors. The new system, GeoWAM, bypasses this confusion by working directly with point clouds. Imagine a 3D map made of millions of tiny dots, where each dot represents a specific spot on a car, a building, or the road. This system takes a sequence of past views from the vehicle's cameras and learns to predict where those dots will be in the future. It does not care if the car is red or if the sun is shining; it cares only about where the car is and how it is moving through space. This direct representation aligns perfectly with the task of driving, which is fundamentally about navigating through a three-dimensional world.

To build this capability, the researchers trained the system in two distinct stages. First, they taught the model to be a geometer. Using a vast collection of driving data from multiple cameras, the system learned to look at a sequence of past scenes and accurately predict the 3D structure of the scene several seconds into the future. It was not asked to recreate the look of the sky or the texture of the asphalt, only to map out the positions of objects. Once the system mastered this ability to forecast the shape of the future, the researchers added a second stage: the action planner. This new component takes the predicted 3D map and asks a simple question: given how the world is going to change, what should the car do next? Because the system is already thinking in terms of 3D movement, the answer comes naturally. It infers the car's future motion directly from the evolving geometry, creating a plan that is grounded in the physical reality of the road rather than the visual patterns of a video.

The results of this approach were tested rigorously on standard driving benchmarks. When asked to predict the future shape of a scene, the new system outperformed previous methods that relied on video generation. It was more accurate in placing objects in the correct 3D space, even as the time horizon extended further into the future. More importantly, when this geometric foresight was used to control a vehicle in a simulated environment, the driving performance improved significantly. In open-loop tests, where the system plans a path without interacting with the environment, and in closed-loop tests, where the system must react to its own mistakes in real-time, GeoWAM consistently produced safer and more reliable trajectories than its image-based counterparts. The system demonstrated a particular strength in complex scenarios, such as navigating tight turns or avoiding obstacles, where understanding the true spatial layout is critical.

The researchers argue that this shift from visual prediction to geometric prediction represents a more natural way for machines to learn about driving. By removing the distraction of appearance and focusing on structure, the system learns the rules of motion more directly. It does not need to guess how light will reflect off a wet road to know that a car is sliding; it simply sees the car's position changing in a way that indicates a slide. This clarity allows the vehicle to make decisions that are robust and consistent. The study confirms that while video generation is a powerful tool for many tasks, it is not the ideal foundation for autonomous driving. Instead, a model that understands the world as a dynamic, three-dimensional structure is better equipped to navigate the real world, offering a promising path forward for the development of truly autonomous vehicles.

Technical Summary: GeoWAM

Problem Statement

Autonomous driving requires agents to anticipate not only the current scene but also its future evolution to constrain actions effectively. While recent World Action Models (WAMs) have emerged to jointly model scene dynamics and ego actions, most existing approaches rely on pixel-space representations (future video generation). The authors argue that pixels are an indirect state representation for driving dynamics because they entangle geometry and motion with appearance, texture, and illumination. Consequently, video-based models must infer 3D transformations from 2D observations, often leaving the underlying physical dynamics implicit and difficult to recover. This misalignment between the visual representation and the 3D space in which driving actions are executed limits the effectiveness of the learned policies.

Methodology: GeoWAM

The authors introduce GeoWAM, a Visual Geometry World Action Model that shifts the state representation from pixels to metric geometry (point clouds). The framework operates in two stages:

1. Visual Geometry World Model Pretraining

Instead of predicting future images, GeoWAM is pretrained to forecast future scene geometry from historical multiview observations.

Encoding: A geometry encoder (based on DVGT-2) processes a sequence of multiview images to produce multi-level historical tokens, comprising both geometry tokens (spatial structure) and ego tokens (motion context).
Decoding: A future geometry decoder utilizes learned queries to predict future geometry tokens. It employs causal temporal self-attention to model evolution across future steps and cross-attention to retrieve context from historical observations.
Output: The model decodes these tokens into dense future point maps and confidence maps using a shared geometry head (Point DPT).
Supervision: Training involves feature-level alignment (cosine distance) with target features derived from future frames and dense point-map regression (Euclidean loss, confidence-aware loss, and surface-normal consistency) against ground-truth geometry.

2. GeoWAM: The Action Model

The pretrained geometry model is extended into a world action model via an inverse-dynamics-like formulation:

Future Ego-Token Decoding: A dedicated branch infers future ego motion from the predicted scene evolution. It uses learned ego-query seeds that cross-attend to both historical memory and the predicted future geometry tokens. Crucially, a stop-gradient operation is applied to the predicted geometry tokens, ensuring that trajectory supervision does not reshape the geometric conditioning signal, thereby preserving the pretraining capabilities.
Trajectory Generation: An action head concatenates historical and predicted future ego tokens, refines them via a causal temporal transformer, and regresses a single future trajectory (position and heading) directly, without trajectory anchors or mode classification.
Training Objective: The model is finetuned with a combined loss function including the pretraining geometry objectives, an $\ell_1$ trajectory regression loss, and an auxiliary pose prediction loss.

Key Contributions

Native State Representation: The paper motivates and demonstrates that geometry (point clouds) serves as a native state space for world action models, explicitly aligning scene dynamics with the 3D coordinate space of driving actions.
Visual Geometry Pretraining: The authors pretrain a visual geometry world model to forecast future scene geometry from multiview images, capturing spatial structure and temporal dynamics without requiring occupancy annotations.
Inverse-Dynamics Formulation: GeoWAM introduces a mechanism to infer future ego motion from predicted geometric dynamics, mapping these representations to ego trajectories via a geometry-conditioned action head.
Empirical Validation: The model is validated through future-geometry prediction and both open-loop and closed-loop planning benchmarks, establishing visual geometry modeling as an effective pretraining objective.

Experimental Results

The authors evaluate GeoWAM on the nuScenes dataset for geometry prediction and NAVSIM v2 for planning.

Future Geometry Prediction (nuScenes): GeoWAM outperforms baselines, including video-based models (Epona, Cosmos 3) that reconstruct geometry from generated RGB frames, and direct geometry forecasting models (VGGT-World). It achieves the lowest Absolute Relative Error (Abs Rel) across all horizons (1s–4s) and the highest threshold accuracy ( $\delta < 1.25$ ).
Open-Loop Planning (NAVSIM navtest): GeoWAM achieves the best overall EPDMS (Extended Predictive Driver Model Score) of 90.2, surpassing strong baselines like DVGT-2, EponaV2, and various VLA-based planners. It matches the best scores in driving-direction and traffic-light compliance.
Closed-Loop Planning (NAVSIM navhard): In a challenging two-stage pseudo-closed-loop benchmark, GeoWAM achieves an EPDMS of 36.6, outperforming all competing methods, including those trained with reinforcement learning or direct PDMS-score supervision.
Qualitative Analysis: Visualizations show that GeoWAM preserves coherent scene structures (e.g., trees, poles, road markings) and captures the dynamics of surrounding agents (e.g., other vehicles following the ego) over the prediction horizon.

Significance and Claims

The paper claims that future-geometry prediction is a superior pretraining objective for autonomous driving compared to future-image generation. By placing world modeling and action prediction in a shared geometric space, GeoWAM provides driving policies with representations explicitly organized around 3D structure and motion. The authors argue that this approach yields substantially stronger driving policies than image-based alternatives because it directly grounds the policy in the physical space where driving actions are executed, avoiding the ambiguity of inferring 3D dynamics from 2D pixel variations. The work establishes that learning to forecast metric geometry enables effective transfer to motion planning without the need for future image synthesis.