🤖 AI

VERDI: VLM-Embedded Reasoning for Autonomous Driving

VERDI is a training-time framework that distills the commonsense reasoning capabilities of large Vision-Language Models into modular differentiable end-to-end autonomous driving stacks by aligning intermediate module outputs with textual reasoning features, thereby achieving superior safety and performance without the high inference costs of deploying massive VLMs.

Original authors: Bowen Feng, Zhiting Mei, Julian Ost, Filippo Ghilotti, Baiang Li, Roger Girgis, Anirudha Majumdar, Felix Heide

Published 2026-04-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Bowen Feng, Zhiting Mei, Julian Ost, Filippo Ghilotti, Baiang Li, Roger Girgis, Anirudha Majumdar, Felix Heide

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to drive a car. You have two main ways to do this:

The "Memorizer" Approach: You show the robot thousands of videos of humans driving. The robot learns to copy exactly what the human did. If the human turned left, the robot turns left. But if the robot sees something it has never seen before (like a giant inflatable duck on the road), it panics because it doesn't understand why the human drove that way, only what they did.
The "Super-Brain" Approach: You give the robot a giant, super-smart brain (a massive AI model) that can read, see, and reason like a human. It can look at the road and say, "That truck is wobbling, so I should slow down." The problem? This super-brain is so heavy and slow that it takes hours to make a single decision. It's like trying to run a marathon while carrying a piano on your back. It's too slow for real-time driving.

Enter VERDI.

The paper introduces VERDI (VLM-Embedded Reasoning for autonomous DrIving), which is like a brilliant new teaching method that gets the best of both worlds.

The Core Idea: The "Shadow Teacher"

Think of VERDI as a training camp where a small, fast student (the driving robot) learns from a giant, slow genius teacher (the Vision-Language Model or VLM).

The Problem: Usually, the student just copies the teacher's final answer (the steering wheel movement).
The VERDI Solution: Instead of just copying the answer, VERDI forces the student to think like the teacher.

Here is how the training works, using a simple analogy:

1. The "Think Aloud" Session

Imagine the teacher (the giant AI) is looking at a driving scene. Instead of just turning the wheel, the teacher talks out loud, step-by-step:

Perception: "I see a red truck ahead and a pedestrian on the left."
Prediction: "The truck is slowing down, and the pedestrian is about to cross."
Planning: "Because the truck is slowing, I need to slow down too, but I must stay in my lane to avoid the pedestrian."

2. The "Mind-Meld"

In the old days, the student robot only saw the final result (the steering wheel turn). With VERDI, the robot is forced to match its internal "thoughts" to the teacher's spoken words.

When the teacher says "I see a red truck," the robot's "eyes" (perception module) must light up with a specific pattern that matches the meaning of "red truck."
When the teacher says "slow down," the robot's "brain" (planning module) must align with that logic.

The robot isn't just memorizing the move; it is internalizing the reasoning. It learns the common sense behind the decision.

3. Graduation Day (Inference)

Once the training is done, the giant teacher (the slow AI) is fired. It's too heavy to carry into the car.

The student robot is now left alone in the driver's seat.
Because it learned how to think during training, it can now make smart, safe decisions instantly, without needing the giant teacher. It has the speed of a race car but the common sense of a human driver.

Why is this a big deal?

Speed: The old "Super-Brain" methods are too slow for real cars (they might take seconds to decide, which is an eternity in a crash). VERDI is as fast as a standard robot, making it safe for real roads.
Safety: Standard robots often crash in weird situations because they haven't "seen" that exact scenario in their training data. VERDI, having learned the logic of driving, can handle new, scary situations (like a deer jumping out) much better because it can reason through them.
Efficiency: It's like taking a PhD thesis and distilling it into a 10-minute summary. The car doesn't need the whole library; it just needs the wisdom.

The Results

The authors tested VERDI in a virtual driving simulator (HugSim) and on real-world data (nuScenes).

The Result: VERDI drove safer and smoother than previous methods. It avoided collisions 10% more often than the best existing robot drivers.
The Analogy: If the old robot was a student who memorized the answer key but failed when the test questions changed, VERDI is the student who actually understood the math and aced the test, even with new questions.

In short: VERDI teaches a fast robot to drive by making it "think" like a smart human during training, so it can drive safely and quickly without needing a supercomputer on board.

1. Problem Statement

Autonomous driving (AD) systems face significant challenges in real-world environments characterized by partial observability, semantic ambiguity, and complex interactions with dynamic agents (pedestrians, other vehicles).

Limitations of Current End-to-End (E2E) Models: While modular differentiable E2E models (e.g., UniAD, VAD) perform well on benchmarks, they rely heavily on supervised learning from human driving trajectories. They lack access to the underlying reasoning processes and commonsense knowledge that human drivers use to make near-optimal decisions in edge cases. Consequently, they struggle with generalization in unseen scenarios.
Limitations of Direct VLM Inference: Recent attempts to use Vision-Language Models (VLMs) directly for planning at inference time offer better reasoning but are impractical for deployment. Large VLMs (e.g., 70B parameters) suffer from high latency (requiring >160GB memory and running at <1 token/sec on some hardware) and lack safety decomposition. Furthermore, direct VLM planning often suffers from hallucinations due to poor spatial reasoning in 2D image-only inputs.

Core Question: How can modular AD models leverage human-like decision-making processes, reasoning, and commonsense knowledge in real-time without incurring the computational costs of running large VLMs at inference?

2. Methodology: VERDI

The authors propose VERDI (VLM-Embedded Reasoning for autonomous DrIving), a training-time framework that distills the latent reasoning capabilities of a large VLM into a lightweight, modular E2E AD stack.

A. Core Concept: Latent Feature Alignment

Instead of querying a VLM during inference, VERDI aligns the intermediate outputs of the AD model with the reasoning outputs of a VLM during training.

Teacher (VLM): A large VLM (Qwen-2.5-VL-72B) is queried using Chain-of-Thought (CoT) prompting to generate text explanations for three specific subtasks:
1. Perception: Describing the scene (lanes, agents, objects, locations).
2. Prediction: Describing the future actions of other agents.
3. Planning: Describing the ego vehicle's intended actions.
Student (E2E Model): A modular differentiable E2E model (based on VAD) processes sensor inputs to generate future trajectories.
Alignment Mechanism: The text responses from the VLM are encoded into latent feature vectors ( $f_M$ ) using a text encoder. The intermediate feature maps of the E2E model's perception, prediction, and planning modules ( $F_P$ ) are projected into the same shared latent space using learnable Progressive Feature Projectors (PFP).
Loss Function: A cosine similarity loss ( $L_f$ ) is applied between the projected E2E features and the VLM text embeddings. The total loss combines standard supervised losses (e.g., trajectory $\ell_2$ distance) with this alignment loss:
$L_i = L_e(\theta_i) + \lambda_i L_f(f^i_P, f^i_M)$
This forces the E2E model to "internalize" the structured reasoning and commonsense knowledge of the VLM within its own latent representations.

B. Architecture Details

Perception Alignment: Aligns BEV (Bird's Eye View) image features with VLM scene descriptions to improve object detection and spatial understanding.
Prediction Alignment: Aligns agent query features with VLM descriptions of other agents' future motions.
Planning Alignment: Aligns the ego planning latent features with VLM descriptions of driving actions (e.g., "turn left slightly," "maintain speed").
Inference: At runtime, the VLM and text encoder are discarded. The E2E model runs independently, utilizing the distilled reasoning capabilities embedded in its weights, ensuring real-time performance.

3. Key Contributions

Novel Framework: Introduction of VERDI, a training-time distillation method that embeds VLM reasoning into modular E2E AD stacks without requiring VLMs at inference.
Latent Space Alignment: A method to align individual driving subtasks (perception, prediction, planning) with corresponding natural language reasoning in a shared latent space, effectively transferring semantic understanding.
Comprehensive Evaluation: Extensive validation on both open-loop (nuScenes) and closed-loop (HugSim) benchmarks, demonstrating that reasoning-enhanced models outperform standard supervised models and large VLM-based baselines in both accuracy and safety.

4. Experimental Results

Open-Loop Evaluation (nuScenes)

Trajectory Accuracy: VERDI achieved an 11% improvement in average $\ell_2$ distance compared to the direct baseline (VAD-Base).
Efficiency: VERDI runs at 4.5 FPS, significantly faster than VLM-based methods like DriveVLM (2.43 FPS) and OmniDrive (0.44 FPS).
Comparison: It outperforms other E2E methods (UniAD) and VLM-based methods that require ego-status information, achieving the best trade-off between speed and accuracy.

Closed-Loop Evaluation (HugSim Simulator)

Safety: VERDI achieved a 10% improvement in Non-Collision (NC) rate compared to VAD-Base.
Overall Performance: It achieved the highest HugSim Driving Score (HDScore) among all baselines, including Alpamayo (a reasoning-based VLA).
- VERDI HDScore: 0.200
- Alpamayo HDScore: 0.041 (approx. 5x lower).
Efficiency: VERDI is ~200x smaller in parameter count and ~10x faster in inference latency than Alpamayo.

Ablation Studies

Module Alignment: Aligning all three modules (Perception, Prediction, Planning) yielded the best results. Aligning only perception helped short-term accuracy, while aligning prediction/planning helped long-term planning.
VLM Quality: Using a larger, higher-quality VLM (Qwen-72B) for distillation significantly outperformed smaller models (Qwen-7B) or random/zero embeddings, confirming that the gains come from meaningful reasoning distillation.
Qualitative Analysis: VERDI demonstrated superior ability to detect occluded agents, predict correct future motions for other vehicles, and plan safer trajectories in edge cases (e.g., avoiding collisions with barriers that baselines missed).

5. Significance and Impact

Bridging the Gap: VERDI successfully bridges the gap between the high reasoning capability of large VLMs and the real-time, safety-critical requirements of autonomous driving deployment.
Safety without Latency: It proves that "human-like" reasoning can be embedded into lightweight models, improving safety (collision avoidance) and generalization in long-tail scenarios without the prohibitive latency of running large models on the vehicle.
Interpretability: By aligning latent features with natural language reasoning, the method offers a pathway toward more interpretable AD systems where the model's internal state corresponds to human-understandable concepts.
Scalability: The approach is scalable; as VLMs improve, the distillation process can be repeated to further enhance the driving stack without changing the inference architecture.

In conclusion, VERDI represents a paradigm shift from "querying a VLM for every decision" to "training a driving model to think like a VLM," offering a practical, high-performance solution for the next generation of autonomous vehicles.