💬 NLP

LLM4AD: Large Language Models for Autonomous Driving -- Concept, Review, Benchmark, Experiments, and Future Trends

This paper introduces the concept of Large Language Models for Autonomous Driving (LLM4AD), provides a comprehensive review and benchmark for evaluating their capabilities, presents extensive real-world experiments on vehicle platforms, explores future trends like Vision-Language Diffusion, and discusses key challenges including safety, latency, and privacy.

Original authors: Can Cui, Yunsheng Ma, Sung-Yeon Park, Zichong Yang, Yupeng Zhou, Peiran Liu, Juanwu Lu, Juntong Peng, Jiaru Zhang, Ruqi Zhang, Lingxi Li, Yaobin Chen, Jitesh H. Panchal, Amr Abdelraouf, Rohit Gupta, K

Published 2026-03-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Can Cui, Yunsheng Ma, Sung-Yeon Park, Zichong Yang, Yupeng Zhou, Peiran Liu, Juanwu Lu, Juntong Peng, Jiaru Zhang, Ruqi Zhang, Lingxi Li, Yaobin Chen, Jitesh H. Panchal, Amr Abdelraouf, Rohit Gupta, Kyungtae Han, Ziran Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine your car isn't just a machine with sensors and computers, but a vehicle with a brain that can actually talk to you, understand your mood, and learn your habits. That's the big idea behind this paper, which introduces a concept called LLM4AD (Large Language Models for Autonomous Driving).

Here is a simple breakdown of what the researchers did, using everyday analogies:

1. The Big Idea: Giving the Car a "Brain"

Traditionally, self-driving cars are like highly skilled but rigid robots. They follow strict rules: "If the light is red, stop. If there is a car ahead, slow down." They don't really understand why you want to go somewhere or how you feel about the drive.

This paper proposes giving the car a Large Language Model (LLM) as its "brain." Think of this LLM as a super-smart co-pilot who has read every traffic law, watched millions of driving videos, and can understand human conversation.

Old Way: You press a button to set the speed.
New Way: You say, "I'm running late, but I don't want to scare my passengers," and the car understands the nuance. It knows to drive a bit faster but keep the turns smooth and gentle.

2. How It Works: The "Brain" vs. The "Hands"

The paper describes a clever division of labor:

The LLM is the Brain: It listens to you, looks at the road (via camera data), and figures out the strategy. It decides, "Okay, the driver wants to be safe, so let's keep a bigger gap between us and the car in front."
The Car's Computer is the Hands: The LLM doesn't directly grab the steering wheel. Instead, it writes a quick "recipe" or code (like a set of instructions) and hands it to the car's existing, ultra-fast safety systems. The "hands" execute the move instantly.

Analogy: Imagine a chef (the LLM) and a sous-chef (the car's computer). The chef tastes the soup, decides it needs more salt, and writes a note: "Add 2 grams of salt." The sous-chef immediately does it. The chef doesn't run to the stove and stir the pot themselves because they are too slow and might burn the soup.

3. The Testing Grounds: Simulations and Real Roads

The researchers didn't just talk about this; they built it and tested it in three ways:

The Video Game Test (Simulation): They put their AI in a driving simulator (like Grand Theft Auto but for scientists). They gave the AI thousands of instructions like "Overtake the slow truck" or "Change lanes to the right." They found that the AI could follow these complex human instructions much better than old-school rule-based robots, especially when given a few examples first.
The "Eye-Test" (Visual Understanding): They tested if the AI could look at a picture of a rainy street and answer questions like, "Is it safe to turn left?" The AI got pretty good at this, proving it can "see" and "think" at the same time.
The Real Car Test: They put a real Lexus on the road with a human safety driver watching. Passengers gave voice commands like, "Drive more conservatively" or "I'm in a hurry."
- Result: The car adjusted its driving style perfectly. If you said you were in a rush, it sped up (safely). If you said you felt motion sickness, it drove smoother.
- The "Memory" Trick: They gave the car a memory bank. If you told it, "I hate sudden stops," it remembered that for the next trip. Over time, the car learned your specific style, making the ride feel personal.

4. The Future: The "Magic 8-Ball" of Driving

The paper also looks ahead to a new technology called Diffusion Models (ViLaD).

Current AI (Autoregressive): Think of this like a person writing a story one word at a time. They have to write "The" then "car" then "is" before they can finish the sentence. This takes time.
Future AI (Diffusion): Think of this like a Magic 8-Ball or a photo developing in a darkroom. Instead of writing word-by-word, the AI sees the whole picture of the future (the next few seconds of driving) all at once and "refines" it. It's much faster and can see the whole road ahead simultaneously, not just the next step.

5. The Hurdles: Why We Aren't There Yet

Despite the cool tech, the paper admits there are big challenges:

Speed (Latency): The "Brain" (LLM) is a bit slow. It takes a second or two to think. In a self-driving car, a split-second delay can be dangerous. So, the LLM is only used for high-level decisions (like "be polite"), while the car's fast computer handles emergency braking.
Hallucinations: Sometimes AI makes things up. If the LLM thinks there is a ghost car that isn't there, it might brake unnecessarily. The researchers use safety "guardrails" to stop the car from doing anything crazy.
Privacy: The car is learning your habits and listening to your voice. The paper discusses how to keep this data safe so strangers can't hack your driving preferences.

The Bottom Line

This paper is a roadmap for turning self-driving cars from strict rule-followers into intelligent, empathetic partners. They can understand your mood, remember your preferences, and explain their decisions to you. While there are still safety and speed hurdles to clear, the future of driving looks less like a robot taxi and more like a car that truly knows you.

1. Problem Statement

The integration of Large Language Models (LLMs) into Autonomous Driving (AD) systems offers the potential to revolutionize human-vehicle interaction, decision-making, and personalization. However, current research lacks a unified framework, standardized benchmarks, and rigorous real-world validation. Key challenges include:

Latency: LLMs often require seconds to process, which is unsafe for real-time, safety-critical control loops.
Hallucinations: LLMs may generate factually incorrect or nonsensical outputs, posing safety risks.
Evaluation Gap: Existing studies lack comprehensive benchmarks for instruction-following, reasoning, and multimodal perception in driving contexts.
Deployment: There is a trade-off between the computational power required for large models and the limited resources of on-board vehicle hardware.
Safety & Trust: Ensuring that LLM-generated policies align with safety constraints and that explanations are transparent and accurate.

2. Methodology and Framework

The paper proposes a comprehensive framework called LLM4AD, where the LLM acts as the "brain" for high-level decision-making, while traditional modules handle perception ("eyes") and low-level control ("hands").

A. The LLM4AD Architecture

The system processes five distinct inputs to generate two primary outputs:

Inputs:
- Human Instructions ( $I$ ): Natural language commands (e.g., "Drive more conservatively").
- System Messages ( $S$ ): Constraints, role definitions, and traffic rules.
- Situation Descriptor ( $C$ ): Structured natural language descriptions of the driving context (derived from sensors).
- History Memory ( $H$ ): User profiles and past interactions for personalization.
- Human Feedback ( $F$ ): Evaluations of previous driving policies.
Outputs:
- Reasoning Thoughts ( $R$ ): Chain-of-thought explanations for transparency.
- Generated Policies ( $P$ ): Executable code (Language Model Programs - LMPs) or control parameters.
Executor: Translates the generated code into actions within the vehicle's control stack (e.g., Autoware).

B. Learning Strategies

The paper reviews and employs various learning methods:

Supervised Fine-Tuning (SFT): Adapting pre-trained models (e.g., LLaMA, Qwen) to driving tasks.
Prompt Engineering: Using Zero-shot, Few-shot, and Chain-of-Thought (CoT) prompting to guide reasoning without parameter updates.
Retrieval-Augmented Generation (RAG): Retrieving relevant historical driving scenarios from a vector database to enhance context and personalization.
Reinforcement Learning (RL): Using human feedback (RLHF) or environment rewards to optimize policies.

C. Future Trend: Vision-Language Diffusion (ViLaD)

The paper proposes ViLaD, a shift from autoregressive VLMs to Masked Diffusion Models. Unlike sequential token generation, ViLaD generates driving actions in parallel, enabling bidirectional reasoning and significantly reducing inference latency.

3. Key Contributions

A. Comprehensive Benchmarks

The authors introduce three new benchmarks to evaluate LLM4AD systems:

LaMPilot-Bench: A dataset of 4.9k semi-human-annotated traffic scenes on the HighwayEnv simulator. It evaluates instruction-following for tasks like lane changing and overtaking.
CARLA Leaderboard 1.0 Benchmark: A modified version of the standard CARLA benchmark using natural language navigation instructions instead of GPS coordinates to test closed-loop performance in complex urban environments.
NuPlanQA: A multi-view Visual Question Answering (VQA) benchmark based on the nuPlan dataset. It evaluates high-level reasoning across three domains: Road Environment Perception, Spatial Relations, and Ego-Centric Reasoning.

B. Real-World Experiments

Two distinct real-vehicle experiments were conducted on a drive-by-wire Lexus RX450h:

Talk2Drive (On-Cloud LLM):
- Goal: Personalized decision-making via cloud-based LLMs.
- Mechanism: Converts voice commands to text, processes them with GPT-4 (cloud), and generates LMPs to adjust controller parameters (e.g., target velocity, look-ahead distance).
- Safety: Implements a two-stage safety check (format and parameter verification) before execution.
On-Board VLM (Edge Deployment):
- Goal: Personalized motion control via an on-board Vision-Language Model.
- Mechanism: Uses a fine-tuned 8B-parameter Qwen-VL model with LoRA and 4-bit quantization (AWQ) for edge deployment. It integrates visual inputs (weather, traffic) and RAG-based memory to adjust PID and MPC controller parameters directly.

C. ViLaD Framework

Proposed a novel end-to-end driving framework using masked diffusion. It demonstrates superior parallel generation capabilities compared to autoregressive models, addressing the "reversal curse" and improving inference speed.

4. Results

A. Benchmark Performance

LaMPilot-Bench: LLMs with human feedback (GPT-4) achieved a driving score of 64.0, significantly outperforming rule-based baselines (IDM: 17.3). However, collision rates remained non-zero (0.9%), highlighting safety gaps in direct control.
CARLA Leaderboard: LLMs with human feedback achieved a Driving Score of 65.4, comparable to expert reinforcement learning baselines (Roach: 63.4), whereas zero-shot LLMs struggled (0.00).
NuPlanQA: The proposed BEV-LLM baseline achieved 78.7% accuracy, outperforming other open-source MLLMs in ego-centric reasoning and traffic light detection.

B. Real-World Experiment Results

Personalization (Talk2Drive):
- Takeover Rate: Reduced significantly across all scenarios. For example, in highway overtaking, takeover rates dropped from 0.77 (baseline) to 0.31 (Talk2Drive) for indirect commands.
- Memory Module: Using a memory module reduced takeover rates by up to 65.2% compared to systems without memory.
On-Board VLM:
- Performance: Outperformed baseline controllers and GPT-4o in safety, comfort, and alignment metrics.
- Takeover Rate: Achieved a 76.9% reduction in takeover rates for indirect instructions compared to the baseline.
- Latency: The on-board VLM achieved inference latencies of ~1.2–1.8s (cloud) and ~0.18–0.32s (on-board optimized), proving feasibility for high-level adjustments but not immediate emergency braking.

C. ViLaD Performance

Planning: ViLaD-SFT achieved an average L2 error of 1.85m on nuScenes with a 0.00% failure rate, outperforming SFT baselines (e.g., Llama-3.2: 2.07m, 0.06% failure).
Efficiency: ViLaD-Opt achieved an inference time of 2.78s (with caching), which is 6.5x to 14x faster than autoregressive VLM baselines (28–60s).

5. Significance and Future Directions

Paradigm Shift: The paper establishes LLMs not as direct low-level controllers, but as high-level "policy adjusters" that personalize driving styles and interpret complex human intent, while leaving safety-critical real-time control to robust, low-latency systems.
Safety First: It highlights that while LLMs excel at reasoning and personalization, their latency and hallucination risks necessitate a hybrid architecture with rule-based safety layers.
Standardization: The introduction of LaMPilot-Bench, CARLA Leaderboard 1.0, and NuPlanQA provides the community with essential tools for fair comparison and progress tracking.
Future Trends: The proposal of ViLaD suggests a move toward diffusion-based models to overcome the sequential bottlenecks of autoregressive LLMs, promising faster, more robust, and bidirectional reasoning for autonomous driving.

In conclusion, this work provides a holistic roadmap for integrating LLMs into autonomous driving, balancing the promise of human-centric, personalized AI with the rigorous safety requirements of real-world deployment.