🤖 machine learning

DriveJudge: Rethinking Autonomous Driving Evaluation with Vision-Language Models

This paper introduces DriveJudge, a novel autonomous driving evaluation agent that integrates Vision-Language Model reasoning with physically-grounded rule functions to achieve superior, interpretable, and context-aware performance on human-aligned benchmark tasks.

Original authors: Xinglong Sun, Kevin Xie, Jenny Schmalfuss, Despoina Paschalidou, Xiuming Zhang, Sanja Fidler, Kashyap Chitta, Jose M. Alvarez

Published 2026-06-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xinglong Sun, Kevin Xie, Jenny Schmalfuss, Despoina Paschalidou, Xiuming Zhang, Sanja Fidler, Kashyap Chitta, Jose M. Alvarez

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a self-driving car how to drive. To grade its performance, you need a judge. But here is the problem: traditional judges are like rigid, by-the-book robots, while new AI judges are like vague, opinionated critics who can’t measure distances accurately.

This paper introduces DriveJudge, a new type of evaluator that combines the best of both worlds. It’s like hiring a human driving instructor who knows when to bend the rules, but who uses a laser-measuring tape to check for safety.

Here is how DriveJudge works, broken down into simple concepts:

1. The Problem: Two Flawed Judges

Currently, we have two main ways to grade self-driving cars, and both have big flaws:

The "Rule Robot" (Traditional Metrics): This judge checks every single rule strictly. Did the car stay in the lane? Did it hit anything? Did it move forward?
- The Flaw: It lacks common sense. If a car swerves slightly out of its lane to avoid a sudden obstacle, the Rule Robot gives it a failing grade for "lane violation," even though the driver did the right thing. It’s like a teacher failing a student for raising their hand without permission, even though they were answering a question correctly.
The "Vague Critic" (Pure AI/VLM Judges): These are large AI models that look at the video and give a score based on "vibes" or general reasoning.
- The Flaw: They are bad at geometry. They might say, "The car looks safe," but they can’t actually calculate if the car is 1 inch or 1 foot away from a pedestrian. They are like a movie critic who says, "The acting felt real," but can’t tell you if the stunt was actually performed safely.

2. The Solution: DriveJudge (The Smart Inspector)

DriveJudge is an AI agent that acts like a smart inspector. It doesn’t just look at the car; it looks at the context first.

Step 1: The Context Check (The Brain): DriveJudge uses a Vision-Language Model (VLM) to understand the scene. It asks: "What is happening here? Is there construction? Is there a sudden obstacle? Is this a normal highway or a busy intersection?"
Step 2: Selective Rule Application (The Filter): Based on the context, DriveJudge decides which rules matter.
- Example: If the car is nudging around a parked truck, DriveJudge says, "Ignore the 'Stay in Lane' rule for this moment. It’s justified."
- Example: If the car is approaching a pedestrian, DriveJudge says, "The 'No Collision' rule is critical. Check it precisely."
Step 3: The Precision Check (The Ruler): For the rules it does decide to apply, DriveJudge uses exact, mathematical calculations (like traditional metrics) to measure safety and compliance. This ensures the judgment is physically accurate, not just a guess.

3. How They Trained It

To teach DriveJudge this nuance, the researchers created a massive dataset of 33,577 challenging driving clips. These weren’t just normal driving; they were "long-tail" scenarios—rare, tricky situations where human drivers might break a rule to stay safe.

Human Labeling: Humans watched these clips and answered: "Was this driving behavior reasonable?"
The Logic: If a human said "Yes, this was reasonable," but the car technically broke a rule (like drifting out of the lane), DriveJudge learned that in this specific context, that rule should be ignored.
Training Method: They used a two-step training process:
1. Supervised Fine-Tuning: Teaching the AI to recognize when to turn rules on or off.
2. Reinforcement Learning: Rewarding the AI when its final judgment matched human preferences (e.g., choosing the safer, more sensible trajectory over the technically "perfect" but dangerous one).

4. The Results: Why It’s Better

The paper shows that DriveJudge is significantly better at judging driving quality than previous methods:

It understands nuance: It correctly identifies that a car swerving to avoid an obstacle is "good driving," whereas the old Rule Robot would call it "bad driving."
It’s more accurate: In tests where humans had to choose between two driving paths, DriveJudge agreed with human preferences 6.5% more often than the previous best AI critic (DriveCritic).
It catches failures better: It is much better at spotting actual dangerous behaviors (like running a red light) because it doesn’t get confused by minor, justified rule-breaking.

5. A Creative Analogy

Think of driving evaluation like judging a dance competition:

The Rule Robot is a judge who only cares if your feet are exactly on the marked spots on the floor. If you step off the mark to avoid tripping over a cable, you get a zero, even if your dance was beautiful.
The Vague Critic is a judge who says, "I felt the rhythm," but can’t tell if you actually hit the high note or just hummed near it.
DriveJudge is a judge who watches the whole performance. If you step off the mark to avoid a cable, they say, "Good save, ignore the foot placement rule." But if you miss a high note, they use a tuner to measure exactly how far off you were. They combine common sense with precise measurement.

Summary

DriveJudge rethinks how we grade self-driving cars by adding context to precision. It stops penalizing cars for sensible rule-breaking and starts rewarding them for smart, safe decisions in complex situations. This makes the evaluation process more human-like, more accurate, and more useful for training better autonomous vehicles.

Technical Summary: DriveJudge: Rethinking Autonomous Driving Evaluation with Vision-Language Models

Problem Statement
The shift toward end-to-end policy learning in autonomous driving has exposed a critical gap in evaluation methodologies, particularly for long-tail scenarios involving non-nominal rare events and complex interactions. Existing driving metrics fall into two categories, both of which exhibit significant limitations:

Imitation-based metrics (e.g., ADE/FDE) measure similarity to expert demonstrations. They assume a fixed correct solution, ignoring the multi-modal nature of driving and the potential suboptimality or unavailability of expert trajectories.
Rule-based metrics (e.g., EPDMS) provide interpretable, physically grounded evaluations but are inherently context-agnostic. They apply rigid rules regardless of scene context, leading to over-penalization of reasonable behaviors that require temporary relaxation of constraints (e.g., nudging around obstacles).
VLM-based critics offer context awareness but suffer from ambiguous outputs, weak physical grounding, and limited spatial precision, making them unreliable for fine-grained safety assessment.

Consequently, there is a lack of long-tail-ready driving metrics that are both interpretable and context-aware.

Methodology
The authors propose DriveJudge, a driving evaluation agent that combines Vision-Language Model (VLM) reasoning with rule-grounded evaluation. DriveJudge formulates evaluation as an agentic decision-making process:

Context Interpretation: A VLM interprets the scene context using multi-modal inputs (camera observations and Bird’s-Eye View (BEV) visualizations of perception labels).
Selective Rule Invocation: Based on the context, the VLM predicts a gating vector $\hat{g}_i$ , where each element $\hat{g}^m_i \in [0, 1]$ indicates the relevance of a specific deterministic rule function $f^m$ (e.g., collision checks, lane deviation analysis). This allows the system to selectively invoke rules, ignoring those that are inappropriate for the specific context (e.g., ignoring lane-keeping constraints during a justified nudge).
Score Aggregation: The final driving score is computed by aggregating the outputs of the invoked rules, weighted by their predicted relevance. This bridges semantic reasoning (VLM) with spatial precision (deterministic rules).

Dataset and Training
To train and evaluate DriveJudge, the authors curated the DriveJudge dataset, comprising 33,577 challenging driving samples mined from the PhysicalAI-Autonomous-Vehicles dataset. These samples focus on long-tail scenarios where human driving may violate standard rule-based metrics but remains reasonable. Each sample includes human annotations on whether the behavior is reasonable.

The model undergoes a two-stage post-training pipeline:

Supervised Fine-Tuning (SFT): The model is trained to predict rule invocation decisions. Ground-truth labels are derived by identifying rules that yield low scores for human-labeled "reasonable" driving, implying those rules are irrelevant in that context.
Reinforcement Learning (RL): Using Group Relative Policy Optimization (GRPO), the model is further optimized for trajectory preference selection. The reward is defined by whether the metric’s ranking of two candidate trajectories aligns with human preference labels.

Key Contributions

DriveJudge Framework: A context-aware evaluation agent that uses VLM-guided tool invocation to selectively apply physically grounded rules, enabling interpretable and context-sensitive assessment.
DriveJudge Dataset: A large-scale dataset with human annotations on complex driving maneuvers, labeling whether observed behavior is ultimately reasonable, enabling the study of driving evaluation in long-tail scenarios.
Benchmark Tasks: The formulation of two human-aligned benchmark tasks for evaluating driving metrics: Driving Quality Classification (distinguishing reasonable from unreasonable driving) and Trajectory Preference Selection (aligning with human preference between candidate actions).
Policy Benchmarking: An evaluation of state-of-the-art driving policies under DriveJudge, revealing that conventional metrics may underestimate the capability of modern policies and that benchmarks like navtest may be approaching saturation.

Results
DriveJudge significantly outperforms existing baselines on both benchmark tasks:

Driving Quality Classification: DriveJudge-8B achieves an AUC of 78.90, outperforming EPDMS by +21.23 AUC and improving failure detection (AP−) by +32.70. It also surpasses zero-shot VLMs (GPT-5, Gemini 3.1) and supervised non-VLM classifiers.
Trajectory Preference Selection: DriveJudge-8B achieves an accuracy of 82.83%, outperforming the specialized preference model DriveCritic by 6.5% and EPDMS by a significant margin.
Ablation Studies: Experiments confirm the necessity of combining VLM reasoning with rule-grounded tool invocation. VLM-only direct scoring suffers from spatial imprecision, while non-VLM agents lack contextual understanding. Furthermore, SFT provides a critical warm start for RL, which yields substantial gains in preference alignment compared to zero-shot initialization.
Policy Evaluation: When benchmarking state-of-the-art planning models (e.g., Hydra-MDP, DriveSuprim, ZTRS), DriveJudge consistently assigns higher scores than EPDMS, suggesting that rule-based metrics undervalue contextually reasonable behaviors. The small performance gaps among top models under DriveJudge indicate potential benchmark saturation.

Significance
The paper argues that DriveJudge sets a new standard for interpretable and precise driving evaluation by addressing the limitations of both rigid rule-based metrics and ambiguous VLM-only judgments. By selectively invoking rules based on scene context, DriveJudge enables a more human-aligned assessment of driving quality, particularly in long-tail scenarios. The authors highlight that this approach reveals stronger driving capabilities in modern policies than conventional metrics suggest and underscores the need for future research beyond incremental rule-based metric optimization.