DriveCritic: Towards Context-Aware, Human-Aligned Evaluation for Autonomous Driving with Vision-Language Models

DriveCritic is a novel framework comprising a curated dataset of context-critical scenarios and a fine-tuned Vision-Language Model that significantly improves autonomous driving evaluation by aligning metric scores with human judgment through enhanced context awareness.

The Gist

Imagine you are teaching a robot to drive a car. You want it to be safe, efficient, and polite, just like a human driver. But how do you know if the robot is doing a good job?

For a long time, the industry has used a "Rulebook" to grade the robot. This rulebook is like a strict math teacher who only looks at numbers: "Did you stay exactly in the center of the lane? Yes/No. Did you move forward exactly 50 meters? Yes/No."

The Problem: The Rulebook is "Context-Blind"
The paper calls this old method EPDMS. The problem is that the Rulebook doesn't understand why a driver does something.

• The Scenario: Imagine a car is stopped in front of you. To get around it, you have to gently nudge your car slightly into the next lane for a few seconds.
• The Human View: A human driver says, "Good job! You were safe, you kept moving, and you only moved over because you had to."
• The Rulebook View: The Rulebook screams, "VIOLATION! You left the center of your lane! You get a bad grade!" It doesn't care that you were avoiding a crash or moving forward; it only sees that you broke the "stay in the middle" rule.

Because of this, the robot might learn to be overly cautious and get stuck, or it might get a bad grade for doing the right thing.

The Solution: Enter "DriveCritic"
The authors of this paper created a new system called DriveCritic. Think of DriveCritic not as a math teacher, but as a seasoned driving instructor sitting in the passenger seat.

Here is how DriveCritic works, broken down into simple parts:

1. The "Tricky Test" (The Dataset)

The researchers didn't just grab random driving clips. They specifically looked for the "tricky" moments where the old Rulebook fails.

• Analogy: Imagine a driving test where the examiner asks, "Is it okay to swerve slightly to avoid a pothole?" The old rulebook says "No, you must stay straight." The new test (DriveCritic) collects thousands of these "gray area" situations and asks real human experts: "Which driver made the smarter choice?"
• They created a massive library of these "tricky" scenarios, labeled with what a human expert would prefer.

2. The "Smart Judge" (The Model)

They built an AI judge using a Vision-Language Model (VLM).

• Analogy: Think of a regular AI as a calculator. It can add numbers. But DriveCritic is like a detective with a camera.
• It doesn't just look at numbers. It looks at the video (what the car sees), the map (where the lanes are), and the context (is there a stopped car? is there a red light?).
• It combines this visual information with the "Rulebook" numbers to make a decision.

3. The "Training Camp" (Two-Stage Learning)

You can't just turn on a smart AI and expect it to be a perfect judge immediately. The authors trained it in two steps:

• Stage 1 (Supervised Fine-Tuning): They showed the AI thousands of examples of "Human Expert vs. Robot" choices and said, "Look, here is what the expert chose, and here is why." The AI learned to mimic the expert's reasoning.
• Stage 2 (Reinforcement Learning): This is like a video game. The AI plays the role of the judge. If it agrees with the human expert, it gets a "point." If it disagrees, it loses a point. Over time, it learns to think exactly like a human expert, not just follow a script.

Why This Matters

The results are impressive. When tested on these tricky scenarios:

• The old Rulebook (EPDMS) got it right only 41% of the time. It was basically guessing.
• The new DriveCritic got it right 76% of the time.

The Big Picture:
DriveCritic is a step toward making self-driving cars that don't just follow rigid rules, but understand the spirit of the road. It helps us evaluate if a self-driving car is truly "safe and smart" in the messy, unpredictable real world, rather than just being good at following a spreadsheet.

In short: DriveCritic teaches us how to grade self-driving cars the way a human would—with common sense and context, not just a ruler.

Technical Summary

1. Problem Statement

The evaluation of autonomous driving (AD) planners faces a critical gap between state-of-the-art (SOTA) quantitative metrics and expert human judgment.

• Limitations of Current Metrics: Existing metrics like the Extended Predictive Driver Model Score (EPDMS) are rule-based and "context-blind." They rely on fixed thresholds for safety, progress, and lane keeping. Consequently, they often penalize contextually appropriate human behaviors (e.g., briefly nudging out of a lane to bypass a stopped vehicle or slowing down for safety in ambiguous situations) while favoring rigid, overly conservative trajectories that lack real-world efficiency.
• The Gap: While closed-loop simulation is the gold standard, it is computationally expensive and difficult to scale. Open-loop evaluation is scalable but currently lacks the nuance to capture the trade-offs human drivers make between safety, progress, and social norms in complex scenarios.
• Goal: To develop an evaluation framework that is context-aware and aligned with expert human preferences, capable of adjudicating between trajectory pairs in ambiguous scenarios where rule-based metrics fail.

2. Methodology

The authors propose DriveCritic, a framework comprising a curated dataset and a Vision-Language Model (VLM) based evaluator.

A. The DriveCritic Dataset

• Source: Constructed from the NAVSIM benchmark (OpenScene/nuPlan data).
• Construction Strategy: The authors identified scenarios where EPDMS scores diverge from human intuition, specifically focusing on two diagnostic cases:
  1. Lane-Progress Trade-off: Scenarios where a human trajectory sacrifices perfect lane keeping (LK=0) to maintain progress, versus a vocabulary trajectory that stays perfectly in-lane (LK=1) but makes less progress.
  2. Progress-only Contrast: Scenarios where a human trajectory shows lower progress due to justified conservatism, versus a vocabulary trajectory with higher progress that might be overly aggressive.
• Annotation: The dataset contains 5,730 trajectory pairs.
  • Test Set: Annotated by a domain expert (5+ years AD research experience) to ensure high-quality ground truth.
  • Train Set: Uses a mix of expert-verified labels and pseudo-labels generated by GPT-5 (which achieved 82% accuracy on the test set) to scale the dataset.
• Format: Pairwise preference data (Trajectory A vs. Trajectory B) with human-selected winners.

B. The DriveCritic Model

• Backbone: A fine-tuned Qwen2.5-VL-7B (Vision-Language Model).
• Inputs: The model integrates multi-modal data:
  1. Visual: Stitched three-camera views (left-front, front, right-front).
  2. Symbolic/Map: Bird's Eye View (BEV) map with overlaid candidate trajectories, drivable areas, and agents.
  3. Ego State: Velocity, acceleration, and driving command.
  4. Rule-based Scores: EPDMS sub-scores (specifically Ego Progress and Lane Keeping) to provide symbolic context.
• Training Pipeline (Two-Stage):
  1. Supervised Fine-Tuning (SFT): The model is first trained on 1,100 pairs using Chain-of-Thought (CoT) reasoning traces generated by GPT-5. This "warms up" the model to follow the evaluation format and reason step-by-step before RL.
  2. Reinforcement Learning Fine-Tuning (RLVR): The model is further refined using the Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO) algorithm. This stage optimizes the model to maximize accuracy and format adherence based on verifiable rewards (matching human labels).

3. Key Contributions

1. Identification of Context Blindness: The paper systematically demonstrates that SOTA rule-based metrics (EPDMS) fail to align with human judgment in nuanced scenarios, particularly regarding the trade-off between lane keeping and progress.
2. DriveCritic Dataset: A new benchmark dataset of 5,730 challenging trajectory pairs annotated with pairwise expert human preferences, specifically targeting the failure modes of current metrics.
3. DriveCritic Model: A novel VLM-based evaluator that leverages visual and symbolic context to make human-aligned judgments. It achieves 76.0% accuracy on the test set, significantly outperforming baselines.
4. Training Paradigm: A robust two-stage training pipeline (SFT + RLVR) that stabilizes VLM training for evaluation tasks and improves robustness against input ordering.

4. Experimental Results

• Accuracy: DriveCritic achieves 76.0% accuracy on the DriveCritic test set.
  • Comparison: It significantly outperforms the rule-based EPDMS (41.4%), zero-shot VLMs like GPT-5 (55.2%) and Qwen2.5-VL (48.0%), and a supervised pairwise classifier (64.8%).
• Ablation Studies:
  • SFT alone improves accuracy to 64.5%.
  • RL alone (without SFT) performs poorly (46.4%), highlighting the necessity of the warm-up stage.
  • The full pipeline (SFT + DAPO + Format/Accuracy rewards) yields the best results.
• Robustness: DriveCritic demonstrates high Robustness Rate (RR) of 81.8% when the order of trajectories (A vs. B) is swapped in the prompt, outperforming both the supervised classifier and base VLMs. This indicates the model learns invariant reasoning rather than relying on positional bias.
• Qualitative Analysis: The model successfully generates reasoning that aligns with human logic (e.g., recognizing that a slight lane deviation is a "justified nudge" rather than a violation), whereas EPDMS strictly penalizes the deviation.

5. Significance and Future Outlook

• Impact: DriveCritic provides a reliable, scalable, and human-centric foundation for evaluating autonomous driving systems. It bridges the gap between rigid rule-based metrics and the complex, context-dependent nature of human driving.
• Applications: The model can be used to generate preference signals for Reinforcement Learning from Human Feedback (RLHF) pipelines to train better end-to-end driving planners (e.g., integrating with TrajHF).
• Limitations:
  • Reliance on VLMs introduces potential sensitivity to prompts and hallucinations.
  • The dataset is currently limited in scale and relies on a single annotator.
  • Computational costs for VLM inference are higher than rule-based metrics.
• Future Work: Expanding the dataset across diverse domains, integrating temporal information, and exploring knowledge distillation to create lighter-weight critics for deployment.

In conclusion, DriveCritic represents a paradigm shift from static, rule-based evaluation to dynamic, context-aware evaluation using Vision-Language Models, offering a more realistic assessment of autonomous driving performance.