KnowVal: A Knowledge-Augmented and Value-Guided Autonomous Driving System

KnowVal is a novel autonomous driving system that enhances planning performance and safety by synergistically integrating visual-language reasoning with a comprehensive driving knowledge graph and a value-guided trajectory assessment model, achieving state-of-the-art results on multiple benchmarks.

The Gist

Imagine you are teaching a brand-new driver how to navigate the world.

The Old Way (Current AI):
Most self-driving cars today learn like a student who only watches hours of driving videos. They mimic exactly what a human driver did in a specific situation.

• The Problem: If the human driver in the video made a mistake, the AI learns that mistake. If the AI encounters a weird situation it hasn't seen before (like a cow on the road in a snowstorm), it panics because it has no "common sense" or "rules" to fall back on. It's like a student who memorized the answers to a math test but doesn't understand why the math works.

The New Way (KnowVal):
The paper introduces KnowVal, a self-driving system that acts less like a mimic and more like a wise, experienced driver who carries a rulebook and a moral compass.

Here is how KnowVal works, broken down into three simple parts:

1. The "Eagle Eye" that Talks to a "Librarian"

Most cars just look at the road with cameras. KnowVal has two superpowers that talk to each other:

• The Eagle Eye (Perception): It sees everything, even the weird stuff. It notices a puddle, a pedestrian in a dark coat, or that it's raining at night.
• The Librarian (Knowledge Retrieval): Instead of just guessing what to do, the Eagle Eye asks the Librarian: "Hey, I see a puddle and a pedestrian. What does the rulebook say?"
• The Magic: The Librarian pulls up the exact traffic law and the "defensive driving principle" (e.g., "Slow down near puddles so you don't splash the pedestrian").
• The Loop: If the Librarian says, "I need more info to be sure," the Eagle Eye zooms in to look closer. They work together to understand the scene perfectly.

2. The "Rulebook" (The Knowledge Graph)

Think of this as a massive, organized library of everything a driver should know. It's not just a list of laws; it's a web of connections.

• It contains Traffic Laws (e.g., "Stop at red lights").
• It contains Defensive Driving (e.g., "If it's foggy, increase your distance").
• It contains Ethics (e.g., "Never swerve into a crowd to avoid a small obstacle").
• The Innovation: Unlike other systems that might "hallucinate" (make things up) when asked a question, KnowVal retrieves the exact original text of the rule. It doesn't summarize it; it quotes the law directly to ensure accuracy.

3. The "Conscience" (The Value Model)

This is the most important part. Once the car sees the road and reads the rules, it needs to decide: "Is this a good idea?"

• Imagine the car generates three possible paths:
  1. Speed up to beat the light.
  2. Slow down and wait.
  3. Swerve slightly.
• The Value Model acts like a strict but fair judge. It looks at the "Conscience" (the retrieved rules) and scores each path.
  • Path 1: "You might make it, but you risk hitting a pedestrian. Score: -1 (Bad)."
  • Path 2: "Safe, legal, and polite. Score: +1 (Good)."
• The car then picks the path with the highest score. This ensures the car isn't just efficient; it's safe, legal, and kind.

Why is this a big deal?

The researchers tested KnowVal in real-world simulations and found:

• Fewer Crashes: It had the lowest collision rate on a major test dataset (nuScenes).
• Better Decisions: It handled tricky situations (like driving through a tunnel or avoiding splashing pedestrians) much better than current top-tier systems.
• Explainable: If you ask the car, "Why did you stop?" it can tell you: "I stopped because I saw a pedestrian near a puddle, and Rule 8 says to slow down to avoid splashing them."

In a nutshell:
Current self-driving cars are like parrots that repeat what they've seen. KnowVal is like a thoughtful human who sees the world, checks the rulebook, thinks about what is right, and then makes a safe decision. It combines the eyes of a machine with the wisdom of a human.

Technical Summary

1. Problem Statement

Current autonomous driving (AD) systems face three critical limitations when operating in open, dynamic, and uncertain environments:

• Lack of Visual-Language Reasoning: Existing End-to-End (E2E) models lack language-grounded reasoning capabilities, while Vision-Language-Action (VLA) models often restrict reasoning to linguistic chains of thought without allowing reasoning outcomes to influence perception.
• Absence of Structured Knowledge: Most systems rely on data-driven imitation learning, struggling to infer complex decision logic (e.g., traffic laws, defensive driving, ethics) from limited human behavior data. Handcrafted rules are too narrow to generalize.
• Missing Value Alignment: Current paradigms often lack a dedicated mechanism to evaluate whether predicted future states are "desirable" or aligned with human social values and safety norms, relying instead on simple reward functions or data fitting.

2. Methodology: The KnowVal Framework

KnowVal proposes a novel architecture that synergistically integrates open-world perception, knowledge retrieval, and value-guided planning. The system operates through three interconnected modules:

A. Reasoning between Perception and Retrieval

The core innovation is a bidirectional guidance mechanism:

1. Retrieval-Guided Open-World Perception:

   • Specialized Perception: Recognizes standard objects (vehicles, pedestrians).
   • Open-ended 3D Perception: Identifies long-tail/uncommon objects (e.g., fire trucks, standing water) without explicit prompts using VL-SAMv2.
   • Abstract Concept Understanding: Captures contextual attributes (e.g., "night scene," "bridge," "rainy") via VLMs.
   • Feedback Loop: If the retrieval module identifies missing information, it prompts the perception module to refine observations in the next timestep.

2. Perception-Guided Knowledge Retrieval:

   • Knowledge Graph Construction: A comprehensive graph is pre-constructed from traffic laws, defensive driving principles, ethical guidelines, and driver interviews.
     • Structure: Organized as a "Knowledge Forest" (hierarchical text) converted into a graph via LLMs.
     • Fidelity: Crucially, the system stores raw, unaltered text clauses (native nodes) to prevent LLM hallucinations or summarization errors.
   • Retrieval Process:
     • A Perception Verbalizer converts 3D perception outputs (bounding boxes, occupancy maps) into structured natural language queries.
     • An LLM-based retriever extracts entities and ranks knowledge entries by relevance.
     • Filtering: Only "native" nodes (original text) are retrieved to ensure factual accuracy.

B. Planning with World Prediction and Value Model

The planning module generates trajectories and evaluates them against retrieved knowledge:

1. World Model (Future State Prediction): The planner (based on architectures like HENet++ or DiffusionDrive) is extended to generate diverse candidate trajectories ($T_i$) and predict corresponding future world states ($S_i$).
2. Value Model (Trajectory Assessment):
   • A dedicated Value Model (Transformer Encoder + MLP Decoder) evaluates each candidate trajectory against the retrieved knowledge entries ($K_j$).
   • It outputs a scalar score $s_{i,j} \in [-1, 1]$ indicating compliance (1 = positive, -1 = violation, 0 = irrelevant).
   • Scoring Strategy: A weighted decay strategy aggregates scores based on relevance, prioritizing the most critical rules (e.g., "yield to pedestrians") to produce a final trajectory score.
3. Decision Making: The trajectory with the highest aggregate value score is selected as the final output.

C. Training Data

• Preference Dataset: A dataset of 160K trajectory-knowledge pairs was curated.
• Annotation: Using Qwen-VL-Max and manual review, each sample is annotated with compliance scores based on specific knowledge clauses, training the Value Model to align with human preferences.

3. Key Contributions

1. Visual-Language Reasoning System: Introduced KnowVal, which enables mutual guidance between perception and knowledge retrieval, allowing the system to "ask" for more visual data based on knowledge gaps.
2. Comprehensive Driving Knowledge Graph: Constructed a structured graph encompassing laws, morals, and defensive driving principles, utilizing an LLM-based retrieval mechanism that preserves original text fidelity to avoid hallucinations.
3. Value-Aligned Planning: Designed a planner integrating a World Model and a Value Model trained on a human-preference dataset, enabling interpretable, value-driven decision-making.
4. Compatibility: Demonstrated that the framework can be seamlessly integrated into existing E2E and VLA architectures without requiring a complete redesign.

4. Experimental Results

KnowVal was evaluated on nuScenes, Bench2Drive, and NVISIM:

• nuScenes (Open-Loop): Achieved the lowest collision rate among all compared methods (including UniAD, VAD, DiffusionDrive). While L2 error was slightly higher (indicating deviation from human trajectories), this reflects the discovery of safer, non-human strategies.
• Bench2Drive (Closed-Loop): Achieved State-of-the-Art (SOTA) performance.
  • Driving Score: 88.42 (+3.35 improvement over SimLingo).
  • Success Rate: 69.03% (+1.76% improvement).
• NVISIM: Showed significant improvements in PDM scores (e.g., +2.8 over DiffusionDrive) when integrated with advanced planners.
• Qualitative Analysis: The system successfully handled complex ethical scenarios (e.g., slowing down for puddles to avoid splashing pedestrians) and legal constraints (e.g., no overtaking in tunnels) where baseline models failed.

5. Significance

KnowVal represents a paradigm shift from purely data-driven imitation learning to knowledge-augmented, value-aligned autonomous driving.

• Safety & Ethics: It explicitly incorporates traffic laws and moral principles into the decision-making loop, addressing the "black box" nature of current E2E models.
• Interpretability: By retrieving specific knowledge clauses to justify decisions, the system offers explainable reasoning for its actions.
• Generalization: The ability to retrieve and apply abstract knowledge allows the system to handle long-tail scenarios and dynamic environments that are underrepresented in training data.
• Practicality: The modular design allows it to enhance existing state-of-the-art planners, making it a viable upgrade path for current autonomous driving stacks.