Vision and Language: Novel Representations and Artificial intelligence for Driving Scene Safety Assessment and Autonomous Vehicle Planning

This paper investigates the integration of vision-language models into autonomous driving systems through three complementary use cases—hazard screening, trajectory planning, and behavioral constraint enforcement—demonstrating that while these representations offer significant promise for safety, their effective deployment requires careful system design and task-aligned grounding rather than direct feature injection.

The Gist

Imagine you are teaching a robot to drive a car. For a long time, we taught the robot by showing it millions of pictures of "cars," "pedestrians," and "stop signs." The robot became very good at spotting these specific things. But what happens when the robot sees something it has never seen before? Maybe a deer jumping out, a pile of strange debris, or a construction zone with weird, hand-drawn signs? Or what if a passenger says, "Stop by that guy in the red hat"?

This paper asks: Can we teach the robot to understand the story of the road, not just the objects, by using language?

The authors tried three different ways to mix "Vision" (what the car sees) with "Language" (what humans say or write) to make self-driving cars safer. Here is what they found, explained simply:

1. The "Smoke Detector" Approach (Hazard Screening)

The Idea: Instead of trying to identify every single object, they asked the AI: "Is there a hazard on the road?" using a model called CLIP (which is like a super-smart librarian that knows how to match pictures with words).

The Experiment: They showed the AI thousands of videos of driving. They asked it to look for specific things like "animals," "falling objects," or "low visibility" (like fog).

• The Analogy: Think of this like a smoke detector. It doesn't need to know exactly what is burning (a toast, a candle, or a fire); it just needs to know, "Hey, something is smoky here, be careful!"
• The Result: It worked surprisingly well for big, obvious problems like fog or animals. However, it struggled with small, tricky things like a tiny piece of trash on the road or flashing emergency lights (because the AI looks at one picture at a time and misses the "flashing" part).
• The Lesson: Language is great for a "first alert" system to say, "Something is weird here, slow down!" but it's not perfect enough to be the only thing the car relies on.

2. The "Confused Navigator" Approach (Trajectory Planning)

The Idea: They tried to feed the car's "brain" (the part that decides where to steer) a giant summary of the scene in language format. They thought, "If the car knows the meaning of the scene, it will drive better."

• The Analogy: Imagine you are driving a car, and a GPS voice suddenly shouts, "This is a busy city with lots of danger and people!" It gives you the vibe of the place, but it doesn't tell you where the pothole is or how far the car in front is.
• The Result: This actually made the car drive worse. The car got confused. The "big picture" language was too vague. The car needed specific coordinates (geometry) to know where to put its wheels, not a poetic description of the scene.
• The Lesson: You can't just dump a "summary" of the world into a steering wheel. The car needs precise, local details to drive safely. General "vibes" don't help with turning the wheel.

3. The "Passenger's Voice" Approach (Human Instructions)

The Idea: They tested what happens if a human passenger gives a specific instruction, like, "Stop by the person in the blue shirt," or "Wait for the dog to cross."

• The Analogy: This is like having a human co-pilot. If the robot sees a crosswalk but isn't sure if it should stop, and the passenger says, "Wait for that kid," the robot listens.
• The Result: This was the biggest success. When the robot got specific instructions based on what it could see, it stopped making dangerous mistakes. It prevented the car from driving into crosswalks or ignoring pedestrians.
• The Catch: The instructions had to be clear. If the passenger said something vague like "Go over there," the robot got confused. Also, if the robot listens too much, it might become too scared to move (like a driver who stops at every shadow).
• The Lesson: Language is most powerful when it acts as a safety constraint. It helps the robot make the right choice in tricky, ambiguous situations where the rules aren't clear.

The Big Takeaway

The paper concludes that Vision-Language models are a powerful tool, but you can't just plug them in and hope for the best.

• Don't use them to replace the car's eyes (geometry) with vague summaries.
• Do use them as a "safety net" to spot weird hazards.
• Do use them as a way for humans to give specific, safety-focused instructions.

In short: Teaching a self-driving car to "speak" the language of safety is promising, but it requires careful engineering. It's not about making the car smarter in a general sense; it's about giving it the right kind of help at the right time to keep everyone safe.

Technical Summary

1. Problem Statement

Autonomous driving systems (ADS) excel in structured environments but struggle with open-world safety challenges involving ambiguity, novelty, and human intent. Traditional pipelines rely on closed-world assumptions (predefined object classes, geometric kinematics) and often fail in edge cases where semantic context is critical (e.g., a pedestrian hesitating, improvised construction signage, or a passenger instruction).

While Vision-Language Models (VLMs) like CLIP offer powerful semantic reasoning capabilities, there is a lack of empirical understanding regarding how to integrate these representations into safety-critical planning pipelines. Specifically, the paper addresses:

• Can VLMs detect out-of-distribution (OOD) hazards without explicit object detection?
• Do global semantic embeddings improve or degrade trajectory planning when injected directly?
• Can natural language serve as an effective, human-authored constraint to suppress catastrophic planning failures?

2. Methodology

The authors investigate three complementary system-level use cases spanning the perception-to-planning continuum:

A. Open-Vocabulary Hazard Screening (Perception)

• Approach: A lightweight, category-agnostic hazard detection system using CLIP (Contrastive Language-Image Pre-training).
• Mechanism: Instead of detecting specific objects, the system computes the image-text similarity score between a driving frame and prompts like "hazard on the road" or specific categories (e.g., "animal," "construction").
• Signal Construction: To reduce false positives, the system calculates the margin between the "hazard" prompt score and a negative prompt (e.g., "normal driving scene").
• Evaluation: Tested on the COOOL benchmark (Challenge Of Out-Of-Label), which contains rare, ambiguous hazards. Metrics include temporal Intersection-over-Union (tIoU) for both hazard presence (positive) and absence (negative).

B. Global Representation Learning for Trajectory Planning

• Approach: Integrating scene-level vision-language embeddings into a Motion Transformer (MTR) based trajectory planner.
• Dataset: Waymo End-to-End Driving Dataset.
• Mechanism: The baseline MTR-VP model conditions on camera images and kinematic history. The authors augmented the planning query with global embeddings from CLIP and DINOv2.
• Goal: To test if high-level semantic priors improve the planner's ability to generate safe trajectories in complex scenarios (construction, intersections, pedestrians).

C. Natural Language as Behavioral Constraints (Planning)

• Approach: Using natural language instructions as explicit constraints on motion planning.
• Dataset: doScenes (an extension of nuScenes with passenger-style instructions) and OpenEMMA (a multimodal planning framework).
• Mechanism: Passenger instructions (e.g., "stop by the person") are injected into the model prompt. The model is instructed to follow the request unless it is unsafe, in which case it must choose a safe alternative and justify the override.
• Goal: To evaluate if language can suppress rare but severe planning failures (e.g., driving into a crosswalk) in ambiguous scenarios.

3. Key Contributions

1. Lightweight Hazard Screening: Demonstrated that CLIP-based image-text similarity can serve as a low-latency, category-agnostic "screening layer" for OOD hazards, generalizing to unseen objects without explicit detection.
2. Negative Result on Global Embeddings: Provided empirical evidence that naïvely injecting global vision-language embeddings directly into trajectory planners degrades performance. This highlights a misalignment between global semantic representations and the spatially grounded requirements of motion planning.
3. Language as a Safety Constraint: Showed that human-authored natural language instructions, when grounded in visual scene elements, effectively suppress catastrophic planning failures (out-of-bounds trajectories) and improve safety in ambiguous situations, even if they do not uniformly improve geometric accuracy.
4. Engineering Insight: Argued that realizing the safety potential of VLMs is an engineering problem requiring careful system design (grounding, task alignment, and trust boundaries) rather than simple feature injection.

4. Key Results

Hazard Screening (COOOL Benchmark)

• Performance: The "Low Visibility" and "Animal" categories achieved the highest Global tIoU (0.765 and 0.657, respectively).
• Fusion Strategy: A Dual-Hazard approach (requiring agreement between a general "hazard" prompt and at least one specific category prompt) significantly reduced false alarms (Video-TNR improved from 34% to 70%) while maintaining high detection rates.
• Limitations: Performance dropped for small, localized hazards (e.g., road debris) and temporal hazards (e.g., flashing emergency lights) due to CLIP's lack of temporal reasoning and spatial downsampling.

Trajectory Planning (Waymo Dataset)

• Performance Degradation: Adding CLIP and DINOv2 embeddings to the MTR planner increased Average Displacement Error (ADE) and lowered Rater Feedback Scores (RFS) across all categories (Construction, Intersection, Pedestrian, etc.).
  • Example: 3-second ADE increased from 1.42 (Base) to 2.01 (Augmented).
• Conclusion: Global semantic cues introduced noise and ambiguity into the planning module, confirming that semantics must be structured and grounded before influencing low-level control.

Instruction-Conditioned Planning (doScenes/OpenEMMA)

• Failure Suppression: The vision-only baseline produced extreme outliers (e.g., driving off-road). Language conditioning suppressed these catastrophic failures.
• Quantitative Impact:
  • Mean ADE dropped drastically from 6201.4 (vision-only, dominated by outliers) to 9.99 (with instructions).
  • After filtering outliers (Q97.5), the best-performing instructions still reduced mean ADE by 5.1%.
• Qualitative Impact: Instructions referencing dynamic elements (e.g., "stop for the pedestrian") successfully corrected unsafe behaviors in ambiguous scenarios where the baseline planner failed.

5. Significance and Future Directions

The paper fundamentally shifts the perspective on VLMs in autonomous driving:

• Not a "Drop-in" Solution: Simply adding VLM embeddings to existing planners does not work; it often harms safety-critical performance due to a mismatch between abstract semantics and geometric precision.
• Role of Language: VLMs are most effective when used to express semantic risk, intent, and constraints (e.g., hazard screening, passenger instructions) rather than as raw features for trajectory generation.
• Engineering Challenge: Future work must focus on structured grounding (bridging semantics to spatial constraints), temporal stability (handling video sequences rather than single frames), and safety arbitration (defining trust boundaries where language can override or be overridden by safety rules).

This research underscores that for autonomous vehicles to operate safely in open-world environments, they require representations that bridge human-interpretable semantics with machine-executable, safety-guaranteed planning.