💬 NLP

DriveSafe: A Framework for Risk Detection and Safety Suggestions in Driving Scenarios

DriveSafe is a novel framework that enhances autonomous vehicle risk assessment by generating spatially grounded, multimodal scene captions to fine-tune a lightweight adapter, thereby achieving state-of-the-art performance in identifying hazards and providing safety suggestions on the DRAMA benchmark.

Original authors: Sainithin Artham, Shankar Gangisetty, Avijit Dasgupta, C. V. Jawahar

Published 2026-05-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sainithin Artham, Shankar Gangisetty, Avijit Dasgupta, C. V. Jawahar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a very smart, well-read robot how to drive a car. You want this robot to not only see the road but also understand why a situation might be dangerous and tell you exactly what to do about it.

This paper introduces a new system called DriveSafe. Think of it as a "safety coach" for self-driving cars that bridges the gap between just "seeing" the road and actually "understanding" the risks.

Here is how it works, broken down into simple concepts:

1. The Problem: The "Bookworm" vs. The "Street Smart" Driver

The researchers found that current "Multimodal Large Language Models" (MLLMs)—which are like super-smart robots that can see images and read text—are great at general tasks. They are like bookworms: they can describe a picture of a street perfectly ("There is a red car and a blue sky").

However, when it comes to driving, these bookworms struggle. They often miss subtle dangers. For example, they might see a truck but fail to notice it is slowing down in a way that blocks the road. They are like a passenger who can describe the scenery but doesn't know how to drive or spot a hazard until it's too late. They also fail to give specific advice like, "Slow down now," instead just saying, "The car is there."

2. The Solution: DriveSafe's "Three-Lens Goggles"

To fix this, the authors created DriveSafe. Instead of just feeding the robot a video, they give it a special set of three lenses to look through before it tries to speak:

The Motion Lens: This tracks how things are moving (like optical flow). It sees speed and direction.
The Depth Lens: This measures how far away things are. It knows if a pedestrian is 10 feet away or 100 feet away.
The Space Lens: This maps the road lanes and boundaries. It knows where the "drivable" area is.

By combining these three lenses with a general description of the scene, the system creates a super-detailed story (a caption) about what is happening. It's like giving the robot a map, a speedometer, and a distance gauge all at once, rather than just a photo.

3. The Two-Step Process

DriveSafe works in two main stages:

Step A: Writing the Story
First, the system takes the video and the "Three-Lens" data to write a very specific, grounded story.

Bad Example (Old Models): "A car is on the road."
DriveSafe Example: "A yellow truck is slowing down in the ego lane, located 20 meters ahead, blocking the path."

Step B: The Safety Coach
This detailed story is then passed to a Large Language Model (the "brain"). Because the story is so precise, the brain can now act like a driving instructor:

Detect Risk: "Yes, this is dangerous."
Pinpoint the Danger: "The yellow truck is the problem."
Give Advice: "Slow down."

4. Training the Coach (Fine-Tuning)

The researchers realized that even with the "Three-Lens" story, the brain still needed practice. So, they didn't just ask the robot to guess; they taught it using a lightweight adapter.

Imagine this as a specialized training manual. They showed the robot thousands of examples where a specific type of danger (like a "slowing truck") always leads to a specific action ("Slow down"). This training helped the robot stop guessing and start giving reliable, actionable advice.

5. The Results: From "Maybe" to "Definitely"

The team tested DriveSafe on a dataset called DRAMA (a collection of driving videos with safety labels).

General AI Models: When asked to spot risks and give advice, they were often wrong or vague (getting about 13–19% accuracy). They were like a tourist trying to drive in a foreign country without a map.
DriveSafe (Zero-Shot): Even without the special training manual, just using the "Three-Lens" stories, DriveSafe did better than the general models (about 23% accuracy).
DriveSafe (Trained): After using the training manual, DriveSafe skyrocketed to 52.85% accuracy. It became significantly better at spotting the exact danger and giving the right advice, far outperforming all other methods tested.

The Bottom Line

The paper claims that DriveSafe is a new framework that makes self-driving cars safer by:

Using extra data (motion, depth, space) to write better descriptions of the road.
Using those descriptions to train the AI to spot risks and give specific safety advice (like "Stop" or "Slow down").
Proving that this method works much better than just using general AI models that haven't been specifically trained for driving risks.

In short, DriveSafe turns a robot that can describe a traffic jam into a robot that can navigate it safely.

Technical Summary: DriveSafe

Problem Statement

Comprehensive situational awareness is critical for autonomous vehicles to identify and mitigate risks in safety-critical environments. While Multimodal Large Language Models (MLLMs) have demonstrated success in general vision-language tasks, the authors find that zero-shot MLLMs underperform in fine-grained, spatially grounded risk assessment for driving. Specifically, existing general-purpose models tend to generate generic, descriptive captions that fail to identify potential hazards or provide actionable safety guidance. Furthermore, prior domain-specific approaches often focus on risk prediction or trajectory forecasting but neglect the generation of explicit, human-understandable safety suggestions necessary for practical deployment. There is also a lack of datasets that explicitly associate critical objects with risk-related behavioral keywords and safety suggestions.

Methodology

The authors propose DriveSafe, a framework designed for risk-aware scene understanding that leverages structured natural language descriptions rather than directly fine-tuning a massive MLLM on raw video. The framework operates in two primary stages:

1. Caption Generation with Multimodal Context

Instead of relying solely on visual encoders, DriveSafe enriches scene understanding by integrating specific contextual cues:

Spatial Context ( $S_t$ ): Derived from road and lane segmentation masks to delineate drivable areas.
Motion Context ( $M_t$ ): Extracted via optical flow to highlight relative movements of agents (vehicles, cyclists, pedestrians).
Depth Context ( $D_t$ ): Computed at the object level to estimate spatial proximity to the ego-vehicle.
Video Description ( $d_v$ ): A high-level semantic summary generated by an MLLM.

These modalities are combined into a unified input ( $X_v$ ) and fed into a Large Language Model (LLM) via a structured prompt. This process generates a geometry-aware caption ( $C_v$ ) that explicitly includes ego-vehicle maneuvers, surrounding agent behaviors, lane positions, and bounding boxes with depth/motion cues.

2. Risk Assessment and Safety Suggestion

The generated caption ( $C_v$ ) is processed by an LLM to infer:

A binary risk label ( $\hat{r}$ : Yes/No).
A refined risk caption ( $C_r$ ) explaining the hazard.
Risk-related keywords ( $\hat{K}$ ) capturing the behavior.
Bounding boxes ( $\hat{b}$ ) for the risky object.

Safety Suggestion Derivation: The extracted keywords are mapped to nine predefined safety suggestion categories (e.g., "Slow down," "Stop," "Yield") using a rule-based mapping function ( $g(\cdot)$ ) derived from the DRAMA dataset.

Training Strategy:

Zero-Shot: The framework can operate without fine-tuning by prompting the LLM directly.
Adapter Fine-Tuning: To improve performance, the authors employ a lightweight adapter module (LLaMA-Adapter) trained on caption-instruction-response triplets. This aligns the LLM with domain-specific risk reasoning without full-scale video pre-training. The training objective minimizes the negative log-likelihood of the structured response (risk label, caption, keywords, and bounding box).

Key Contributions

DriveSafe Framework: A novel pipeline that avoids direct MLLM fine-tuning on video. Instead, it integrates multimodal cues (optical flow, segmentation, depth) with video descriptions to produce driving-specific captions, which are then processed by an LLM for risk prediction and safety suggestions.
Dataset Extension: The authors extend the DRAMA dataset by explicitly associating risky behaviors with critical objects and introducing safety-suggestion annotations. They utilize an LLM for automatic annotation of the training set while reserving manual annotations for the test set to ensure reliability.
Actionable Safety Guidance: Unlike prior works focusing solely on detection, DriveSafe explicitly generates grounded safety suggestions linked to detected hazards.

Experimental Results

Experiments were conducted on the DRAMA benchmark, comparing DriveSafe against existing risk assessment models (LCP, VTS, HoP) and general-purpose MLLMs (Qwen2.5-VL, LLaVA-NeXT, VideoLLaMA3).

Caption Generation: DriveSafe-Finetuned significantly outperformed both zero-shot MLLMs and specialized baselines. For instance, it achieved a BLEU-4 score of 60.38 compared to ~56 for HoP and <30 for general VLMs.
Risky Object Grounding: The fine-tuned model achieved an Acc@0.5 of 74.8 and a Mean IoU of 59.8, far surpassing general VLMs which struggled with localization (e.g., Qwen2.5-VL had 0.0 Acc@0.5). The authors note that IoU alone can be misleading for zero-shot models, as high IoU scores can result from rare correct matches while missing most risks; DriveSafe maintains high accuracy and IoU simultaneously.
Safety Suggestion Prediction: DriveSafe-Finetuned achieved 52.85% accuracy and a 37.15 weighted F1 score, more than doubling the performance of its zero-shot counterpart (23.49% accuracy) and substantially outperforming all baselines (which ranged from 13–19% accuracy).
Qualitative Analysis: Case studies show that DriveSafe-Finetuned reduces hallucinations and exaggerations common in zero-shot models, providing grounded interpretations (e.g., correctly identifying a stopped vehicle vs. a moving one) and context-appropriate suggestions.

Significance and Claims

The paper claims that DriveSafe addresses a critical gap in autonomous driving by bridging risk understanding with actionable safety guidance. By conditioning risk assessment on explicit, language-based scene representations enriched with geometric and motion cues, the framework achieves state-of-the-art performance in risk detection and safety suggestion generation.

The authors emphasize that their approach offers transparency by explicitly linking identified risks to specific safety recommendations, which is essential for building trust in autonomous systems. They position this work as a foundational step, noting that while the current risk-to-safety mapping is static, it establishes a novel framework for safety-critical reasoning. The paper concludes that domain-specific adaptation (via lightweight adapters and multimodal context) is necessary to overcome the limitations of general-purpose VLMs in safety-critical driving scenarios.