💻 computer science

Vision-Language Assistant for Emotional Reactions to Risky Driving

This paper introduces "Keep Yelling Assistant" (KYA), a vision-language pipeline that combines YOLOv8-based risky driving detection with large language models to generate real-time, emotionally adaptive verbal responses tailored to driver preferences, thereby enhancing safety and emotional well-being in vehicles.

Original authors: Harine Choi, Eun Hak Lee, Zhengzhong Tu

Published 2026-07-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Harine Choi, Eun Hak Lee, Zhengzhong Tu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are driving down a busy street, and suddenly, a car zooms in front of you without signaling. Your heart skips a beat, your grip tightens on the wheel, and you might even shout, "Hey, watch out!" That split-second explosion of emotion is a very human reaction to danger. For a long time, the computers inside our cars have been like stoic robots: they see the danger, they calculate the math, and they beep a warning, but they never feel the stress or the annoyance. This paper lives in the world of "Vision-Language Models," which are basically super-smart computer brains that can look at a picture (vision) and then talk about it using human language (language). Think of them as a camera that can also write a diary entry about what it sees. The big question researchers are asking is: Can we teach these car computers to not just see a risky driver, but to react with the same emotional flair a human would? It turns out, if we can do this, it might make driving less stressful and safer, because a car that understands your mood could help calm you down or warn you in a way that actually feels real.

Enter KYA, which stands for the Keep Yelling Assistant. You can think of KYA as a digital co-pilot with a personality. Instead of just saying "Warning: Car ahead," KYA is designed to look at the road, spot a dangerous move like a sudden cut-in, and then shout out a response that matches your mood. If you like to be funny, KYA might crack a joke about the other driver. If you want to be serious, it might give you a calm, analytical breakdown of the situation. The researchers built this system by connecting two main parts: a pair of "eyes" and a "brain." The eyes are a computer vision tool called YOLOv8 (which is like a super-fast security guard that spots cars and tracks their speed). The brain is a Large Language Model (LLM), which is the same kind of technology that powers smart chatbots. The vision part measures how close the other car is and how fast it's coming, then passes that data to the brain, which turns those numbers into a spoken sentence with the right emotional tone.

To see if this idea works, the team tested it with real dashcam videos of risky driving and asked 108 people to play along. They showed the participants different "personalities" for the assistant, ranging from "Neutral" and "Analytical" to "Humorous," "Angry," and even "Magnanimous" (which means being super forgiving). They then asked the participants to pick which computer-generated response felt the most right for the situation. The results were pretty clear: people generally liked the idea of an emotional assistant. When it came to the specific personalities, most people preferred the Humorous and Analytical styles, while the ones that were just plain angry were less popular. It seems we want our car to be witty or smart, not just a screaming match.

When it came to the "brain" behind the voice, the researchers tested four different AI models: ChatGPT-4o, Claude 3, Gemini 2.5, and Copilot. They found that ChatGPT-4o was the crowd favorite, scoring the highest overall (4.29 out of 5.00). It was particularly good at being funny and analytical. However, the study also found that different people liked different voices; for example, female participants tended to prefer the Claude 3 model for its emotional nuance, while male participants leaned toward Copilot. Interestingly, the study suggests that a robotic voice didn't work well—people wanted a human-like voice, with a slight preference for female voices, perhaps because they felt warmer or more reassuring in a scary moment.

The paper doesn't claim to have solved all driving problems or to have a perfect system ready for your car tomorrow. In fact, the researchers are careful to note that they only tested this using pre-recorded videos and a survey, not by actually driving cars on the road in real-time. They also admit that their "eyes" (the vision system) work best for specific types of risks, like cars cutting in, and might need tweaking for other dangers like tailgating. But the main takeaway is a hopeful one: by combining a camera that sees the road with a brain that understands human feelings, we might be able to build a car assistant that doesn't just keep us safe, but also keeps us sane. The study suggests that when we feel understood by our machines, even in a stressful traffic jam, the whole experience of driving could become a little less terrifying and a lot more human.

Technical Summary: Vision-Language Assistant for Emotional Reactions to Risky Driving

Problem Statement
Modern urban driving environments are increasingly characterized by risky behaviors, such as sudden lane changes without signaling and tailgating, which statistically correlate with a high incidence of collisions. While these behaviors trigger intense emotional reactions in human drivers (ranging from irritation to panic), existing Advanced Driver Assistance Systems (ADAS) and Vision-Language Models (VLMs) largely remain limited to neutral, task-driven outputs. Current systems often fail to address the emotional dimension of human driving experiences, potentially leading to user disengagement or mistrust when hazards are detected but not communicated with appropriate urgency or affective context. There is a gap in integrating real-time perception of risky maneuvers with emotionally adaptive, persona-aligned dialogue to support driver awareness and comfort.

Methodology
The study introduces Keep Yelling Assistant (KYA), a modular vision-language framework designed to detect risky driving behaviors and generate emotionally expressive verbal responses. The pipeline operates in two primary stages:

Vision Module (Perception and Behavior Recognition):
- Input: Raw dashcam footage from real-world scenarios, specifically focusing on sudden cut-in events.
- Detection: The system employs YOLOv8 variants (nano, small, medium, large) pretrained on datasets like BDD100K and nuScenes. The models detect nearby vehicles, extracting attributes such as image-plane coordinates, estimated speed, distance, vehicle type, and color.
- Normalization: To address camera angle variability, the system applies a normalization procedure using a scale anchor to preserve relative proximity relationships, prioritizing scale-invariant cues over absolute depth accuracy.
- Risk Quantification: The system identifies the "aggressor" vehicle (the one posing the highest risk) by analyzing spatiotemporal patterns. It calculates a specific risk indicator called Projected Reach Time (PRT), defined as $PRT = d_{rel} / (v_{ego} + \epsilon)$ , where $d_{rel}$ is the relative distance and $v_{ego}$ is the ego vehicle's speed. PRT estimates the time required for the ego vehicle to reach the aggressor's current position, serving as a conservative measure of temporal proximity suitable for single-camera environments.
- Output: A structured behavior log containing relative distance, relative speed, PRT, and contextual metadata.
Language Module (Reasoning and Response Generation):
- Input: The structured behavior log from the vision module and user-defined emotional preference settings (personas).
- Personas: Users select from seven predefined emotional tones: Neutral, Light Anger, Noticeable Anger, Intense Anger, Analytical, Humorous, and Magnanimous.
- Models: The system utilizes four state-of-the-art Large Language Models (LLMs): ChatGPT-4o, Claude 3, Gemini 2.5, and Copilot.
- Process: The LLM interprets the traffic scenario based on the behavior log and generates a natural language response aligned with the selected persona. The output is then converted to speech via a Text-to-Speech (TTS) engine, adapting vocal characteristics (e.g., tone, inflection) to match the chosen emotional style.

Data and Evaluation

Vision Evaluation: The system was tested on 20 curated dashcam video sequences containing 5,019 frames, with 1,189 frames identified as high-risk cut-in interactions.
User Study: A survey involving 108 participants (53 male, 55 female) was conducted to evaluate emotional alignment. Participants rated model outputs based on fluency, emotional alignment, and persona consistency using a 5-point Likert scale. They also performed blind selections of the best response for specific personas.

Key Results

Vision Performance: Among the YOLOv8 variants, YOLOv8l achieved the highest overall performance (Recall: 0.869, F1: 0.921, mAP@0.5: 0.998). However, YOLOv8s demonstrated a favorable balance between computational efficiency and accuracy (Recall: 0.762, Precision: 0.982), making it suitable for real-time deployment in emotionally responsive applications.
Behavioral Metrics: The analysis of risky interactions showed an average relative distance of 5.43 meters and a mean PRT of 1.80 seconds. Over 75% of threats occurred within a 3-second threshold, often violating the standard "three-second rule" for safe following distance.
Language Model Performance:
- ChatGPT-4o consistently outperformed other models, achieving the highest overall score of 4.29/5.00 in the few-shot setting. It was particularly preferred for Analytical (41.6% selection) and Humorous (42.2% selection) personas.
- Claude 3 showed competitive performance in emotionally charged styles, particularly Light Anger and Intense Anger.
- User Preferences: Participants favored Humorous (48.1%) and Analytical (42.3%) personas over intense anger. There was a strong preference for female voices (53.8%) over male or robotic voices.
- Statistical Significance: A chi-square test indicated statistically significant differences in model selection across personas ( $\chi^2 = 12.20, p = 0.0067$ ).

Significance and Contributions
The paper claims three primary contributions:

Framework Design: The proposal of an integrated perception–behavior–language architecture that bridges low-level visual perception with high-level affective interaction. Unlike existing VLMs focused on situational reasoning or task guidance, KYA is designed to produce emotionally expressive responses grounded in real-time driving scenarios.
Practical Vision Module: The development of a practical vision module using lightweight YOLOv8 models to detect cut-in behaviors from unconstrained dashcam footage. The authors introduce a reproducible method using PRT to quantify risk without relying on complex trajectory prediction or precise sensor calibration.
Personalized Language Generation: The implementation of a language model that adapts responses based on explicit user preferences. This study is noted as the first to directly integrate user-defined emotional personas into the response-generation process of an LLM-based assistant, enabling granular personalization.

Limitations and Future Work
The authors maintain a modest stance regarding the current scope. The evaluation was conducted offline using pre-recorded footage and controlled user studies; real-time in-vehicle validation has not yet been achieved. The participant sample was limited in demographic scope. While the framework is behavior-agnostic in structure, the current empirical evaluation focused exclusively on sudden cut-in events. Future work is expected to focus on real-world deployment, the incorporation of additional risky behaviors (e.g., tailgating, sudden braking), and broader user validation to assess longitudinal safety and usability impacts.

Technical Summary: Vision-Language Assistant for Emotional Reactions to Risky Driving

More like this