Technical Summary: Responsibility Distribution Estimation in Ego-View Accident Videos with Multimodal Large Language Models

Problem Definition

Current multimodal traffic accident analysis relies heavily on infrastructure-camera footage, satellite imagery, or structured crash records. These data sources present two critical limitations: they are costly to deploy at scale, and more importantly, they fail to objectively capture the visual perspective available to the driver prior to an accident. Consequently, they are ill-suited for reasoning about accident avoidability and driver responsibility from the involved vehicle's viewpoint.

This paper introduces Responsibility Distribution Estimation as a new task for ego-view traffic accident videos. Unlike conventional tasks that predict a single accident label, severity, or causal explanation, this task requires a model to predict a proportional responsibility distribution (percentages summing to 100%) among all involved agents (e.g., ego vehicle, other vehicles, pedestrians). The model must reason about what information was available to the ego driver, identify the primary responsible party, and allocate liability percentages that reflect shared responsibility, mirroring real-world traffic reasoning.

Methodology

Dataset Construction

The authors constructed a dataset based on MM-AU, a collection of ego-view dashcam accident videos with existing text annotations (scene description, causes, preventive measures). Since MM-AU lacks responsibility labels, the authors developed an LLM-assisted annotation pipeline:

• Input: For each video, 8 uniformly sampled frames spanning the abnormal event window (ensuring the accident frame is included) are presented to an LLM alongside scene metadata and text annotations.
• Output Generation: The LLM is prompted to output integer percentages for the ego vehicle and other parties (summing to 100%), identify the primary responsible party, categorize the other party, and provide a rationale grounded in visual/textual evidence.
• Constraints: Temperature is set to zero to maximize consistency. The resulting dataset is roughly balanced between ego-primary (52%) and other-party-primary (48%) cases.

Model and Training

The study fine-tunes Qwen3-VL-4B, a multimodal large language model, using LoRA (Low-Rank Adaptation) on all linear projection layers (rank $r=16$, scaling factor $\alpha=32$).

• Input: The model receives $k=10$ sampled frames, scene metadata, and text annotations.
• Output: The model autoregressively generates a JSON-formatted output containing the predicted responsibility distribution, primary party, other-party label, and reasoning.
• Constraint Learning: The requirement that percentages sum to 100 is enforced implicitly through the prompt instructions and the training data distribution, rather than explicit post-processing constraints.

Experimental Setup

The authors evaluate the fine-tuned model against a GPT-5 zero-shot baseline under three input conditions:

1. RGB: Raw frames + text annotations.
2. RGB+Seg: Semantic segmentation overlays + text annotations.
3. Text: Text annotations only (no images).

Evaluation Metrics:

• Exact Match (EM): Percentage of cases where the predicted ego-vehicle responsibility matches the ground truth exactly.
• Within-10%/20% (W-10/W-20): Proportion of cases with absolute error $\le$ 10 or 20 percentage points.
• MAE/RMSE: Mean Absolute Error and Root Mean Squared Error on the ego-vehicle percentage.
• Primary Party Accuracy (PP): Accuracy in identifying the primary responsible agent.

Key Results

The fine-tuned Qwen3-VL-4B demonstrates significant improvements over the zero-shot GPT-5 baseline, particularly under RGB input:

• Exact Match: The fine-tuned model achieves 79.1%, compared to 38.5% for GPT-5 zero-shot.
• Primary Party Accuracy: The fine-tuned model reaches 96.7%, outperforming GPT-5's 89.0%.
• Error Reduction: The Mean Absolute Error (MAE) for the ego-vehicle percentage drops from 12.1 (GPT-5) to 4.9 (fine-tuned), a reduction of more than half.

Ablation and Input Analysis:

• Visual Necessity: Removing visual input (Text-only condition) causes a sharp performance degradation for both models. GPT-5's MAE more than doubles (12.1 $\to$ 24.9), and the fine-tuned model's MAE increases drastically (4.9 $\to$ 35.1). This confirms that ego-view frames contain critical responsibility information not recoverable from text alone.
• Segmentation Impact: Replacing raw RGB frames with semantic segmentation overlays (RGB+Seg) yields comparable or slightly worse results, suggesting that raw visual appearance provides sufficient or superior information for this task compared to structured segmentation overlays in this setting.
• Fine-tuning vs. Zero-shot: While GPT-5 zero-shot shows reasonable "Within-10%" performance (79.1%), indicating it can roughly reason about the direction of responsibility, the fine-tuned model's high Exact Match score demonstrates that training on LLM-annotated labels effectively anchors the model to specific percentage values rather than just ranking parties correctly.

Significance and Contributions

The paper claims three primary contributions:

1. New Task Formulation: It introduces "Responsibility Distribution Estimation" as a novel task for multimodal accident understanding, moving beyond binary classification or causal explanation to proportional allocation.
2. Ego-Centric Foundation: It highlights the unique value of ego-view videos for driver-centered reasoning, establishing the first benchmark for this specific task.
3. Feasibility Demonstration: The results establish a strong initial benchmark, demonstrating that multimodal LLMs can effectively perform nuanced, constraint-based reasoning (allocating percentages summing to 100) when fine-tuned with appropriate data.

The authors position this work as a foundation for "socially and legally meaningful multimodal reasoning," suggesting that ego-centric videos offer a promising basis for understanding accident avoidability and responsibility distribution that external viewpoints cannot provide.

Limitations and Ethical Considerations

The paper maintains a modest stance regarding its claims:

• Label Validity: The responsibility labels are generated by an LLM and serve as a proxy for evaluation; they are not human expert annotations or legal ground truth. The current process does not account for jurisdiction-specific laws.
• Data Scope: The model relies on sampled frames and text, lacking explicit physical evidence (e.g., exact speed, braking force, trajectories), which may limit fine-grained temporal reasoning.
• Complexity: The current formulation focuses on two-party interactions; accidents involving three or more agents require more complex decomposition not yet addressed.
• Ethical Warning: The system is explicitly defined as an exploratory research prototype. It must not be used for automated legal, judicial, or insurance decision-making. The authors emphasize the risk of automation bias and the necessity of a human-in-the-loop architecture for any future real-world application. Additionally, privacy concerns regarding personally identifiable information in dashcam videos are noted.