Technical Summary: From Accuracy to Visual Dependence: Auditing and Filtering Modality Collapse in Traffic VideoQA

1. Problem Statement

High benchmark accuracy in Video Question Answering (VideoQA) does not guarantee that models are utilizing visual evidence. In the domain of traffic accident analysis, correct answers should theoretically depend on scene-specific visual cues (e.g., vehicle interactions, temporal dynamics, causal structures). However, models often exhibit modality collapse, where they rely on textual shortcuts, language priors, or dataset-specific artifacts to answer correctly, effectively ignoring the video input.

This failure mode is particularly critical in safety-critical applications like traffic accident analysis, where systems must succeed because they understand the visual scene, not because they exploit annotation artifacts. The paper argues that current benchmarks often fail to distinguish between genuine visual reasoning and shortcut exploitation, as high accuracy can be achieved even when video input is removed or when adding more frames degrades performance.

2. Methodology

The authors propose a diagnostic framework to audit and filter modality collapse at both the dataset and instance levels.

2.1 Dataset-Level Diagnostics

The authors introduce two metrics to quantify visual dependence across a benchmark:

• Blind Gap (BG): Measures how far text-only accuracy exceeds random chance.
  $$\text{BlindGap}(D) = \text{Acc}_{\text{blind}} - \text{Acc}_{\text{random}}$$
  A high BG indicates that the benchmark is solvable using text alone.
• Visual Gain (VG): Measures the marginal benefit of adding video input.
  $$\text{VisualGain}(D) = \text{Acc}_{\text{video}} - \text{Acc}_{\text{blind}}$$
  A positive VG indicates useful visual evidence, while a negative VG suggests that video input actively degrades performance or is ignored.

2.2 Instance-Level Shortcut Diagnostics

To identify specific questions that drive shortcut behavior, the authors define a continuous Shortcut Score ($S(q)$) for each question $q$:

• Textual Evidence Score ($T(q)$): Measures the confidence and correctness of text-only Large Language Models (LLMs). It is calculated as the average of confidence-weighted correct predictions across a panel of text-only models.
  $$T(q) = \frac{1}{|\mathcal{M}_{\text{LLM}}|} \sum_{m \in \mathcal{M}_{\text{LLM}}} [\text{Correct}_m(q)] \cdot (1 - H_{\text{norm}, m}(q))$$
• Visual Necessity Score ($V(q)$): Measures whether adding video increases the probability assigned to the correct answer.
  $$V(q) = \frac{1}{|\mathcal{M}_{\text{VLM}}|} \sum_{m \in \mathcal{M}_{\text{VLM}}} \frac{\Delta_m(q)}{1 - 1/K}$$
  where $\Delta_m(q)$ is the probability gain for the correct answer when video is added.
• Shortcut Score ($S(q)$): Combines these signals:
  $$S(q) = T(q) - V(q)$$
  A high $S(q)$ indicates a shortcut-prone question (high text solvability, low visual necessity), while a low or negative $S(q)$ indicates a visually useful question.

2.3 Shortcut-Aware Filtering

The authors propose constructing a "grounded" subset of the dataset by retaining only questions where $S(q) \leq \tau$. This filtering is training-free and continuous, allowing for a trade-off between data coverage and the removal of shortcut-prone samples.

3. Key Contributions

1. Audit of Traffic VideoQA Benchmarks: The paper audits four public benchmarks (MM-AU, VRU-Accident, SUTD-TrafficQA, and AccidentBench) using open-weight Vision-Language Models (VLMs) and text-only LLMs.
2. Diagnostic Metrics: Introduction of Blind Gap and Visual Gain to quantify modality collapse at the dataset level.
3. Shortcut Score: A novel, continuous, instance-level metric that combines text-only confidence with visual necessity signals to identify and filter shortcut-prone questions without auxiliary training.
4. Filtering Framework: A method to generate "grounded" evaluation subsets that reduce shortcut bias and improve visual grounding metrics.

4. Experimental Results

The authors evaluated multiple VLMs (e.g., InternVL, Qwen-VL, LLaVA-OneVision) and text-only LLMs across the four benchmarks.

• Benchmark Disparities:
  • MM-AU: Exhibits extreme modality collapse. Removing video consistently improved accuracy, and adding more frames (up to 256) further degraded performance. The Blind Gap was high (up to 40.33%), and Visual Gain was negative for all VLMs. Text-only LLMs matched or exceeded VLM performance.
  • VRU-Accident: Showed positive Visual Gain but still suffered from a high Blind Gap (up to 25%), indicating partial text solvability.
  • SUTD-TrafficQA: Demonstrated the strongest visual dependence, with low Blind Gap (<9%) and consistently positive Visual Gain (~11-12%).
  • AccidentBench: Fell between these extremes with low Blind Gap and modest Visual Gain.
• Filtering Efficacy:
  • Applying the Shortcut Score filter (with $\tau = 0.1$) to MM-AU and VRU-Accident significantly reduced the Blind Gap and turned negative Visual Gain into positive values.
  • On MM-AU, filtering reduced BG from 25.70% to 10.38% and shifted VG from -0.24% to 7.42%.
  • Binary filtering baselines (LLM-Inspector, B-Clean) were less effective; B-Clean, for instance, removed visually useful questions along with shortcuts, resulting in strongly negative Visual Gain.
• Root Cause Analysis: The paper identifies repetitive text patterns and answer phrasing (e.g., specific sentence structures appearing hundreds of times) as a primary driver of shortcuts in MM-AU.
• Training Impact: Fine-tuning models on traffic data improved video accuracy but did not necessarily improve visual grounding (BG and VG remained flat or negative on standard subsets), suggesting that accuracy gains do not equate to genuine visual reasoning.

5. Significance and Claims

The paper claims that benchmark accuracy alone is insufficient for evaluating progress in safety-critical VideoQA. The findings reveal that:

• Several widely used traffic accident benchmarks allow models to succeed by exploiting textual shortcuts rather than visual evidence.
• Visually grounded evaluation, measured by metrics like Blind Gap and Visual Gain, is essential to distinguish genuine reasoning from shortcut exploitation.
• The proposed Shortcut Score provides a practical, training-free mechanism to curate evaluation subsets that better reflect a model's ability to use visual evidence.
• Future dataset construction must avoid repetitive question formats and identical answer phrasing to prevent models from exploiting high-frequency patterns.

The authors conclude that for multimodal systems to be reliable in safety-critical domains, they must be evaluated on their ability to depend on visual evidence, not just their final accuracy scores.