CrashSight: A Phase-Aware, Infrastructure-Centric Video Benchmark for Traffic Crash Scene Understanding and Reasoning

1. Problem Statement

Cooperative Autonomous Driving (CDA) requires traffic scene understanding from both vehicle and infrastructure perspectives. While Vision-Language Models (VLMs) have shown promise in general traffic scenarios, their performance in safety-critical crash events remains insufficiently evaluated.

• The Gap: Existing benchmarks (e.g., DriveQA, SUTD-TrafficQA) are predominantly ego-vehicle centric (dashcam views). They lack infrastructure-based viewpoints (roadside cameras), which are crucial for V2X (Vehicle-to-Everything) perception, post-incident analysis, and traffic management.
• The Challenge: Current infrastructure datasets either lack textual annotations, focus only on general anomaly detection (e.g., theft, fighting), or fail to provide structured, phase-aware temporal reasoning required for complex crash mechanics and causal attribution.

2. Methodology: The CrashSight Benchmark

The authors introduce CrashSight, the first large-scale VQA benchmark specifically designed for infrastructure-based crash understanding.

A. Dataset Construction

• Source: 250 real-world crash videos sourced from the TAD corpus, captured by roadside surveillance cameras.
• Selection Criteria: Videos must contain a visible collision/near-miss, sufficient pre-crash context for causal reasoning, and observable post-crash aftermath.
• Split: 60% Training, 20% Validation, 20% Testing (clip-level disjoint).

B. Phase-Aware Annotation Pipeline

A key innovation is the Phase-Aware Dense Captioning approach, which decomposes crashes into four temporal phases to preserve narrative structure:

1. Pre-crash Context: Road environment and vehicle movements.
2. Collision Dynamics: Impact configuration and trajectories.
3. Aftermath: Post-crash scene state and emergency response.
4. Potential Causes: Expert-level causal analysis.

The annotation process involves a three-stage pipeline:

1. VLM-Assisted Drafting: InternVL3-80B generates initial captions with explicit phase boundaries.
2. Human Expert Refinement: Three experts correct entity precision, spatial relations (critical for oblique angles), phase boundaries, and causal specificity. (Approx. 90% of drafts required substantial correction).
3. LLM-Driven Verification: Ensures internal consistency and standardizes terminology.

C. VQA Taxonomy (13K QA Pairs)

The benchmark generates ~13,000 multiple-choice questions across 7 categories, organized into two tiers:

• Tier 1 (Visual Grounding):
  • Scene Identification (SI): Road type, weather, lighting.
  • Involved Parties (IP): Vehicle/VRU types, counts, features.
  • Post-Crash Outcome (PCO): Damage status, emergency response.
• Tier 2 (Reasoning & Inference):
  • Crash Mechanics (CM): Collision type, impact dynamics.
  • Fault Determination (FD): Causal attribution and traffic violations.
  • Temporal Sequence (TS): Event ordering and phase identification (unanswerable by single frames).
• Robustness:
  • Hallucination Probes: Questions testing if models can identify "cannot be determined" scenarios (e.g., driver BAC, phone usage) or reject false premises.

3. Experimental Setup

• Models Evaluated: 8 VLM configurations (including LLaVA-OneVision, Qwen2.5-VL, LLaVA-NeXT, InternVL3) ranging from 0.5B to 8B parameters.
• Settings: Zero-shot evaluation and Fine-tuning (using QLoRA on Qwen2.5-VL 3B/7B).
• Human Baseline: Three experts answered test questions via majority vote to establish an upper bound.
• Input Processing: 4 frames uniformly sampled at 1 FPS, resized to ~100k–200k pixels.

4. Key Results

• Performance Gap: Even the best zero-shot models (InternVL3-8B) achieved only 68.7% average accuracy, significantly below the human expert baseline (94.7%).
• Impact of Fine-Tuning: Domain-specific fine-tuning yielded substantial gains:
  • +16.1% average accuracy improvement for the 3B model.
  • +13.5% for the 7B model.
  • The fine-tuned 3B model (74.7%) outperformed the zero-shot 8B model (68.7%), proving that domain adaptation outweighs scale for this specific task.
• Category Difficulty:
  • Hardest: Involved Parties (IP) and Crash Mechanics (CM). Even fine-tuned models struggled (max ~69%), highlighting difficulties in identifying entities under partial occlusion and understanding collision dynamics from oblique angles.
  • Easiest: Post-Crash Outcome (PCO) and Robustness (after fine-tuning), where models learned to select "cannot be determined" correctly.
• Architecture vs. Scale: In zero-shot settings, InternVL3 outperformed larger models (e.g., 2B model beat 7B Qwen), suggesting its multi-image processing pipeline offers superior temporal reasoning capabilities.

5. Systematic Error Analysis

The authors identified three primary bottlenecks preventing models from reaching human-level performance:

1. Insufficient Visual Token Budget: Uniform sampling (4 frames) fails to capture short-duration crash events (often <2s) in long videos (up to 70s). Low resolution compresses spatial details, making small entities (pedestrians, cyclists) hard to distinguish.
2. Quantization Overhead: 4-bit NF4 quantization degrades fine-grained feature discrimination needed for entity recognition.
3. Frozen Visual Encoder: Fine-tuning only the language backbone and projector leaves the visual encoder frozen. Models improve in reasoning about crash concepts but fail to improve perception of surveillance-specific challenges (oblique angles, occlusion).

Error Transition: Fine-tuning eliminated "format-level" errors (e.g., treating video as static images, hallucinating unobservable facts) but could not resolve "visual-perception" errors (e.g., misidentifying vehicle types or crash mechanics).

6. Significance and Contributions

• First Infrastructure-Centric Crash Benchmark: Fills a critical gap in CDA research by shifting focus from ego-vehicle to roadside infrastructure perspectives.
• Structured Evaluation Framework: Introduces a phase-aware taxonomy that separates visual grounding from higher-level causal and temporal reasoning.
• Scalable Annotation Methodology: Demonstrates a hybrid pipeline (VLM drafting + Human refinement + LLM verification) for constructing high-quality safety-critical datasets.
• Actionable Insights: Identifies that future progress in traffic crash understanding relies less on scaling parameters and more on adaptive frame sampling, higher-resolution visual encoders, and spatial grounding mechanisms tailored for surveillance data.

The benchmark and code are publicly available at https://mcgrche.github.io/crashsight/.