Metadata-Aware Multi-Prompt Reasoning for Zero-Shot Accident Understanding

Technical Summary: Metadata-Aware Multi-Prompt Reasoning for Zero-Shot Accident Understanding

Problem Statement
The paper addresses the challenge of zero-shot accident understanding in surveillance videos. Unlike standard event detection, this task requires a model to simultaneously identify three distinct components without access to labeled real-world training data:

1. When: The precise temporal occurrence of the impact.
2. What: The specific category of the accident (e.g., rear-end, T-bone, head-on).
3. Where: The spatial coordinates of the physical point of impact within the video frame.

The authors note that real-world CCTV footage presents significant difficulties, including short impact durations, visually similar accident categories, and the tendency of models to focus on prominent objects rather than the actual interaction between road agents. Existing Vision-Language Models (VLMs) often struggle when asked to solve this complex, multi-faceted task via a single end-to-end query, leading to unstable predictions.

Methodology
The proposed solution is a three-stage pipeline that decomposes the problem into When, What, and Where, utilizing open-weight models without fine-tuning.

1. Temporal Detection (When):

   • Mechanism: The system uses Meta's Perception Encoder (PE), a contrastive vision-language model, to score sampled frames (at 8 FPS) based on their semantic alignment with the text query "traffic accident."
   • Process: Frames are ranked by cosine similarity. The top-$K$ frames are selected, and a temporal window is defined by expanding the timestamps of the earliest and latest selected peaks by a fixed offset ($\delta = 2$ seconds).
   • Output: The predicted accident time is the midpoint of this expanded window.

2. Metadata-Aware Multi-Prompt Classification (What):

   • Mechanism: A generative VLM (Qwen-3.5-VL 9B) is queried with five complementary prompts over the selected key frames and metadata (scene layout, weather, time of day).
   • Prompt Views: The five prompts cover: (1) baseline direct classification, (2) motion analysis, (3) geometric contact reasoning, (4) contrastive elimination, and (5) a tiebreaker.
   • Adjudication: Outputs are aggregated via an uncertainty-aware voting procedure. If the vote distribution is ambiguous (low margin or high entropy), the system escalates to a two-step resolution: a structured tiebreaker prompt followed, if necessary, by a focused pairwise adjudication between the top two candidates conditioned on impact geometry.

3. Spatial Impact Localization (Where):

   • Mechanism: An open-vocabulary detector (OWL-v2) is used to localize the impact.
   • Conditioning: Instead of generic queries, the detector is prompted with phrases conditioned on both the predicted accident type (from Stage 2) and the scene layout (from metadata). For example, "car crashing into back of another car on a highway."
   • Aggregation: Detections across key frames are pooled. The final impact point is calculated as the score-weighted centroid of the top-$K$ bounding box centers, and the impact region is the union of these boxes. This aggregation enforces temporal consistency, down-weighting transient false positives.

Key Contributions

• Decomposed Framework: A three-stage "When-What-Where" pipeline for zero-shot accident understanding from CCTV, where each stage is independently upgradable.
• Multi-Prompt Reasoning: A five-prompt classification scheme with entropy-gated pairwise adjudication that resolves ambiguous votes without requiring labeled calibration data.
• Type-Conditioned Localization: A strategy that conditions the spatial detector on the predicted accident type and scene context, which the authors argue mitigates foreground bias in existing pointing models.
• Efficiency: The system operates on a single 24 GB GPU using only open-weight models and requires no fine-tuning.

Results
The method was evaluated on the ACCIDENT@CVPR 2026 benchmark, which utilizes synthetic CARLA data for development and real fixed-view CCTV clips for testing. The evaluation metric is the harmonic mean of temporal accuracy ($T$), spatial accuracy ($S$), and classification accuracy ($C$).

• Performance: The proposed pipeline achieved a Private Leaderboard score of 0.4015, significantly outperforming rule-based baselines (Baseline A: 0.2734; Baseline B: 0.3188).
• Component Analysis:
  • Spatial Localization: Provided the largest performance gain. Using OWL-v2 with type/scene conditioning improved the score by 0.053 over a center-of-frame baseline.
  • Temporal Detection: The $\delta$-expanded window improved the score by 0.039 over using a single top-similarity frame.
  • Classification: The multi-prompt ensemble with adjudication provided a consistent, albeit smaller, improvement (0.0054) over a single-prompt setting.
• Error Analysis: Residual errors were primarily attributed to distant collisions where the detector missed the impact or selected a foreground vehicle, as well as adverse conditions like rain, night, or occlusion.

Significance and Claims
The paper claims that decomposing zero-shot video understanding into temporal localization, semantic classification, and spatial grounding enables more reliable reasoning with Vision-Language Models than direct, single-step prompting. The authors highlight that explicitly conditioning the spatial detector on the predicted accident type reduces foreground-object bias, a common failure mode in pointing models. The work demonstrates that a modular, perception-first pipeline can achieve robust zero-shot performance on challenging surveillance footage without the need for domain-specific fine-tuning.