A Modular Zero-Shot Pipeline for Accident Detection, Localization, and Classification in Traffic Surveillance Video

1. Problem Statement

The paper addresses the ACCIDENT @ CVPR 2026 challenge, which requires analyzing traffic surveillance videos to predict three specific attributes of a traffic accident without using any labeled real-world training data:

1. Temporal Localization: The exact time (in seconds) the accident occurs.
2. Spatial Localization: The normalized $(x, y)$ coordinates of the impact point within the video frame.
3. Collision Classification: The type of collision from a five-class taxonomy: head-on, rear-end, sideswipe, single-vehicle, and t-bone.

Key Constraints:

• Zero-Shot Setting: Participants are provided only with synthetic training data (generated via the CARLA simulator) and real-world test data (CCTV footage). No fine-tuning on real-world labels is allowed.
• Domain Gap: The system must generalize from synthetic, high-quality renders to real-world footage characterized by varying resolutions, lighting, compression artifacts, and camera angles.
• Scoring: The final score is the harmonic mean of temporal accuracy, spatial accuracy, and classification accuracy. A failure in any single dimension results in a score of zero for that component, heavily penalizing partial failures.

2. Methodology

The authors propose a modular, zero-shot pipeline consisting of three independent components. No domain-specific fine-tuning is performed; the system relies entirely on pre-trained model weights and classical signal processing.

A. Temporal Localization (Peak Detection)

• Goal: Identify the frame index of the collision.
• Mechanism:
  1. Frame Differencing: Computes the mean absolute difference between consecutive grayscale frames ($I_t$ and $I_{t+1}$).
  2. Smoothing: Applies a centered rolling mean (window size $w=5$) to suppress short-lived noise (e.g., camera shake).
  3. Anomaly Detection: Normalizes the smoothed signal into z-scores.
  4. Selection: Frames exceeding a threshold ($\tau = 1.5$) are candidates. The frame with the highest z-score is selected as the accident time ($t^*$). If no threshold is crossed, the global maximum is chosen.

B. Spatial Impact Localization (Optical Flow Centroid)

• Goal: Determine the $(x, y)$ coordinates of the impact.
• Mechanism:
  1. Windowing: Centers a 30-frame window on the predicted accident time $t^*$.
  2. Optical Flow: Computes dense optical flow using the Farneback algorithm (multi-scale Gaussian pyramid) for consecutive frames in the window.
  3. Accumulation: Sums the magnitude of flow vectors ($M(u,v)$) across the window to create a cumulative motion map.
  4. Thresholding: Applies a 90th-percentile threshold to the magnitude map to suppress diffuse background motion (e.g., moving clouds or distant traffic) and retain only high-magnitude clusters.
  5. Centroid Calculation: Computes the weighted centroid of the thresholded map to determine the impact coordinates $(c^*_x, c^*_y)$.

C. Collision Type Classification (CLIP)

• Goal: Classify the accident type into one of five categories.
• Mechanism:
  1. Prompt Engineering: For each class, five distinct natural language descriptions are created (e.g., "two cars colliding head-on"). These are averaged to create a robust text embedding ($t_k$) using the CLIP text encoder.
  2. Visual Encoding: Eight frames centered around the predicted accident time are extracted, resized, and encoded using the CLIP visual encoder (ViT-B/32). These are averaged into a single image embedding ($v$).
  3. Similarity Matching: The predicted class is the one with the highest cosine similarity between the image embedding and the text embeddings.

3. Key Contributions

• Zero-Shot Architecture: Demonstrates a fully functional accident detection pipeline that operates without any labeled real-world data, relying solely on pre-trained models (CLIP) and classical algorithms (Farneback).
• Modularity: The pipeline is designed with three decoupled modules. This allows individual components (e.g., swapping Farneback for a learned optical flow estimator like RAFT) to be upgraded without retraining the entire system.
• Signal Processing Integration: Effectively combines statistical anomaly detection (z-scores) and image moment analysis (weighted centroids) with modern vision-language models to solve a complex spatiotemporal problem.
• Public Implementation: The code is released as a public Kaggle notebook, promoting reproducibility.

4. Results and Performance

• Leaderboard Score: The pipeline achieved a public leaderboard score of 0.2523 on the real-world CCTV test set.
• Component Analysis (Calibration Subset):
  • Temporal ($T$): Mean score of 0.438. The system is generally effective at detecting the moment of high motion.
  • Spatial ($S$): Mean score of 0.168. Performance is moderate; errors often arise when multiple vehicles move simultaneously, causing the centroid to drift.
  • Classification ($C$): Mean score of 0.0 on the calibration subset. The model failed to predict "head-on" collisions correctly, defaulting to "t-bone."
• Error Patterns:
  1. False Positives: Large background motion (swaying trees, clouds) triggers the temporal detector.
  2. Centroid Drift: Simultaneous movement of multiple vehicles dilutes the optical flow centroid.
  3. Domain Shift in Classification: CLIP struggles with CCTV angles (overhead/oblique) compared to its training data (eye-level internet photos). The model predicted "sideswipe" and "single-vehicle" most frequently, inverting the distribution seen in the synthetic training data (where "rear-end" was dominant).

5. Significance and Future Directions

• Significance: This work proves that accident detection is feasible in a zero-shot setting, offering a baseline for emergency dispatch systems that do not require expensive, site-specific data annotation. It highlights the potential of combining classical computer vision with foundation models.
• Limitations: The primary bottleneck is the classification module, which suffers from a significant domain gap between synthetic/CLIP training data and real-world surveillance footage.
• Future Work:
  • Replacing Farneback optical flow with learned estimators (e.g., RAFT) to improve accuracy on low-resolution inputs.
  • Fine-tuning the CLIP visual encoder on the synthetic dataset to bridge the gap between internet imagery and surveillance stills, which the authors identify as the most promising path to improving the overall score.

In conclusion, the paper presents a robust, modular framework that successfully localizes accidents in time and space using zero-shot techniques, while identifying that improving the visual-language alignment for classification is the critical next step for real-world deployment.