💻 computer science

Two-Pass Zero-Shot Temporal-Spatial Grounding of Rare Traffic Events in Surveillance Video

This paper introduces a zero-shot, no-fine-tuning pipeline that leverages a two-pass coarse-to-fine strategy and specialized vision-language model role assignment to achieve state-of-the-art temporal-spatial grounding of rare traffic accidents in surveillance video, outperforming existing baselines by 12.7% on the ACCIDENT@CVPR 2026 benchmark.

Original authors: Jiantang Huang

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiantang Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a crime that happened in a busy city square, but you only have a single, grainy security camera feed that is 30 seconds long. Your job is to find three specific things: exactly when the crash happened, exactly where on the screen it happened, and what kind of crash it was (e.g., a rear-end collision vs. a sideswipe).

The problem? You can't hire a team of human experts to watch every video, and you aren't allowed to "study" real crash videos to learn how to spot them (because of privacy laws). You have to solve this using only "off-the-shelf" AI tools that haven't been specifically trained on accidents.

This paper presents a clever, two-step strategy to solve this puzzle using two different types of AI "detectives."

The Problem: The "One-Shot" Mistake

If you ask a standard AI to watch the whole 30-second video and guess the crash time, location, and type all at once, it often gets confused. It's like asking a person to watch a whole movie and tell you the exact second a specific actor sneezed, while also pointing to the exact spot on the screen and naming the sneeze type. They might guess the right movie, but get the timing wrong by 20 seconds or mix up the sneeze with a cough.

The authors found that previous AI attempts were often off by huge margins (like guessing the crash happened 21 seconds after it actually did) or got the crash type completely wrong.

The Solution: The "Two-Pass" Detective Team

The authors created a pipeline that uses two different AI models working together in a specific order, like a senior detective and a specialist.

Pass 1: The "Wide-Angle" Scan (The Generalist)

First, they use a powerful AI (called Qwen3-VL) to watch the video at a slow speed (1 frame per second).

The Analogy: Imagine a detective walking through a crime scene quickly, looking at the whole room. They don't stop to examine every fingerprint yet. They just get a "gut feeling" about where the action was and roughly when it happened.
The Output: This AI gives a "coarse" guess: "The crash probably happened around second 15, near the middle of the screen, and it looks like a single-car accident."
The Safety Net: If the AI gets confused or the API (the internet connection to the AI) fails, the system has a backup plan using simple physics rules (like tracking car movement) to make a best guess, so it never leaves the video blank.

Pass 2: The "Zoom-In" Refinement (The Specialist)

Next, the system takes that rough guess and creates a tiny, high-definition "zoomed-in" clip of just the 6 seconds around the suspected crash time.

The Analogy: Now, the detective puts on a magnifying glass and watches just those 6 seconds in slow motion. They look for the exact moment of impact and the precise spot where the cars touched.
The Safety Gates: The system has two "safety checks."
1. Time Check: If the zoomed-in AI says, "I can't see a crash in this specific 6-second window," or if it guesses a time right at the edge of the window (which usually means it's guessing), the system ignores the new guess and sticks with the original "gut feeling" from Pass 1.
2. Space Check: If the zoomed-in AI points to a spot right on the edge of the screen (which is often a mistake), the system ignores it and uses the original location from Pass 1.

The "Specialist" for Crash Types

Finally, the system realizes that the first AI isn't very good at naming the type of crash (e.g., confusing a "sideswipe" with a "rear-end").

The Analogy: So, they hand the short, zoomed-in clip to a second, different AI expert (called Gemini 3.1) who is a specialist in identifying crash types. This expert looks only at the crash moment and says, "This is definitely a sideswipe."

The Results: Beating the Competition

The authors tested this "Two-Pass" team on a real-world benchmark with over 2,000 real CCTV videos.

The Score: They achieved a score of 0.539.
The Comparison: This was significantly better than the previous best attempts (which scored around 0.412) and much better than just using one AI model alone.
The Cost: The whole process cost about $20 to run on all 2,000 videos (roughly 1 cent per video), proving you don't need a supercomputer to get good results.

What They Found (The "Failure Analysis")

The paper also admits where the system still struggles, acting like a honest detective reporting their limitations:

Late Bias: The AI tends to guess the crash happened slightly after it actually did (about 1.5 seconds late). It often sees the wreckage after the impact rather than the moment of impact.
Confusing Types: The system is still bad at telling the difference between a "head-on" crash and a "T-bone" crash, or a "sideswipe" and a "rear-end." It gets these mixed up about 40-80% of the time.
Video Length: The longer the video, the harder it is for the AI to find the exact moment, similar to how it's harder to find a needle in a 1-hour haystack than in a 10-minute one.

Summary

In short, this paper shows that you don't need to train a new AI from scratch to find rare traffic accidents. Instead, you can use a smart "Two-Pass" strategy: Scan broadly first, zoom in carefully second, and use a different expert for naming the crash type. This approach, combined with simple safety checks to avoid bad guesses, creates a system that is much more accurate than previous methods, all while keeping costs low and respecting privacy rules.

Technical Summary: Two-Pass Zero-Shot Temporal-Spatial Grounding of Rare Traffic Events

Problem Statement
The paper addresses the challenge of grounding traffic accidents in real-world CCTV footage, specifically targeting the joint localization of time, spatial coordinates, and collision type. This is framed as a "rare-event" problem where the volume of labeled accident footage required for modern supervised learning is unavailable. Furthermore, privacy and liability concerns often prohibit training on real incident data. The work focuses on the ACCIDENT@CVPR 2026 benchmark, which provides 2,211 synthetic CARLA videos for development but strictly forbids training on the 2,027 real CCTV test clips. The evaluation metric is ACCS, the harmonic mean of scores for temporal (T), spatial (S), and collision type (C) accuracy, which penalizes systems with weak components in any single subtask.

Methodology
The authors propose a no-fine-tuning pipeline that leverages frozen Vision-Language Models (VLMs) through a two-pass decomposition and specialist role assignment. The system operates without training on real accident data, relying instead on the native capabilities of pre-trained models.

Coarse-to-Fine Two-Pass Decomposition:
- Pass 1 (Coarse Grounding): The full video is sampled at 1 fps (up to 30 frames) and processed by Qwen3-VL-Plus. The model is prompted to output a coarse tuple $(t_1, x_1, y_1, c_1)$ representing the impact time, location, and type. To aid temporal alignment, frames are serialized with explicit [Frame at t.ts] tags, allowing the model to treat time localization as a frame-naming task rather than continuous regression.
- Pass 2 (Fine Refinement): A $\pm 3$ second window around the coarse time estimate $t_1$ is extracted and sampled at 5 fps (up to 30 frames at 1024px resolution). Qwen3-VL-Plus re-evaluates this window to refine the time ( $t_2$ ) and location ( $x_2, y_2$ ) with 0.1s precision.
- Confidence Gates: Two deterministic gates determine whether to accept the refined Pass 2 estimates or revert to the robust Pass 1 estimates:
  - Temporal Gate: If Pass 2 returns a negative time or a time near the window boundary (indicating low confidence), the system reverts to $t_1$ .
  - Spatial Gate: If Pass 2 coordinates are "edge-clamped" (near the 0 or 1000 boundaries of the normalized grid), the system reverts to $(x_1, y_1)$ .
Specialist Role Assignment:
- Recognizing that a single VLM may not excel at all subtasks, the pipeline splits responsibilities. Qwen3-VL-Plus handles the temporal and spatial grounding (T+S).
- Gemini 3.1 Flash-Lite is assigned the specific role of collision type classification (C). A 5-second clip centered on the final time estimate is cropped (2.5x zoom around the Pass 1 center) and passed to Gemini with a closed-world prompt enumerating the five collision types. This specialization improves type accuracy significantly compared to using a single model for all tasks.
Fallback Mechanism:
- In cases of API failure (observed in ~17% of videos), the system falls back to a frozen detector pipeline (YOLO26x + ByteTrack) combined with rule-based physics scoring. This ensures no video is left unprocessed.

Key Contributions

No-Fine-Tuning Pipeline: A method that elicits joint T+S+C outputs from frozen VLMs, avoiding the need for labeled real-world accident data.
Two-Pass Decomposition with Gates: A strategy that refines high-salience neighborhoods while using deterministic confidence checks to prevent error propagation from the second pass.
Specialist Role Split: Demonstrating that assigning type classification to a second, specialized VLM (Gemini) lifts type accuracy from 0.474 to 0.591, a critical improvement for the harmonic mean score.
Systematic Failure Analysis: The paper provides a detailed diagnosis of failure modes, including a +1.55s late bias in temporal grounding, length-dependent degradation, and a physics-VLM oracle analysis.

Results
On the ACCIDENT@CVPR 2026 test set (2,027 videos), the proposed pipeline achieves an ACCS of 0.539 (95% CI [0.525, 0.553]).

Comparative Performance: This represents a +0.127 improvement over the benchmark paper's best-of-baselines oracle (0.412) and a +0.143 improvement over the strongest single-VLM baseline, Molmo-7B (0.396).
Subtask Breakdown: The method achieves T=0.50, S=0.54, and C=0.59. Notably, the type accuracy (C) is the highest reported, while spatial accuracy trails the best single-VLM baseline slightly.
Cost Efficiency: The full run costs approximately $20 for the entire test set (approx. $0.01 per video), utilizing up to three API calls per video.
Failure Diagnostics: The analysis reveals a mean temporal late bias of +1.55s (the model often identifies the post-collision wreckage rather than the contact moment) and significant confusion between "head-on" and "t-bone" collisions.

Significance and Claims
The paper claims that closing the gap between automated systems and human performance (Human ACCS = 0.843) in rare-event grounding without supervised training requires moving beyond hand-crafted primitives and single-shot prompting. The authors argue that their approach demonstrates the viability of training-free, zero-shot VLM pipelines for safety-critical tasks where data scarcity and privacy are primary constraints.

The authors remain modest regarding the current state of the system, noting that the residual temporal ambiguity (median 1.1s error) and specific type confusions (head-on/sideswipe) are not yet acceptable for fully automated first-notice-of-loss (FNOL) systems without human-in-the-loop gates. They conclude that future improvements should focus on adaptive sampling and physics-VLM fusion rather than simply scaling model size. The work establishes that specialized role assignment and coarse-to-fine refinement are effective levers for improving zero-shot grounding in complex, real-world surveillance scenarios.