💻 computer science

GridVAD: Open-Set Video Anomaly Detection via Spatial Reasoning over Stratified Frame Grids

GridVAD is a training-free, open-set video anomaly detection pipeline that leverages Vision-Language Models as anomaly proposers refined by Self-Consistency Consolidation and grounded by detection-segmentation models to achieve state-of-the-art pixel-level performance without domain-specific training.

Original authors: Mohamed Eltahir, Ahmed O. Ibrahim, Obada Siralkhatim, Tabarak Abdallah, Sondos Mohamed

Published 2026-03-27

📖 5 min read🧠 Deep dive

CC0 1.0

Original authors: Mohamed Eltahir, Ahmed O. Ibrahim, Obada Siralkhatim, Tabarak Abdallah, Sondos Mohamed

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the head of security for a massive, busy city square. Your job is to spot anything weird happening—someone running the wrong way, a car driving on the sidewalk, or a fight breaking out.

In the past, security systems were like trained dogs. You had to teach them specifically what "bad" looked like (e.g., "a dog barking"). If a cat jumped over a fence, the dog didn't know what to do because it had never been trained on cats.

Then, we got Vision-Language Models (VLMs). Think of these as super-smart, well-read librarians who have read every book and seen every movie in the world. They can describe anything they see in perfect English. "That's a person running," or "That's a car driving on the grass."

The Problem: The Librarian is Too Chatty (and Hallucinates)

The researchers in this paper realized that if you just ask the Librarian, "Is this an emergency?" they often get it wrong.

Why? The Librarian has seen millions of videos of people running on sidewalks (which is normal) and millions of videos of people running in movies (which might be an emergency). They don't know the specific rules of your city square.
The Result: If you ask the Librarian to judge every single second of video, they will scream "EMERGENCY!" every time someone jogs, or even when a shadow moves. They are too sensitive and make up things that aren't there (hallucinations).

The Solution: GridVAD (The "Propose-Ground-Propagate" Team)

Instead of asking the Librarian to be the judge, the authors created a new team workflow called GridVAD. They split the job into three distinct roles, like a well-oiled machine:

1. The Librarian becomes the "Idea Generator" (The Proposer)

Instead of asking the Librarian to say "Yes/No," they ask them to just list ideas.

The Trick: They don't show the Librarian the video one second at a time. That's too much work and too confusing.
The Grid: They take a 10-second clip and chop it into a 9-panel comic strip (a grid). They show the Librarian this whole comic strip at once.
The Ask: "Look at this comic strip. Write down a list of anything that looks weird, and tell me when it happened."
The Benefit: The Librarian is great at describing weird things ("A person is skateboarding in a no-skate zone"), but bad at deciding if it's actually an emergency. So, we just let them write the list.

2. The "Reality Check" (Self-Consistency Consolidation)

Here is the magic part. The Librarian is a bit jittery. If you ask them the same question 5 times, they might give 5 slightly different answers. Sometimes they get it right; sometimes they make up a ghost.

The Strategy: The system asks the Librarian to look at the same video clip 5 different times (using slightly different random frames).
The Filter: If the Librarian says, "I see a skateboarder!" in 4 out of 5 times, the system says, "Okay, that's probably real." If they say, "I see a flying unicorn!" only once, the system says, "Nah, that was a hallucination. Ignore it."
The Analogy: It's like asking 5 different witnesses to describe a crime. If 4 of them say "a red car," you trust it. If one guy says "a purple dragon," you know he's making it up.

3. The "Detective and the Paintbrush" (Grounding & Propagation)

Now that we have a trusted list of weird events (e.g., "Skateboarder from 2:00 to 2:15"), we need to find them on the screen.

The Detective (Grounding DINO): We take the description ("Skateboarder") and ask a specialized detective model to find the exact box around that skateboarder in the video.
The Paintbrush (SAM2): Once the detective finds the skateboarder in one frame, we use a "smart paintbrush" (SAM2) to automatically trace the skateboarder's shape in every single frame before and after that moment. This creates a perfect, moving mask that highlights exactly who is doing the weird thing.

Why is this a Big Deal?

No Training Needed: You don't need to feed the system thousands of videos of "bad things" to teach it. It works out of the box (Zero-Shot) because it uses the Librarian's general knowledge.
Saves Money: Instead of asking the expensive Librarian to look at every single frame (which would cost a fortune in computer time), GridVAD only asks them 5 or 6 times per video clip, regardless of how long the video is. It's 2.7 times more efficient.
Better Accuracy: On the UCSD Ped2 test (a standard security video dataset), this method found pixel-level anomalies better than any other method, even ones that were specially trained on that specific data.

The Catch

The system is very good at precision (when it finds something, it's almost certainly real and perfectly outlined). However, it sometimes misses things if the Librarian doesn't think to mention them in the first place. It's better to have a few perfect alerts than a thousand false alarms.

In short: GridVAD stops trying to make the AI a "judge" and turns it into a "witness." It gathers multiple witness accounts, filters out the liars, and then sends a specialized team to find and highlight the actual troublemakers.

1. Problem Statement

Video Anomaly Detection (VAD) aims to identify and localize events that deviate from normal scene dynamics. It is inherently an open-set, unbalanced problem because anomalies are rare, diverse, and context-dependent (e.g., running is normal on a sidewalk but anomalous in a factory).

Current Limitations: Traditional VAD methods rely on training data (supervised, weakly supervised, or one-class), limiting their ability to generalize to unseen anomaly types.
The VLM Challenge: While Vision-Language Models (VLMs) possess broad visual knowledge and can reason about novel concepts, using them directly as anomaly detectors is fragile. VLMs are trained on internet-scale data where anomalies are not rare or systematically labeled. Consequently, they produce uncalibrated anomaly scores, leading to high rates of false alarms (hallucinations) and missed detections when asked to make binary "anomaly vs. normal" decisions.

2. Methodology: The "Propose-Ground-Propagate" Pipeline

The authors propose a paradigm shift: instead of using VLMs as direct detectors, they should function as open-set proposers. The pipeline, named GridVAD, is training-free (no fine-tuning on anomaly data) and consists of four stages:

A. Stratified Grid Sampling

To handle temporal context efficiently without processing every frame individually:

A video clip is divided into $K$ equal-width temporal bins.
The clip is represented as a stratified spatial grid of $K$ frames (e.g., $3 \times 3$ grid = 9 frames).
$M$ independent samplings are performed. In each sampling, one frame is randomly drawn from each bin and tiled into a single image. This ensures every temporal bin is represented while providing diverse views of the clip at negligible computational cost.

B. Open-Set Anomaly Proposal (VLM)

Each stratified grid image is fed into a VLM (e.g., Qwen3-VL) with a structured prompt.
The VLM generates free-form natural language descriptions of potential anomalies, including temporal intervals ( $t_s, t_e$ ) and confidence scores.
Crucially, the VLM is not asked to decide if something is an anomaly; it only describes what it sees as unusual.

C. Self-Consistency Consolidation (SCC)

This is the core innovation for filtering noise:

The $M$ sets of proposals are pooled.
A text-only LLM acts as a semantic consolidator, grouping proposals that refer to the same object/event across different samplings.
Filtering Mechanism: Only proposals that appear in at least $\tau$ independent samplings (support count $\sigma_j \geq \tau$ ) are retained.
Logic: Genuine anomalies are visually consistent and will recur across samplings; hallucinations are stochastic and will likely appear only once. This converts a noisy proposer into a reliable signal.

D. Spatial Grounding and Temporal Propagation

Grounding: The natural language description of a surviving proposal is passed to Grounding DINO (a zero-shot open-vocabulary detector) to generate a bounding box in the most informative anchor frame.
Propagation: The bounding box is used as a prompt for SAM2 (Segment Anything Model 2) to propagate the mask forward and backward through the temporal window, generating dense, pixel-level anomaly masks.

3. Key Contributions

VLM-as-Proposer Principle: The paper articulates that VLMs should be used to generate open-set descriptions rather than binary scores. This separates "recognition/description" (VLM's strength) from "decision/localization" (handled by specialized models).
Self-Consistency Consolidation (SCC): A lightweight, parameter-free mechanism that treats the VLM as a stochastic sensor. It filters hallucinations by retaining only statistically supported proposals, enabling a controllable precision-recall tradeoff.
Stratified Grid Representation: Converts temporal detection into a single-pass spatial reasoning task. This allows for budget-controlled VLM querying (fixed $M+1$ calls per clip regardless of video length), making it significantly more efficient than per-frame querying.

4. Experimental Results

The method was evaluated on UCSD Ped2 and ShanghaiTech Campus benchmarks.

Performance on UCSD Ped2:
- Achieved the highest Pixel-AUROC (77.59) among all compared methods, surpassing even the partially fine-tuned state-of-the-art method TAO (75.11).
- Outperformed other zero-shot approaches on object-level metrics (RBDC) by over 5×.
Efficiency:
- GridVAD is 2.7× more call-efficient than uniform per-frame VLM querying.
- It produces dense segmentation masks, whereas uniform sampling often lacks temporal context or requires linear scaling of API calls.
Ablation Studies:
- Removing SCC ( $M=1$ ) increases object-level recall (RBDC/TBDC) but drastically reduces pixel-level precision (due to hallucinated masks).
- SCC effectively filters noise, improving Pixel-AUROC by ~7.3 points and Pixel-F1 by ~13.8 points on ShanghaiTech, at the cost of some object-level recall.

5. Significance and Limitations

Significance:

Training-Free: GridVAD achieves state-of-the-art performance without any dataset-specific fine-tuning or closed-set category lists.
Robustness: The propose-ground-propagate decomposition effectively mitigates the "hallucination" problem inherent in using VLMs for detection.
Scalability: The fixed budget per clip makes it practical for long video surveillance where per-frame processing is computationally prohibitive.

Limitations:

Proposal Recall Bottleneck: The system can only detect anomalies the VLM successfully proposes. If the VLM misses an event in all $M$ samplings, it is undetected.
Object-Level Metrics: While pixel-level metrics are superior, object-level recall (RBDC/TBDC) is lower than exhaustive tracking methods like TAO because GridVAD prioritizes mask precision over exhaustive recall.
Clip Independence: Anomalies spanning clip boundaries may be missed or duplicated, as the pipeline processes clips independently.

In conclusion, GridVAD demonstrates that by repositioning VLMs as open-set proposers and using statistical consistency to filter noise, it is possible to achieve highly accurate, training-free video anomaly detection with dense pixel-level localization.