QVAD: A Question-Centric Agentic Framework for Efficient and Training-Free Video Anomaly Detection

1. Problem Statement

Video Anomaly Detection (VAD) is a critical challenge in computer vision, particularly due to the open-set nature of anomalies (i.e., anomalies are unbounded and often unseen during training).

• Limitations of Traditional Methods: Supervised models are brittle, requiring extensive domain-specific data and failing to generalize to new environments without expensive retraining.
• Limitations of Current Training-Free Approaches: Recent methods leveraging Vision-Language Models (VLMs) and Large Language Models (LLMs) often rely on static prompts (e.g., "Is there an anomaly?"). To compensate for the ambiguity of these static prompts, they require massive, resource-intensive foundation models (e.g., 80B+ parameters). This restricts deployment to server-grade hardware, making them unsuitable for edge devices.
• Core Bottleneck: The authors argue the bottleneck is not model capacity, but the static nature of inquiry.

2. Methodology: QVAD Framework

QVAD proposes a Question-Centric Agentic Framework that treats VAD as a dynamic dialogue between an LLM agent and a VLM perception module. Instead of a one-shot query, it uses an iterative "prompt-updating" mechanism to refine understanding on the fly.

Key Components:

1. Two-Stage Spatiotemporal Abstraction:
   • Uniform Sampling: Reduces a 128-frame window to 32 frames to ensure temporal coverage.
   • Motion-Aware Salience Selection: Further selects the top 8 frames based on motion saliency (Gaussian-blurred frame differences), retaining the first and last frames for context. This ensures the VLM focuses on the most dynamic segments.
2. Iterative Agentic Reasoning Loop:
   • Initialization: The VLM generates an initial caption ($C_t$) based on a broad system prompt.
   • Reasoning & Scoring: The LLM agent analyzes the caption, history, and retrieved context to output a binary prediction, anomaly probability ($p_t$), and reasoning.
   • Dynamic Question Generation: If the anomaly probability is below a threshold ($\theta = 0.7$), the LLM generates a targeted clarifying question ($q_{t}$) to resolve ambiguity (e.g., "Is the person being physically restrained?").
   • Focused Response: The VLM answers the specific question using a focused prompt, and the history is updated. This loop repeats until confidence is high or a max turn limit (2 turns) is reached.
3. Vector Memory Module:
   • To handle long-term dependencies without exceeding context window limits, QVAD uses a semantic vector memory.
   • It stores scene descriptions and predictions as embeddings. During inference, it retrieves relevant historical context via semantic search (RAG-like mechanism) to inform the current decision, allowing the model to correlate events across time.
4. Post-Processing:
   • Frame-level scores are aggregated from overlapping windows using max-pooling.
   • Multi-scale Gaussian smoothing is applied to ensure temporal consistency and robustness.

3. Key Contributions

• Dynamic Question Refinement: Introduces an inference-time mechanism where an LLM agent adaptively updates VLM prompts based on visual context and uncertainty, maximizing information retrieval without parameter updates.
• Efficient Deployment (Democratization): Demonstrates that high-performance VAD can be achieved using compact models (e.g., Qwen-1.7B/2B/4B) rather than massive foundation models.
  • Memory Footprint: Achieves state-of-the-art (SOTA) performance with a memory footprint as low as 4.20 GB, enabling deployment on edge devices like the NVIDIA Jetson Orin Nano.
• Generalization to Open-Set Scenarios: Successfully generalizes to complex, context-dependent anomalies in single-scene environments (ComplexVAD) and open-set datasets (UBNormal), outperforming static prompting methods.

4. Experimental Results

The framework was evaluated on four benchmarks: UCF-Crime, XD-Violence, UBNormal, and ComplexVAD.

• Performance:
  • UCF-Crime: Achieved 84.28% AUC, matching SOTA methods like PANDA and VADTree while using significantly fewer parameters.
  • XD-Violence: Achieved 68.53% AP.
  • UBNormal: Achieved 79.6% AUC (SOTA), significantly outperforming PANDA (75.78%) and LAVAD (64.23%).
  • ComplexVAD: Achieved 68.02% AUC, demonstrating superior ability to handle subtle, long-term anomalies compared to VADTree (54.20%).
• Efficiency:
  • Parameters: Uses ~8B parameters total (4B LLM + 4B VLM) compared to competitors requiring 22B–80B+ or massive API calls.
  • Hardware: Runs on a single NVIDIA RTX 5090 at ~6 FPS. The memory-optimized pipeline allows it to run on consumer-grade edge hardware.
• Ablation Studies:
  • Turns: Performance jumps significantly from Turn 0 (static) to Turn 2, proving the value of iterative reasoning.
  • Components: Removing Vector Memory drops AUC by ~2.6%, highlighting the importance of long-term context retrieval.
  • Model Size: Even the 1.7B/2B configuration retains strong performance (73.88% AUC on UBNormal) with only 4.20 GB VRAM usage.

5. Significance

• Shift in Paradigm: QVAD shifts the focus from "scaling up model size" to "scaling up interaction quality." It proves that smaller models can outperform larger ones if guided by an intelligent, iterative questioning strategy.
• Edge AI Viability: By drastically reducing memory requirements and computational cost, QVAD makes advanced, training-free video anomaly detection feasible for real-world, resource-constrained edge devices (e.g., surveillance cameras, drones), a capability previously limited to cloud servers.
• Interpretability: The framework provides explicit reasoning and evidence for its decisions through the dialogue history, enhancing trust in automated surveillance systems.

In conclusion, QVAD represents a significant step forward in making robust, training-free video anomaly detection practical for real-world deployment by leveraging agentic reasoning to unlock the latent capabilities of lightweight foundation models.