Are Multimodal LLMs Ready for Surveillance? A Reality Check on Zero-Shot Anomaly Detection in the Wild

This paper evaluates the viability of zero-shot Multimodal LLMs for real-world video anomaly detection, revealing that while prompt engineering can significantly improve F1-scores, a persistent conservative bias toward the "normal" class severely limits recall, highlighting a critical gap between current MLLM capabilities and the demands of practical surveillance.

The Gist

Imagine you've hired a brilliant, world-traveling detective named MLLM (Multimodal Large Language Model) to watch over a busy city square. This detective is famous for reading books, watching movies, and understanding complex stories. You ask them to sit in front of a security camera and shout out immediately if they see anything weird happening, like a fight, a theft, or someone running away from a crime.

This paper is essentially a reality check on whether this super-smart detective is actually ready for the job.

Here is the breakdown of the study using simple analogies:

1. The Big Idea: From "Movie Critic" to "Security Guard"

For a long time, these AI models were trained to be Movie Critics. They are great at watching a whole movie, understanding the plot, and answering questions like, "Why was the hero sad?" or "What happens next?"

But being a Security Guard is a totally different job.

• The Movie Critic watches a polished, edited film where the camera angles are perfect, the lighting is great, and the story makes sense.
• The Security Guard watches a grainy, shaky, 24/7 live feed where people are just walking around, and the "bad stuff" (anomalies) is rare, subtle, and looks very similar to normal behavior.

The researchers asked: Can our brilliant Movie Critic switch hats and become a reliable Security Guard without any special training?

2. The Problem: The "Overly Cautious" Detective

The researchers tested the AI on two famous surveillance datasets (ShanghaiTech and CHAD). They gave the AI short video clips (1 to 3 seconds long) and asked a simple question: "Is there something weird happening here?"

The Result: The AI was scared to make a mistake.

• It acted like a security guard who is terrified of crying wolf.
• If the AI saw anything that wasn't 100% obviously a crime, it decided, "Nah, that's probably normal. I'll say 'No' just to be safe."
• The Outcome: The AI was extremely precise (when it did say "Yes, it's a crime," it was usually right), but it had zero recall (it missed almost every actual crime).
• Analogy: Imagine a smoke detector that only goes off if the entire house is on fire. It never gives a false alarm, but it also never warns you about a small kitchen fire until it's too late.

3. The Fix: Giving the Detective a "Wanted Poster"

The researchers realized the AI wasn't "blind"; it just didn't know what to look for. It was too busy trying to be polite and not make mistakes.

So, they changed the instructions (the "Prompt"). Instead of just asking, "Is this weird?" they gave the AI a specific Wanted Poster.

• Old Prompt: "Look at this video. Is anything wrong?"
• New Prompt: "You are looking for shoplifters. Specifically, look for people hiding items in their coats or running away. If you see that, scream 'YES'."

The Result: This simple change was a game-changer.

• By telling the AI exactly what kind of weirdness to look for, it stopped being so cautious.
• On the ShanghaiTech dataset, the AI's ability to catch crimes (F1-score) jumped from a terrible 0.09 (almost useless) to a solid 0.64 (quite good).
• Analogy: It's like telling a guard dog, "Don't just bark at everything; specifically bark if you see a raccoon." Suddenly, the dog starts barking at raccoons instead of ignoring them.

4. The Surprises: More Detail Doesn't Always Help

The researchers tried giving the AI longer, more detailed instructions (like a 5-page manual vs. a 1-sentence note).

• Finding: The medium-length instructions worked best.
• Why? Giving the AI a 5-page manual (too much detail) actually confused it. It got distracted by the extra words. A clear, medium-length "Wanted Poster" was the sweet spot.

5. The Reality Check: High Definition Isn't a Magic Bullet

They also tested the AI on a newer, higher-quality video dataset (CHAD) that looked more like real life.

• Expectation: Better video quality = Better AI performance.
• Reality: The AI still struggled. Even with crystal-clear video, the AI couldn't figure out the context as well as it did on the older, grainier videos.
• Lesson: Just having a better camera doesn't fix the AI's brain. The AI still needs help understanding context (e.g., knowing that running in a park is fine, but running in a bank lobby is suspicious).

The Bottom Line

Are Multimodal LLMs ready for surveillance?
Not quite yet.

• The Good News: They have the intelligence to understand video. If you give them the right instructions (a specific "Wanted Poster"), they can actually do the job.
• The Bad News: Left to their own devices, they are too cautious. They would miss 90% of crimes because they are afraid of making a false alarm.
• The Future: To make this work in the real world, we can't just rely on the AI's "general smarts." We need to build systems that specifically train the AI to be less afraid of false alarms and give it very clear, specific rules about what constitutes a crime in that specific neighborhood.

In short: The AI is a genius student, but it needs a very strict teacher to tell it exactly what to look for, or it will just sit there and do nothing.

Technical Summary

1. Problem Statement

The paper addresses the critical gap between the impressive general video understanding capabilities of Multimodal Large Language Models (MLLMs) and their operational reliability in Video Anomaly Detection (VAD) for real-world surveillance.

• The Challenge: Traditional VAD systems rely on reconstruction errors or pose deviations. MLLMs offer a paradigm shift by treating anomaly detection as a language-guided reasoning task. However, it is unclear if general-purpose MLLMs can function as dependable, zero-shot anomaly detectors in noisy, open-world surveillance environments where anomalies are rare, context-dependent, and subtle.
• The "Decision Gap": Existing benchmarks often use ranking metrics (AUC-ROC) or curated datasets that do not reflect real surveillance conditions (low resolution, occlusions, long temporal contexts). Furthermore, standard evaluation does not force models to commit to actionable decision boundaries (binary classification) required for live security systems.
• Core Question: Can general-purpose MLLMs serve as reliable anomaly decision-makers under open-world uncertainty without extensive task-specific training?

2. Methodology

The authors propose a framework that integrates state-of-the-art MLLMs directly into the VAD pipeline as prompt-conditioned decision modules.

• Task Formulation: VAD is reformulated as a binary classification task under weak temporal supervision. Instead of ranking clips, the model must output a binary decision (0 = Normal, 1 = Anomalous) for a given video clip.
• Datasets: Evaluation was conducted on two benchmarks:
  • ShanghaiTech: A standard surveillance dataset.
  • CHAD (Charlotte Anomaly Dataset): A higher-resolution, more complex real-world dataset.
• Data Processing:
  • Videos are segmented into non-overlapping clips of 1s, 2s, and 3s.
  • Clip-level labels are derived using an "any-anomaly" rule: if a clip contains at least one anomalous frame, the entire clip is labeled as anomalous.
• Prompt Engineering Strategy: The study systematically varies prompt specificity to analyze its impact on decision boundaries:
  • Base Prompts: Human-authored and LLM-generated prompts (ChatGPT, Gemini variants) ranging from Short (concise) to Medium (structured) to Long (comprehensive definitions).
  • Class-Specific Instructions ("+ class"): A critical variable where prompts explicitly define the types of anomalies to look for (e.g., "unsafe," "unexpected," "out of the ordinary") to shift the model's decision boundary.
• Model Selection: Gemini 2.5 Flash Lite was selected as the primary MLLM because it offers native video analysis (processing raw video streams) without requiring manual frame decomposition, preserving temporal continuity better than models that rely on "filmstrip" conversions.

3. Key Contributions

1. Deployment-Oriented Formulation: The paper casts anomaly detection as a prompt-guided binary decision problem, explicitly targeting the real-world requirement for actionable decision boundaries rather than offline ranking.
2. Prompt Design Analysis: A controlled study on how prompt specificity (verbosity vs. conciseness) and the inclusion of class-specific context shape MLLM judgments.
3. Temporal Context Probing: Evaluation of how short clip windows (1s–3s) modulate model sensitivity and stability in surveillance contexts.
4. Reality Check: A comprehensive assessment revealing that while MLLMs have high precision, they suffer from a severe "conservative bias" in zero-shot settings, often failing to detect anomalies without specific guidance.

4. Experimental Results

The evaluation revealed a significant precision-recall trade-off and highlighted the limitations of current MLLMs in surveillance.

• Conservative Bias: In zero-shot settings (without class-specific instructions), models exhibited extreme conservatism. They achieved near-perfect precision (often 100%) but catastrophically low recall (<5%). They defaulted to the "normal" class to avoid false positives, rendering them useless for practical detection.
• Impact of Class-Specific Prompts: Adding explicit class context ("+ class") was the most significant factor in performance improvement.
  • On ShanghaiTech, the peak F1-score jumped from 0.09 (Human base) to 0.64 (GPT instant medium + class).
  • Recall improved dramatically (e.g., from ~4% to ~53% in some configurations), proving the bottleneck was a lack of "categorical confidence" rather than visual recognition.
• Prompt Length: Contrary to intuition, longer, more detailed prompts did not consistently improve performance. "Medium" length prompts often outperformed "Long" ones, suggesting excessive detail introduces semantic noise.
• Temporal Window: Increasing clip length from 1s to 3s generally helped on the lower-resolution ShanghaiTech dataset but showed marginal or negative returns on the higher-resolution CHAD dataset, indicating that more temporal context does not always resolve semantic confusion.
• Dataset Complexity: Performance on the more complex CHAD dataset was lower (Peak F1 ~0.48) compared to ShanghaiTech (Peak F1 ~0.64), indicating that higher visual fidelity does not automatically solve the challenges of behavioral interpretation in anomaly detection.

5. Significance and Conclusion

The paper concludes that while MLLMs represent a powerful new paradigm for video understanding, they are not yet ready for reliable, zero-shot deployment in real-world surveillance.

• The "Intent" Gap: The primary failure mode is not a lack of visual "sight" but a lack of "intent." Without explicit guidance on what constitutes an anomaly, MLLMs default to a safe, non-trigger state.
• Future Directions: The authors argue that progress in MLLM-based VAD depends less on increasing data fidelity and more on:
  • Recall-oriented prompting strategies: Explicitly defining anomaly classes to shift decision boundaries.
  • Model calibration: Adjusting models to handle the high-stakes sensitivity required for security (where missing an event is critical).
  • Evaluation Protocols: Moving beyond ranking metrics to decision-boundary metrics that reflect real operational needs.

In summary, the study provides a necessary "reality check," demonstrating that while MLLMs can reason about video, their application in surveillance requires careful prompt engineering and calibration to overcome inherent biases toward normalcy.