No Need For Real Anomaly: MLLM Empowered Zero-Shot Video Anomaly Detection

This paper introduces LAVIDA, a zero-shot video anomaly detection framework that leverages a Multimodal Large Language Model and an Anomaly Exposure Sampler to train exclusively on pseudo-anomalies, achieving state-of-the-art performance across multiple benchmarks without requiring real anomaly data.

The Gist

Imagine you are hiring a security guard to watch over a thousand different places: a busy park, a quiet library, a chaotic construction site, and a high-speed racetrack.

The Old Problem:
Traditionally, to train this guard, you'd have to show them thousands of videos of specific bad things happening in specific places.

• "Here is a fight in a park."
• "Here is a car crash on a racetrack."
• "Here is a robbery in a bank."

If a new type of trouble happens—say, someone starts juggling chainsaws in the library—the guard has no idea what to do because they've never seen "chainsaw juggling" before. They are stuck in a "closed world" where they only know what they were explicitly taught.

The New Solution (LAVIDA):
The paper introduces LAVIDA, a new kind of AI security guard. Instead of memorizing specific bad events, LAVIDA learns the concept of "badness" and uses a super-smart brain (a Multimodal Large Language Model, or MLLM) to understand context.

Here is how it works, broken down with simple analogies:

1. The "Fake Crime" Training Camp (Anomaly Exposure Sampler)

You can't train a guard on real crimes because real crimes are rare and dangerous to film. So, LAVIDA uses a clever trick.

• The Analogy: Imagine you have a photo album of normal animals (parrots, dogs, elephants). To teach the guard what "weird" looks like, you take a picture of a parrot and tell the guard, "In this scene, the parrot is the intruder." Then you take a picture of a car and say, "Here, the car is the intruder."
• How it works: The AI takes normal images from the internet (like segmentation datasets) and randomly labels different objects as "suspicious." It creates thousands of "fake anomalies." This teaches the AI to look for anything that doesn't fit the current scene, without ever needing a single video of a real crime.

2. The "Super-Brain" Translator (MLLM Integration)

LAVIDA connects a video camera to a super-smart language brain (like a very advanced version of ChatGPT that can see).

• The Analogy: A normal security camera just sees pixels moving. LAVIDA's brain can read a prompt like, "Find the thing that is hurting someone." It understands that "hurting" means different things in different places. In a kitchen, it might be a knife fight; in a street, it might be a car hitting a pedestrian.
• The Magic: Because this brain understands language and deep meaning, it can generalize. If it sees a "riot" in a video, it understands the concept of chaos, even if it's never seen a riot before. It doesn't just match patterns; it understands the story.

3. The "Noise-Canceling" Filter (Token Compression)

Videos are huge. A 10-second clip has thousands of pixels, most of which are just boring background (sky, walls, grass). Processing all of that is slow and expensive.

• The Analogy: Imagine trying to find a needle in a haystack. The old way was to look at every single piece of hay. LAVIDA uses a "Reverse Attention" filter. It quickly identifies the "hay" (the boring background) and ignores it. It focuses only on the "needle" (the weird, moving, suspicious stuff).
• The Result: It throws away 80% of the visual data that doesn't matter, making the AI faster and cheaper to run, while keeping its eyes glued to the interesting parts.

4. The "Zoom-In" Lens (Multi-Level Mask Decoder)

Finally, LAVIDA doesn't just say, "Something is wrong." It tells you exactly where.

• The Analogy: It's like a detective who can point to a specific person in a crowd and say, "That guy in the blue shirt is the problem," rather than just pointing at the whole street. It can highlight the exact pixels of the anomaly on the screen.

Why is this a Big Deal?

• Zero-Shot: It works on scenarios it has never seen before. You don't need to retrain it for every new city or new type of crime.
• Open-World: It can detect any anomaly, from a falling person to an explosion to a robbery, as long as you can describe it in words.
• No Real Data Needed: It was trained entirely on "fake" anomalies made from normal pictures. This solves the huge problem of not having enough real crime data to train AI.

In Summary:
LAVIDA is like a security guard who has read every book on human behavior and watched every movie ever made. Instead of memorizing a list of "banned actions," it understands the meaning of a scene. If something looks weird, feels wrong, or breaks the story of the video, it spots it immediately, even if it's a situation the guard has never encountered in real life.

Technical Summary

1. Problem Statement

Video Anomaly Detection (VAD) faces three critical challenges in open-world scenarios:

• Data Scarcity and Diversity: Real-world anomaly data is rare, spatio-temporally sparse, and difficult to collect. Existing datasets are limited to specific scenarios (e.g., street surveillance) and anomaly types, preventing models from generalizing to unseen environments or novel anomalies.
• Semantic Understanding Gap: Traditional methods rely on statistical deviations or closed-set classifications. They lack the deep semantic understanding required to interpret context-dependent anomalies (e.g., distinguishing "running" as normal in a gym vs. anomalous in a library).
• Computational Inefficiency: Anomalies often occupy small spatial/temporal regions. Processing full video sequences with high-resolution tokens using Multimodal Large Language Models (MLLMs) incurs prohibitive computational costs and memory usage.

Current solutions either require extensive training on specific anomaly data (failing in zero-shot settings) or rely on frame-wise text generation from MLLMs, which is computationally expensive and lacks pixel-level localization.

2. Methodology: LAVIDA Framework

The authors propose LAVIDA (LLM-Assisted VIdeo Anomaly Detection Approach), an end-to-end zero-shot framework that trains solely on pseudo-anomaly data derived from external segmentation datasets, without using any real VAD data.

The framework consists of five key components:

A. Anomaly Exposure Sampler

To overcome the lack of real anomaly data, the authors transform standard semantic segmentation datasets (e.g., containing objects like cars, dogs, parrots) into a pseudo-anomaly training set.

• Mechanism: For a given video clip, the model is presented with a mix of "genuine" object categories (from the clip) and "irrelevant" categories (sampled from other clips).
• Labeling: Samples are randomly labeled as "normal" (containing only irrelevant categories) or "anomaly" (containing a mix of genuine and irrelevant categories).
• Goal: This forces the model to learn to distinguish relevant semantic anomalies from background noise without ever seeing a real "fight" or "explosion" during training.

B. Visual Token Compression (Reverse Attention)

To address the sparsity of anomalies and reduce MLLM computational costs:

• Background Identification: The model calculates local density in the KNN neighborhood of visual tokens. High-density tokens are identified as background.
• Reverse Attention: A reverse attention mechanism is applied where tokens are assigned to their nearest background token. Features that are dissimilar to the background (i.e., potential anomalies) are highlighted and aggregated.
• Result: This compresses the visual token sequence, retaining only anomaly-relevant features while discarding redundant background information, significantly lowering GPU memory usage.

C. Semantic Feature Extraction (MLLM)

• Prompt Engineering: The model uses a Multi-modal Large Language Model (MLLM) with a special <SEG> token.
• Input: Text prompts describe potential anomaly categories (e.g., "Find the anomaly... types may contain fight, explosion").
• Output: The MLLM generates embeddings for the <SEG> token, which encapsulates deep semantic understanding of the anomaly context, bridging the gap between visual input and textual definitions.

D. Multi-Scale Semantic Projector

To bridge the gap between video-level semantic understanding and frame/pixel-level detection:

• Architecture: A Q-Former-like two-way transformer architecture.
• Function: It fuses the video-level semantic features (from the MLLM) with frame-level visual features (from the vision backbone) using cross-attention.
• Output: It generates learnable query tokens that contain both global context and local details, projecting them into the latent space of the decoder.

E. Multi-Level Mask Decoder

• Base: Initialized from SAM2 (Segment Anything Model 2).
• Function: Takes the projected features and visual tokens to output:
  1. Frame-level scores: Confidence that an anomaly exists in the frame.
  2. Pixel-level masks: Precise spatial localization of the anomaly.

3. Key Contributions

1. Zero-Shot Training Strategy: The first end-to-end VAD framework trained exclusively on pseudo-anomalies synthesized from segmentation datasets, eliminating the need for real anomaly data.
2. Anomaly Exposure Sampler: A novel data augmentation strategy that repurposes diverse semantic segmentation targets to simulate arbitrary anomaly types, enhancing generalization to unseen scenarios.
3. Token Compression via Reverse Attention: A training-free method to compress visual tokens by filtering out background noise, enabling efficient MLLM processing for high-resolution video.
4. Dual-Granularity Detection: The ability to perform both frame-level (temporal) and pixel-level (spatial) anomaly detection in a zero-shot setting, leveraging the semantic power of MLLMs.

4. Experimental Results

The model was evaluated on four benchmark datasets (UBnormal, ShanghaiTech, UCF-Crime, XD-Violence) and one pixel-level dataset (UCSD Ped2) under zero-shot conditions.

• Frame-Level Performance:
  • UBnormal: 76.45% AUC (SOTA among zero-shot/few-shot).
  • ShanghaiTech: 85.28% AUC.
  • UCF-Crime: 82.18% AUC (Outperforms unsupervised methods).
  • XD-Violence: 90.62% AP (Surpasses weakly-supervised methods).
• Pixel-Level Performance:
  • UCSD Ped2: 87.68% AUC, representing a 12.57% improvement over the previous state-of-the-art.
• Efficiency: The token compression module reduced GPU memory usage to 54.1% of the baseline while maintaining or slightly improving performance.

5. Significance

• Paradigm Shift: LAVIDA challenges the conventional reliance on collecting massive, labeled anomaly datasets. It demonstrates that generalizable anomaly detection can be achieved by teaching models to understand "what is anomalous" semantically rather than memorizing specific anomaly patterns.
• Open-World Applicability: By leveraging the open-world knowledge of MLLMs, the system can detect arbitrary anomaly types (e.g., "intentional injury," "explosion") in diverse, unseen scenarios without retraining.
• Practical Deployment: The reduction in computational cost via token compression makes MLLM-based VAD feasible for real-time applications, addressing a major bottleneck in deploying large models for video analysis.

In summary, LAVIDA successfully bridges the gap between the semantic reasoning capabilities of Large Language Models and the spatial-temporal requirements of video anomaly detection, achieving state-of-the-art zero-shot performance without requiring real anomaly data for training.