Towards Video Anomaly Detection from Event Streams: A Baseline and Benchmark Datasets

This paper addresses the lack of resources for event-based video anomaly detection by introducing synchronized benchmark datasets and proposing the EWAD framework, which leverages event-centric spatiotemporal modeling and RGB-to-event knowledge distillation to significantly outperform existing approaches.

The Gist

Imagine you are trying to spot a pickpocket in a crowded train station.

The Old Way (RGB Cameras):
Traditional security cameras are like a person taking a photo every 1/30th of a second. They capture everything: the static walls, the people standing still, the posters on the wall. To find the pickpocket, a computer has to look at thousands of these photos, trying to ignore the boring stuff and find the one tiny moment where someone's hand moves strangely. It's like trying to find a needle in a haystack, but the haystack is made of hay, and the computer has to look at every single blade of hay in every single photo. It's slow, data-heavy, and often misses fast movements.

The New Way (Event Cameras):
Now, imagine a different kind of camera. This camera doesn't take photos. Instead, it's like a swarm of millions of tiny, hyper-sensitive ants. Each ant only screams out when it sees something change. If a wall is still, the ants are silent. If a person walks by, the ants covering that person start screaming "I'm moving! I'm moving!"

This is Event-Based Vision. It ignores the boring, static background and only records motion. It's incredibly fast, uses very little data, and is great at privacy (since it doesn't record clear faces, just moving shapes).

The Problem

The researchers in this paper realized that while this "ant camera" is perfect for spotting weird movements (anomalies), nobody had built a proper training school for it.

1. No Data: There were no big libraries of "event camera" videos showing crimes or accidents to teach computers what to look for.
2. No Rules: The old computer brains (AI models) were trained to look at photos, not at streams of "screaming ants." They didn't know how to process this new language.

The Solution: EWAD

The team built a new system called EWAD (Event-centric Video Anomaly Detection). Think of it as a three-step training program for a security guard who only speaks "Event."

1. The Smart Filter (Dynamic Sampling)

• The Analogy: Imagine you are reading a book, but you only want to read the exciting chapters. A normal reader reads every page. A smart reader skips the boring descriptions of the weather and jumps straight to the fight scenes.
• How it works: Since event cameras produce a "burst" of data when something fast happens (like a fight) and very little when things are calm, EWAD doesn't waste time looking at the calm parts. It automatically zooms in on the "busy" moments where the data is dense, ensuring it doesn't miss the action while ignoring the silence.

2. The Time-Adjusting Lens (Density-Modulated Attention)

• The Analogy: Imagine you are listening to a song. When the music is slow and quiet, you listen carefully to every note. When the music is a fast drum solo, you focus on the rhythm and the gaps between the beats.
• How it works: In event data, time isn't uniform. Sometimes there are thousands of events in a split second; other times, there are none. EWAD changes how it "perceives" time based on how busy the scene is. It stretches or compresses its attention span to make sure it understands the relationship between events, even if they happen very far apart in time.

3. The Mentor System (Knowledge Distillation)

• The Analogy: Imagine a student (the Event model) who has never seen a real crime scene. They are learning from a teacher (an RGB model) who has watched thousands of crime movies but speaks a different language.
• How it works: The Event model is "blind" to some things because it only sees motion. So, the researchers let the "Teacher" (a standard video AI) watch the original video and tell the "Student" (the Event AI): "Hey, that movement looks suspicious, even if you can't see the face." The Student learns the concepts of what is normal and what is weird from the Teacher, without needing the Teacher to be there during the actual security check.

The Results

The researchers created a massive new library of simulated event videos (since real event cameras are expensive and rare) covering crimes like fights, falls, and shootings.

They tested EWAD on these datasets and found:

• It works better: It caught anomalies much more accurately than previous methods that tried to force old cameras to work with new data.
• It's efficient: Because it ignores the static background, it's much faster and lighter.
• It can point to the culprit: Not only did it say "Something bad is happening," but it could also draw a box around where it was happening, even though it only had motion data to work with.

Why This Matters

This paper is like laying the foundation for a new type of security system. It proves that we don't need heavy, privacy-invading cameras to catch bad guys. We can use these "motion-only" cameras that are faster, cheaper, and better at spotting the weird stuff, provided we teach our computers how to speak their language.

In short: They built the first dictionary and the first school for a new kind of camera that only sees movement, and the students are already getting top grades.

Technical Summary

1. Problem Statement

Video Anomaly Detection (VAD) is critical for public safety but faces significant limitations when relying on traditional RGB cameras. RGB videos suffer from fixed frame rates, high data redundancy, and delayed dynamic perception, making them less effective in high-density or rapidly changing environments.

While event cameras offer a promising alternative due to their asynchronous, high-temporal-resolution, and low-redundancy nature (capturing only brightness changes), their application to VAD has been hindered by two main challenges:

1. Data Scarcity: There is a lack of dedicated, large-scale event-stream VAD datasets.
2. Modeling Gap: Existing VAD models designed for synchronous RGB frames cannot effectively handle the asynchronous, sparse, and non-uniform nature of event data. Furthermore, current event-based approaches often fail to leverage the powerful semantic priors available in pre-trained Vision-Language Models (VLMs).

2. Methodology: The EWAD Framework

The authors propose EWAD (Event-centric Video Anomaly Detection), a unified framework designed to bridge the gap between event streams and high-level anomaly understanding. The framework operates in two phases: Training (using paired RGB and Event data) and Inference (using Event data only).

A. Dataset Construction

To address the data scarcity, the authors constructed the first large-scale event-based VAD benchmarks by synchronizing event streams with three existing RGB datasets:

• UCF-Crime
• CCTV-Fight
• UBnormal
• Generation Method: They used the v2e simulator to convert RGB videos into realistic event streams, preserving spatiotemporal dynamics and anomaly labels. They also introduced an Adaptive Event Frame Generation strategy to balance temporal resolution and data sparsity, preventing artifacts like ghosting or blur.

B. Core Components of EWAD

The framework consists of three key innovations:

1. Event-Density Aware Dynamic Sampling (EDS):

   • Problem: Uniform sampling wastes resources on static background frames and misses critical dynamic bursts where anomalies occur.
   • Solution: An adaptive sampling strategy that calculates event density per frame. It employs a dual-interval nucleus sampling approach:
     • Frames are sorted by density.
     • High-density frames (likely containing anomalies) are prioritized.
     • Low-density frames are retained to preserve semantic context (background).
   • Benefit: Ensures the model focuses on informative segments while maintaining temporal coverage.

2. Event-Modulated Distance-Decay Attention (EDA):

   • Problem: Event streams are non-uniform in time; standard attention mechanisms do not account for varying event densities or inter-event intervals.
   • Solution: A plug-and-play attention mechanism that jointly encodes temporal distance and event density.
   • Mechanism: The attention weight between tokens is modulated by the density of the target token. Denser tokens (indicating higher motion/activity) receive higher weights, allowing the model to dynamically adjust its "sense of time" and capture long-range dependencies in sparse sequences.

3. RGB-to-Event Knowledge Distillation (KD):

   • Problem: Event data is sparse and lacks the rich texture of RGB, making it difficult to learn high-level semantics from weak supervision (video-level labels) alone.
   • Solution: A cross-modal distillation strategy where a pre-trained RGB-based VLM (Teacher) guides the Event-based model (Student).
   • Mechanism:
     • Binary Logit Distillation: Aligns anomaly confidence scores between the teacher and student.
     • Multi-class Logit Distillation: Aligns categorical predictions using logit standardization to transfer inter-class relationships and feature space structures.
   • Note: This is applied only during training. During inference, the model runs solely on event streams.

3. Key Contributions

1. First Large-Scale Benchmarks: Introduced the first publicly available, large-scale event-based VAD benchmarks (UCF-Crime, CCTV-Fight, UBnormal converted to events) covering diverse real-world scenarios.
2. Novel Baseline (EWAD): Proposed a specialized framework featuring:
   • EDS: Dynamic sampling tailored to event density.
   • EDA: Density-modulated temporal attention for sparse sequences.
   • KD: Cross-modal knowledge transfer from RGB to Event domains.
3. Spatial Localization: Demonstrated the feasibility of using event streams for training-free spatial anomaly localization (generating heatmaps/bounding boxes) without RGB input.
4. State-of-the-Art Performance: Established a strong baseline that significantly outperforms existing SNN-based and ViT-based event methods.

4. Experimental Results

The method was evaluated on the three constructed benchmarks.

• Temporal Anomaly Detection:

  • EWAD achieved State-of-the-Art (SOTA) performance across all datasets.
  • On UCF-Crime, EWAD achieved an AUC of 76.55%, surpassing the previous best event-based method (MSF, 65.01%) by 11.54% and outperforming adapted RGB methods (e.g., VadCLIP at 73.01%).
  • Ablation studies confirmed that each component (EDS, EDA, and KD) contributes incrementally to performance, with the full model showing a 3.54% absolute gain over the baseline.

• Spatial Anomaly Localization:

  • Using a training-free approach, EWAD achieved a TIoU of 13.28% on UCF-Crime.
  • While slightly lower than advanced RGB methods (e.g., VadCLIP at 22.05%), it significantly outperformed earlier RGB-only methods (e.g., C3D at 7.20%), proving that event data contains sufficient cues for fine-grained localization.

• Real-World Validation:

  • Tested on real-world EventVOT data (normal clips). The model correctly assigned low anomaly scores to 17/18 samples, confirming it learns motion semantics rather than simulation artifacts.

5. Significance

This work represents a foundational step in establishing Event-based VAD as a unified research direction.

• Paradigm Shift: It moves beyond low-level event tasks (like optical flow) to high-level semantic understanding (anomaly detection).
• Efficiency & Privacy: By leveraging the inherent sparsity and privacy-preserving nature of event cameras, the proposed method offers a viable solution for resource-constrained and privacy-sensitive surveillance applications.
• Future Direction: The paper highlights the potential of multimodal fusion (RGB + Event) and the need for real-world event datasets, setting the stage for future research in event-centric foundation models.