Weakly-Supervised Spatiotemporal Anomaly Detection

Technical Summary: Weakly-Supervised Spatiotemporal Anomaly Detection

Problem Statement
The proliferation of surveillance cameras has created a bottleneck where human operators cannot effectively monitor vast amounts of footage to identify rare abnormal events. While automated anomaly detection is a practical solution, existing approaches face limitations. Unsupervised methods, which learn only from normal data, are often dataset-specific because "normal" behavior is context-dependent. Conversely, fully supervised methods require expensive, frame-level or spatiotemporal annotations, which are time-consuming to produce. This paper addresses the challenge of detecting anomalies with only weak, video-level labels (indicating whether a video contains an anomaly or not), while extending the scope from purely temporal detection to spatiotemporal detection to localize the specific regions where anomalies occur.

Methodology
The proposed approach adapts the Multiple Instance Learning (MIL) framework, originally applied to temporal detection by Sultani et al., to the spatiotemporal domain. The core architecture and process are as follows:

• Feature Extraction: Video clips are processed using the I3D network. Specifically, features are extracted from the output of the sixth inception module (4e layer). Unlike previous works that process whole clips or use optical flow, this method focuses on appearance features from the raw video input.
• Spatiotemporal Cuboids: Instead of treating an entire video clip as a single instance, the feature representation (a 4D tensor) is divided into spatial regions. A video clip is split into 49 equally sized spatiotemporal cuboids (2x2 spatial grid). Each cuboid represents an "instance" within a "bag."
• Bag Definition:
  • Positive Bag: An anomalous video clip. The assumption is that at least one spatiotemporal instance (cuboid) within this clip contains the anomaly.
  • Negative Bag: A normal video clip. The assumption is that none of the instances within this clip contain an anomaly.
• Classification and Loss: Each spatiotemporal cuboid is passed through a classifier network consisting of a 3D average pooling layer, five fully connected layers (with batch normalization, ReLU, and dropout), and a final Sigmoid activation to output an anomaly score between 0 and 1.
• Training Objective: The model is trained using a Multiple Instance Ranking (MIL) loss. The objective function enforces that the maximum anomaly score among the instances in a positive bag must be greater than the maximum anomaly score in a negative bag. This allows the network to learn to identify the specific spatiotemporal region responsible for the anomaly without explicit region-level supervision during training.

Key Contributions

1. Spatiotemporal Extension: The paper extends the weakly supervised temporal anomaly detection paradigm to include spatial localization. By dividing feature maps into spatiotemporal cuboids, the method attempts to pinpoint where in the frame an anomaly occurs, not just when.
2. Weakly Supervised Framework: The approach demonstrates that spatiotemporal detection is feasible using only video-level labels, avoiding the need for costly bounding box or tube annotations during the training phase.
3. Architecture Adaptation: The work modifies the standard MIL pipeline by utilizing I3D features from the sixth inception block and applying spatial division to create instance-level representations from video clips.

Results
The method was evaluated on the UCF Crime2Local dataset (a subset of UCF-Crime with spatiotemporal annotations) and the full UCF-Crime dataset.

• UCF Crime2Local: The model achieved an Area Under the Curve (AUC) of 68%. This result outperformed the weakly supervised "Video Segment" baseline from Landi et al. (56.12%), which also lacked spatiotemporal annotations. However, it remained lower than Landi et al.'s fully supervised "Oracle Tube" approach (74.73%), which utilized ground-truth bounding boxes during training.
• Full UCF-Crime: On the larger dataset, the model achieved an AUC of 63%, compared to approximately 82% for the state-of-the-art method by [3].
• Qualitative Analysis: Visualizations of predicted anomaly scores against ground truth showed that the model correctly identifies the temporal presence of anomalies (low scores for normal frames, high scores for anomalous frames). However, the model often predicts multiple "blobs" as anomalous rather than isolating a single region.

Significance and Claims
The paper claims that incorporating spatiotemporal features into weakly supervised anomaly detection is a promising direction that may offer advantages over purely temporal detection. The authors acknowledge that while their current results do not yet match state-of-the-art fully supervised methods or other strong baselines, the approach successfully demonstrates the feasibility of localizing anomalies without dense annotations.

The authors modestly conclude that the method requires further refinement to reach competitive performance. They identify specific areas for improvement, including:

• Adding regularization to prevent the model from predicting multiple anomalous regions per video.
• Incorporating motion information, potentially through a two-stream approach using optical flow, to complement the appearance-based features currently used.