Technical Summary: Learning Where and When: Patch-Based Spatiotemporal Localization in Weakly Supervised Video Anomaly Detection

1. Problem Statement

Weakly Supervised Video Anomaly Detection (WSVAD) has traditionally focused on temporal localization, identifying when an anomaly occurs within a video sequence using only video-level labels. However, this approach largely neglects spatial localization, failing to determine where within a frame the anomaly manifests.

The authors argue that spatial localization is critical for:

• Interpretability: Allowing human operators to verify alerts and understand the cause of an anomaly.
• Decision Making: Enabling fine-grained actions like tracking suspicious objects or prioritizing resources.
• Robustness: Providing contextual cues to distinguish true anomalies from background clutter, thereby reducing false positives caused by background biases.

The core challenge is learning precise spatial localization signals without explicit bounding-box supervision, as obtaining such annotations is expensive and subjective. Existing methods often rely on global representations or clip-level scores, limiting their ability to capture localized abnormal patterns.

2. Methodology

The paper proposes a patch-based spatiotemporal framework that jointly models the occurrence of anomalies in both space and time under a Multiple Instance Learning (MIL) paradigm.

2.1. Architecture Overview

The input video is partitioned into temporal segments (16 frames each) and processed through a pretrained feature extractor (UniFormer) to generate patch-level spatiotemporal representations. The grid size is set to $7 \times 7$ patches per segment.

The architecture consists of two main branches:

1. Temporal Aggregation: Features are reshaped into patch-wise temporal sequences and processed by a Temporal Transformer to model dependencies for each spatial patch.
2. Spatial Aggregation: The temporally aggregated features are rearranged and passed through a Spatial Transformer to capture relationships among patches within each segment.

These branches feed into two prediction heads:

• Coarse-Grained Branch: Produces segment-level anomaly scores by pooling spatial features.
• Fine-Grained Branch: Generates patch-level anomaly probabilities ($7 \times 7$ grid) for each segment.

2.2. Proximity-Aware Top-k MIL Loss

A key innovation is the Proximity-Aware Top-k selection strategy, which replaces conventional top-k selection based solely on the highest individual scores. The authors observe that anomalous events typically occur over temporally contiguous intervals.

The loss function operates as follows:

• Temporal Selection: Instead of picking the $K$ highest-scoring frames globally, the model uses a sliding window to find a contiguous block of $K$ segments with the highest cumulative anomaly score.
• Spatial Selection: Within these selected segments, the model identifies the most discriminative spatial region by sliding a convolutional window ($z \times z$) over the patch grid and selecting the block with the highest summed response.
• Loss Calculation:
  • Anomalous Videos: The selected contiguous temporal blocks and their corresponding high-response spatial patches are treated as positive instances for Binary Cross-Entropy (BCE) loss.
  • Normal Videos: All patches are treated as negative instances.
  • Regularization: $L1$-norm is applied to anomalous sets to encourage sparsity, while $L2$-norm is applied to normal sets to ensure scores remain close to zero.

3. Key Contributions

The paper outlines three primary contributions:

1. Novel Architecture: A patch-based weakly supervised spatiotemporal architecture that produces anomaly scores for individual patches, enabling fine-grained spatial localization without bounding-box supervision during training.
2. Proximity-Aware MIL Loss: A new neighbor-aware top-k loss function where selected instances are formed via clustering based on temporal and spatial proximity, rather than purely by highest scores.
3. New Annotations: The release of frame-level bounding-box annotations for the test sets of two widely used datasets, XD-Violence and UBnormal, facilitating standardized evaluation of spatial localization in WSVAD.

4. Experimental Results

The method was evaluated on four benchmarks: UCF-Crime, ShanghaiTech, XD-Violence, and UBnormal.

4.1. Metrics

The authors utilize two complementary metrics:

• Temporal Intersection over Union (TIoU): Measures overlap between predicted and ground-truth regions.
• Track-Based/Region-Based Detection Rate (TBDR/RBDR): A more comprehensive metric that accounts for false positives and evaluates spatiotemporal consistency (tracks) rather than just frame-level overlap.

4.2. Performance

• TIoU: The proposed method ("OURS") outperforms all state-of-the-art baselines (including Deep-MIL, VADCLIP, ST-Prompt, and LSTM-MIL) across all four datasets. For example, on UCF-Crime, it achieves 40.51%, surpassing the previous best (LSTM-MIL at 32.91%) by nearly 7.6%.
• TBDR/RBDR: The method achieves the highest scores in both metrics across all datasets. On UCF-Crime, it reaches 77.20% TBDR and 69.83% RBDR. On UBnormal, it achieves 57.23% TBDR and 33.29% RBDR, demonstrating strong generalization to subtle anomalies.
• Zero-Shot Performance: In a zero-shot setting (training without specific anomaly classes), the proposed model showed the smallest performance drop (3.08% TBDR drop) compared to baselines, indicating superior generalization.
• Efficiency: The model requires approximately 44.23 ms per 60-frame input, slightly higher than baselines due to refinement modules but still suitable for real-time inference.

4.3. Ablation Studies

• Component Removal: Removing the spatial transformer caused the largest performance drop (19.15% TBDR), highlighting the necessity of spatial attention. Removing the temporal transformer also significantly reduced performance.
• Loss Function: Replacing standard MIL with Proximity MIL improved AUC across various existing frameworks (Deep-MIL, URDMU, VADCLIP), confirming the loss function's generalizability.
• Hyperparameters: A $7 \times 7$ grid with a spatial window size of $z=2$ yielded optimal results.

5. Significance and Claims

The paper claims to address a fundamental limitation in WSVAD by shifting from purely temporal localization to joint spatiotemporal detection. By reformulating MIL at the spatial patch level, the authors enable region-level scoring without bounding-box supervision.

The significance of the work lies in:

• Improved Interpretability: Providing explicit spatial maps of anomalies rather than just temporal timestamps.
• Reduced Bias: Explicitly modeling spatial evidence helps mitigate the background biases found in previous methods that relied on global representations.
• Community Resource: By releasing bounding-box annotations for XD-Violence and UBnormal, the authors aim to standardize the evaluation of spatial localization, a task previously hindered by the lack of ground truth in these datasets.

The authors conclude that their framework achieves substantial gains in localization accuracy while maintaining practical computational efficiency, paving the way for more reliable deployment in surveillance and safety-critical applications.