Technical Summary: NSP-ML for Abnormal Behavior Recognition

Problem Statement

Abnormal behavior recognition in campus environments is critical for intelligent surveillance and safety management. While existing methods have achieved progress, they predominantly rely on visual information. This reliance creates a significant limitation: visual representations alone often fail to distinguish between behaviors that appear similar but carry different meanings based on contextual constraints (e.g., time, location, and expected activity patterns). In campus settings, an action like "running" may be normal on a playground but abnormal in a laboratory corridor. Current multimodal approaches often treat textual inputs as static labels or generic descriptions, failing to explicitly encode situational norms or the prior knowledge associated with specific spatiotemporal conditions. Consequently, these models struggle with context-dependent abnormal behaviors where the definition of "abnormality" relies on deviation from expected patterns rather than visual appearance alone.

Methodology

The paper proposes NSP-ML (Normality-Guided and Spatiotemporal Prior-Aware Multimodal Learning), a framework designed to fuse visual representations with two specific types of contextual textual knowledge: normality-guided logs and spatiotemporal prior-aware logs.

1. Contextual Text Construction

Instead of using static class labels, the framework constructs a context-conditioned behavioral hypothesis ($H_{i,c}$) for each video clip $i$ and candidate class $c$. This hypothesis is formed by concatenating:

• Behavior Prototype ($B_c$): A short semantic description of the behavior.
• Normality Log ($N_i$): Text describing expected behaviors under the specific campus conditions.
• Spatiotemporal Prior Log ($P_i$): Text summarizing risk levels, activity patterns, and sensitivity related to the specific location and time.

2. Spatiotemporal Prior Embedding

Beyond textual logs, the model encodes structured signals directly into the text branch:

• Temporal Embedding: Timestamps are converted into periodic embeddings using sine/cosine functions.
• Geographic Embedding: Locations are encoded based on coordinates, zone density, and management sensitivity levels.
• These structured embeddings are projected into the text feature space to align with the textual prior descriptions.

3. Prior-Aware Multi-Head Attention

The framework introduces a Prior-Aware Multi-Head Attention mechanism. Unlike standard attention, this module injects the projected context vector ($\bar{C}_i$) into the query vectors. This allows the model to perform token-level reasoning guided by spatiotemporal priors, enhancing the interaction between the visual evidence and the contextual hypotheses.

4. Gated Contextual Fusion

A lightweight gating mechanism adaptively balances the contributions of the behavior prototype, normality log, and prior log. This allows the model to dynamically determine how much weight to assign to normality expectations versus prior constraints for each specific classification decision.

5. Optimization Objectives

The framework is trained using a composite loss function ($L$) comprising three components:

• Contextual Classification Loss ($L_{cls}$): Standard cross-entropy loss for matching visual representations to context-conditioned hypotheses.
• Symmetric Contrastive Loss ($L_{con}$): A vision-language alignment objective to ensure visual features match their corresponding contextual textual hypotheses.
• Counterfactual Context Loss ($L_{cf}$): A novel loss that introduces mismatched hypotheses (e.g., swapping the normality log of one scene with another) to explicitly teach the model to distinguish correct scene logic from incorrect logic, thereby strengthening contextual discrimination.

Key Contributions

1. Novel Multimodal Framework: The study introduces NSP-ML, which fuses visual data with normality-guided logs and spatiotemporal prior-aware logs. This integrated representation enables a more context-sensitive interpretation of subtle abnormal behaviors by leveraging the synergy between visual evidence and situational expectations.
2. Explicit Modeling of Contextual Priors: Unlike previous approaches that treat text as static labels, this work models abnormal behaviors in relation to their deviation from contextually expected patterns. The introduction of structured spatiotemporal embeddings and prior-aware attention mechanisms strengthens cross-modal interaction.
3. Robust Performance and Generalizability: The framework demonstrates effectiveness across multiple datasets. It achieves state-of-the-art results on the campus-specific CABRH8 dataset and maintains strong performance on general action recognition benchmarks (UCF-101 and HMDB-51), confirming its robustness beyond specific campus scenarios.

Experimental Results

Experiments were conducted primarily on the CABRH8 dataset (5,950 clips, 8 categories of campus abnormal behaviors), with additional validation on UCF-101 and HMDB-51.

• Performance on CABRH8: The proposed NSP-ML-L variant (using a ViT-L/14 backbone) achieved a Top-1 accuracy of 81.52% and a Top-5 accuracy of 99.45%. This outperforms existing visual-only methods (e.g., TimeSformer, Swin-B) and static text-guided approaches (e.g., CECL-L, which achieved 80.26% Top-1).
• Efficiency: Despite higher accuracy, NSP-ML-L maintains a favorable efficiency profile, requiring 295 GFLOPs and running at 32 FPS, which is significantly more efficient than high-cost competitors like HCMT (992 GFLOPs).
• Generalization: On UCF-101, NSP-ML-L achieved 98.68% accuracy, and on HMDB-51, it achieved 73.32%, outperforming several state-of-the-art baselines and demonstrating strong transferability to general action recognition tasks.
• Ablation Studies: Experiments confirmed that:
  • Combining normality logs and spatiotemporal priors yields better results than using either alone or using only behavior prototypes.
  • Structured spatiotemporal embeddings (time and location) provide complementary gains over textual priors alone.
  • The counterfactual context loss ($L_{cf}$) is crucial for improving the model's ability to distinguish correct scene logic from incorrect logic.

Significance

The paper claims that incorporating normality guidance and spatiotemporal priors provides an effective pathway for improving context-sensitive abnormal behavior recognition in intelligent campus environments. By shifting from purely visual matching to reasoning about context-conditioned behavioral hypotheses, the NSP-ML framework addresses the ambiguity inherent in dynamic, crowded, and context-dependent scenes. The results suggest that this approach allows models to better distinguish between visually similar actions that differ in abnormality based on situational constraints, offering a more robust solution for intelligent surveillance systems.