Action Hints: Semantic Typicality and Context Uniqueness for Generalizable Skeleton-based Video Anomaly Detection

1. Problem Statement

Zero-Shot Video Anomaly Detection (ZS-VAD) aims to temporally localize abnormal events in surveillance videos without using any training data from the target domain. This is critical for scenarios involving data privacy or new surveillance deployments where target data is unavailable.

• Current Limitations:
  • Frame/Object-based methods: Suffer from domain shifts in background and human appearance when moving to new scenes.
  • Existing Skeleton-based methods: While they eliminate background/appearance disparities, they rely on low-level skeleton representations (e.g., coordinate prediction, reconstruction) learned via self-supervision. They lack semantic understanding, making them unable to distinguish novel anomalies that resemble normal patterns.
  • Domain-Limited Boundaries: Existing methods blindly rely on normality boundaries defined by the source training data, leading to the misclassification of unseen normal events as anomalies in new scenes.

2. Methodology

The authors propose a novel framework that leverages Action Semantic Typicality and Context Uniqueness to achieve generalizable ZS-VAD. The approach consists of two main modules:

A. Language-Guided Typicality Modeling (Training Phase)

This module aims to learn high-level semantic understanding of actions rather than low-level coordinates.

1. Skeleton-Text Alignment: The model uses an external action recognition dataset (Kinect-400) rather than VAD-specific data. It aligns skeleton snippets with text labels using a skeleton encoder ($E_s$) and a text encoder ($E_t$), projecting skeletons into a language-aligned semantic space.
2. Typicality Knowledge Selection (LLM Distillation):
   • A Large Language Model (LLM) is prompted to identify the 20 most typical normal actions (e.g., walking, reading) and 20 most typical abnormal actions (e.g., fighting, skydiving) from the 400 action classes.
   • High-quality skeleton snippets corresponding to these selected "typical" classes are filtered based on skeleton-text similarity.
3. Typicality Distribution Learning:
   • A Normalizing Flow (NF) is trained on the selected typical normal and abnormal features.
   • The NF models the probability distribution of these behaviors. During inference, the anomaly score ($S^t$) is derived from the negative log-likelihood of a test snippet under the learned "normal" distribution. This allows the model to generalize to new scenes based on semantic priors rather than specific dataset statistics.

B. Test-Time Context Uniqueness Analysis (Inference Phase)

To handle scenarios where actions are not strictly "typical" (e.g., a normal action that is anomalous in a specific context), the model analyzes spatio-temporal context.

1. Context Graph Construction: For each skeleton snippet, two graphs are built:
   • Cross-person Graph ($G_c$): Connects the current person to nearby individuals in space.
   • Self-Inspection Graph ($G_s$): Connects the current person to their own past/future states in time (excluding immediate neighbors to avoid redundancy).
2. Uniqueness Scoring: The model calculates the feature distance between the query snippet and its neighbors in both graphs. Since anomalies are rare and unique, a high distance indicates an anomaly. The uniqueness score ($S^u$) is the maximum of the cross-person and self-inspection distances.
3. Holistic Scoring: The final anomaly score is a normalized combination of the Typicality Score ($S^t$) and the Uniqueness Score ($S^u$):
   $$S_i = \frac{S^t_i - \mu_t}{\sigma_t} + \frac{S^u_i - \mu_u}{\sigma_u}$$
   This integrates semantic priors (what should be normal/abnormal) with contextual evidence (what is actually unique in this scene).

3. Key Contributions

1. Novel Framework: A skeleton-based ZS-VAD framework that moves beyond low-level representation learning to semantic typicality and context uniqueness.
2. Language-Guided Typicality Module: Introduces a method to distill LLM knowledge of typical normal/abnormal behaviors into a skeleton encoder, creating a generalizable semantic space without target domain training.
3. Test-Time Uniqueness Analysis: Proposes a dynamic, scene-adaptive boundary mechanism that analyzes spatio-temporal differences between individuals and time steps to detect context-specific anomalies.
4. State-of-the-Art Performance: Achieves superior results on four large-scale datasets (ShanghaiTech, UBnormal, NWPU, UCF-Crime) covering over 100 unseen surveillance scenes, outperforming both full-shot and other zero-shot methods.

4. Experimental Results

The method was evaluated on four datasets with no training data from the target domains:

• Performance: The proposed method achieved State-of-the-Art (SOTA) results across all datasets.
  • ShanghaiTech: 84.1% AUC (vs. 85.9% full-shot STG-NF, but 0-shot).
  • UBnormal: 74.5% AUC (a significant +5.7% improvement over the zero-shot baseline STG-NF).
  • NWPU: 62.1% AUC (+4.2% over baseline).
  • UCF-Crime: 62.7% AUC (+10.8% over baseline).
• Comparison with Full-Shot: Remarkably, the zero-shot performance of this method is comparable to, and in some cases exceeds, the performance of full-shot methods trained on the target data.
• Efficiency: The model is lightweight (~5.0M parameters), significantly smaller than Large Vision-Language Model (LVLM) based approaches (e.g., LAVAD with 13B+ parameters). The inference speed is dominated by skeleton extraction, not the anomaly scoring logic.

5. Significance and Impact

• Privacy-Preserving Surveillance: The ability to deploy anomaly detection in new environments without collecting or training on local video data addresses critical privacy concerns.
• Generalization: By shifting from "learning normality from data" to "learning normality from semantic concepts and context," the model overcomes the domain gap that plagues traditional VAD systems.
• Practical Deployment: The framework is computationally efficient and does not require complex multi-stage reasoning or massive LLMs during inference, making it suitable for real-world edge deployment.
• Paradigm Shift: The paper demonstrates that skeleton data, when combined with semantic priors and contextual analysis, is a highly effective modality for zero-shot anomaly detection, outperforming frame-based and object-based approaches in cross-scenario settings.