💻 computer science

DIFEM: Key-points Interaction based Feature Extraction Module for Violence Recognition in Videos

This paper proposes DIFEM, a lightweight and efficient feature extraction module that leverages human skeleton key-points to capture dynamic violence indicators like velocity and joint intersections, achieving superior performance on standard datasets compared to state-of-the-art deep learning methods while significantly reducing parameter costs.

Original authors: Himanshu Mittal, Suvramalya Basak, Anjali Gautam

Published 2026-03-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Himanshu Mittal, Suvramalya Basak, Anjali Gautam

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a security guard watching a bank of 50 TV screens, all showing different streets in a busy city. Your job is to spot a fight before it gets out of hand. If you stare at these screens for hours, your eyes will glaze over, and you might miss a punch because you're tired.

This paper introduces a new, super-smart "digital assistant" to help you. Instead of trying to watch every pixel of the video (which is like trying to count every grain of sand on a beach), this assistant only watches the skeleton of the people.

Here is how it works, broken down into simple concepts:

1. The Skeleton Sketch (The "Stick Figure" Trick)

Most modern AI systems try to understand a video by looking at the colors, the clothes, and the background. This is like trying to recognize a person by their outfit. But in a fight, the clothes don't matter as much as the movement.

This paper's method, called DIFEM, uses a tool (OpenPose) to instantly turn every person in the video into a simple stick figure. It ignores the t-shirts and jeans and focuses entirely on the joints: the elbows, knees, wrists, and neck.

2. The Two Clues: "Speed" and "Closeness"

Once the AI has the stick figures, it looks for two specific things that usually happen during a fight, using a simple logic:

Clue A: The "Speedometer" (Temporal Dynamics)
Think of a normal conversation. People move their hands slowly to emphasize a point. Now, think of a fight. Hands fly around, heads duck, and legs kick. The movement is fast and erratic.
The AI calculates the velocity of every joint. If a wrist moves from point A to point B in a split second, the "speedometer" goes off the charts. It's like a car speeding on a highway versus a car parked in a driveway.
Clue B: The "Personal Space" Alarm (Spatial Dynamics)
In a normal crowd, people try to keep their distance. If you walk past someone, you give them space.
In a fight, people invade each other's personal space. The AI checks if a joint (like an elbow or a knee) from Person A is physically overlapping with the body of Person B. It's like a "personal space alarm" that goes off when two stick figures start hugging too aggressively or hitting each other.

3. The "Mathy" Detective (Machine Learning)

Once the AI has these two clues (Speed and Closeness), it doesn't need a super-complex brain to decide if it's a fight. It passes this simple data to a "detective" (a machine learning algorithm like Random Forest or Decision Tree).

The Old Way: Imagine a detective who has read every book in the library, memorized every movie ever made, and has a PhD in physics. This is Deep Learning. It's incredibly powerful but requires a massive library (huge computer power) and takes a long time to think.
The New Way (DIFEM): This detective is like a seasoned police officer who knows the rules of the road. "If the car is speeding and it's swerving into another car, it's an accident." It's simple, fast, and doesn't need a massive library to work.

4. Why This Matters

The authors tested this "stick figure + speed + closeness" method on three different video datasets (including real street fights and hockey games).

The Result: It worked just as well, and sometimes even better, than the massive, expensive Deep Learning systems.
The Benefit: Because it's so simple, it runs incredibly fast. It's like comparing a Ferrari (Deep Learning) to a reliable, fuel-efficient motorcycle (DIFEM). Both can get you to the destination (detecting violence), but the motorcycle uses way less gas (computer power) and is easier to maintain.

The One Catch

The only downside is that this system relies on the "stick figure" tool (OpenPose) working perfectly. If the camera is blurry or the lighting is bad, and the AI can't draw the stick figure correctly, the detective won't have any clues to work with.

In a Nutshell

This paper says: "You don't need a supercomputer to spot a fight. You just need to watch how fast people's elbows are moving and how close they are getting to each other." It's a lightweight, efficient, and surprisingly accurate way to keep our cities safe.

1. Problem Statement

Violence detection in surveillance videos is critical for public safety but poses significant challenges. Existing state-of-the-art (SOTA) methods often rely on deep learning techniques (e.g., 3D CNNs, GCNs, Two-Stream networks) which are:

Computationally Expensive: They require heavy resources and high inference times.
Prone to Overfitting: Violence datasets are relatively small compared to the complexity of deep learning models, leading to poor generalization.
Resource Intensive: They are often unsuitable for real-time deployment on standard hardware.

The authors argue that violent behavior is fundamentally characterized by rapid movement of specific body parts and close spatial proximity (collisions) between individuals. They propose that these inherent properties can be captured using simpler, lightweight statistical methods rather than complex neural networks.

2. Methodology

The proposed approach, DIFEM (Dynamic Interaction Feature Extraction Module), bypasses end-to-end deep learning for feature extraction. Instead, it uses a pipeline of pose estimation followed by statistical feature calculation and traditional machine learning classification.

A. Pipeline Overview

Pose Estimation: The system uses OpenPose to extract 2D skeleton key-points $(x, y)$ for every person in every video frame.
Key-point Selection: From the standard 25 OpenPose key-points, the authors select 11 specific joints most relevant to physical confrontations:
- Wrists, Elbows, Hips, Knees, Ankles, and Neck.
- Specific weights are assigned to these joints (e.g., wrists and hips have a weight of 1.0, elbows/knees 0.8) to prioritize their contribution.
Feature Extraction (DIFEM): The module calculates two types of dynamics:
- Temporal Dynamics (Velocity):
  - Calculates the Euclidean distance a joint moves between consecutive frames ( $t$ and $t+1$ ).
  - Formula: $v_{i,j,t} = \sqrt{w_{i,j,t} \cdot ((x'_{i,j,t+1} - x_{i,j,t})^2 + (y'_{i,j,t+1} - y_{i,j,t})^2)}$
  - Aggregates these velocities across the video to compute Mean, Maximum, and Variance.
- Spatial Dynamics (Joint Overlap):
  - Detects interactions by checking if a joint of Person A falls within the bounding box of Person B.
  - Formula: Counts intersections where $x^{(1)}_j \in [x^{(2)}_{min}, x^{(2)}_{max}]$ and $y^{(1)}_j \in [y^{(2)}_{min}, y^{(2)}_{max}]$ .
  - Aggregates overlap counts to compute Mean and Variance.
Feature Vector: The final input for the classifier is a 5-dimensional vector consisting of: Mean Velocity, Max Velocity, Velocity Variance, Mean Overlap, and Overlap Variance.
Classification: These lightweight features are fed into traditional machine learning classifiers: Random Forest, Decision Tree, AdaBoost, and k-Nearest Neighbor (k-NN).

3. Key Contributions

DIFEM Module: A novel, lightweight feature extraction module that leverages simple heuristics (velocity and spatial overlap) to capture the essence of violence without deep neural networks.
Efficiency: The approach significantly reduces parameter count and computational cost compared to deep learning SOTA methods, making it suitable for resource-constrained environments.
Performance: Demonstrates that simple statistical features derived from skeleton key-points can outperform complex deep learning models on standard violence datasets.
Comprehensive Evaluation: Extensive testing on three diverse datasets (RWF-2000, Hockey Fight, Crowd Violence) with ablation studies proving the necessity of both temporal and spatial features.

4. Experimental Results

The method was evaluated on three datasets: RWF-2000, Hockey Fight, and Crowd Violence.

Performance Metrics:
- RWF-2000: The Random Forest classifier achieved 96.50% accuracy, surpassing SOTA methods like Hachiuma et al. (93.4%) and Zhang et al. (93.3%).
- Hockey Fight: Achieved 98.80% accuracy (Random Forest), outperforming methods like Islam et al. (99.0% is noted as a competitor, but DIFEM is highly competitive) and Tran et al. (71.39%).
- Crowd Violence: Achieved 96.33% accuracy, beating SPIL (94.5%) and Garcia-Cobo et al. (94.30%).
Efficiency:
- Inference Time: Extremely fast, ranging from 0.0232 ms to 0.0533 ms per video depending on the classifier.
- Model Size: The Random Forest model required only 13,202 tree nodes, a fraction of the parameters in 3D CNNs or GCNs.
Ablation Study:
- Using only spatial overlap resulted in lower accuracy (e.g., 73.16% on Crowd dataset) due to false positives from crowd occlusion.
- Using only velocity was insufficient.
- Combining both yielded the best results, confirming that violence requires both rapid movement and physical proximity.

5. Significance and Limitations

Significance:

Paradigm Shift: The paper challenges the assumption that violence detection requires massive deep learning models. It proves that domain-specific heuristics (rapid joint movement + collision) are sufficient for high accuracy.
Real-World Applicability: The low computational cost makes this approach viable for real-time surveillance systems on edge devices where deep learning is impractical.
Interpretability: Unlike "black box" deep learning models, DIFEM features (velocity, overlap) are physically interpretable.

Limitations:

Dependency on Pose Estimation: The method relies entirely on the accuracy of OpenPose. If the pose estimator fails (e.g., due to heavy occlusion or poor lighting), the DIFEM features will be incorrect.
Parameter Overhead: While the classifier is light, the OpenPose model itself adds a significant number of parameters to the overall pipeline. The authors suggest future work on more parameter-efficient skeleton extraction algorithms.
Edge Cases: Qualitative analysis showed failures in scenarios with low movement (violence without rapid motion) or high movement without violence (e.g., large walking strides).

Conclusion

The DIFEM approach successfully demonstrates that for violence recognition, feature engineering based on human dynamics can outperform complex deep learning architectures. By focusing on the physical reality of a fight (speed and collision), the authors created a system that is faster, cheaper, and more accurate than many existing SOTA methods.