🤖 AI

FuseMamba-VD: Dual Branch VideoMamba with Gated Class Token Fusion for Violence Detection

The paper proposes FuseMamba-VD, an efficient dual-branch VideoMamba architecture with gated class token fusion that achieves state-of-the-art violence detection performance on multiple benchmarks by effectively balancing accuracy and computational efficiency through a state-space model backbone.

Original authors: Damith Chamalke Senadeera, Muhammad Awais, Shibo Li, Dimitrios Kollias, Gregory Slabaugh

Published 2026-07-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Damith Chamalke Senadeera, Muhammad Awais, Shibo Li, Dimitrios Kollias, Gregory Slabaugh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to spot a fight breaking out in a crowded city square using security camera footage. It's a tough job. If you just look at a single snapshot, you might see two people standing close together, but you can't tell if they are hugging or fighting. If you just watch the movement without looking at the details, you might miss the specific gesture that turns a shove into a punch.

This paper introduces a new computer program called FuseMamba-VD designed to solve this problem. Think of it as a highly trained security guard who uses a special "dual-brain" approach to watch the cameras.

Here is how it works, broken down into simple concepts:

1. The Problem with Old Methods

Previous computer programs tried to do this in two main ways, but both had flaws:

The "Slow" Method (CNNs): These were like taking a photo every few seconds. They were good at seeing details but missed the long story of how a fight started and ended.
The "Expensive" Method (Transformers): These were like trying to read every single word in a library at once to understand a story. They were very smart but required so much computer power that they were slow and expensive to run on real-world cameras.

2. The New Solution: A Two-Person Team

The authors built a system with two separate "brains" (branches) working at the same time, inspired by a new type of efficient technology called Mamba (State-Space Models).

Brain A (The Detail Detective): This branch looks at the video frame-by-frame, focusing on spatial details. It asks: "What do the people look like? Are their fists clenched? How close are they?" It prioritizes the shape and appearance of the scene.
Brain B (The Motion Tracker): This branch looks at the video as a continuous flow, focusing on temporal dynamics. It asks: "How are they moving? Is the speed increasing? Is there a sudden lunge?" It prioritizes the movement and timing.

3. The Secret Sauce: The "Gated Fusion"

Usually, you might wait until the very end to let these two brains talk to each other. But FuseMamba-VD is different. It uses a mechanism called Gated Class Token Fusion (GCTF).

Imagine the two brains are having a conversation every single step of the way as they watch the video.

The "Detail Detective" (Brain A) whispers clues to the "Motion Tracker" (Brain B) constantly.
A special gate (a smart switch) decides how much of that information to let through at any given moment. If the movement is confusing, the gate might say, "Listen more to the details." If the details are blurry, it might say, "Focus on the movement."

This continuous, layer-by-layer conversation allows the system to refine its understanding constantly, rather than waiting until the end to combine notes.

4. The "Crop" Trick

Before the brains even start watching, the system has a Cropping Module. Think of this as a camera operator who zooms in on the people in the video and cuts out the background (like trees, cars, or empty sky). This ensures the computer isn't wasting energy looking at things that aren't people, making it much faster and more accurate at spotting human interactions.

5. The Results: Faster and Smarter

The authors tested this new system on a massive collection of real-world surveillance videos (merging four different datasets into one giant test) and a new dataset called DVD.

Accuracy: It beat all the previous "state-of-the-art" models. It was better at spotting violence than the old heavy-hitters.
Efficiency: This is the big win. The new model is much lighter. It uses less than half the computer power (FLOPs) and has fewer "neurons" (parameters) than its closest competitor, CUE-Net.
Speed: Because it is so efficient, it runs about 4.7 times faster on high-end hardware than the previous best model.

Summary

In short, FuseMamba-VD is a video violence detector that doesn't just "watch" videos; it has two specialized experts who constantly chat with each other while zooming in on the action. This teamwork allows it to spot fights more accurately than before, while using a fraction of the computer power, making it a practical solution for real-world security cameras.

Technical Summary: FuseMamba-VD

Problem Statement

The rapid proliferation of surveillance cameras has created an overwhelming volume of video data, making continuous human monitoring for violence detection impractical and unreliable. While existing deep learning approaches, such as 3D Convolutional Neural Networks (CNNs) and Transformer-based architectures, have shown success in extracting spatio-temporal features, they face significant limitations. 3D CNNs struggle with capturing long-range dependencies, whereas Transformers incur quadratic computational complexity relative to sequence length, limiting their scalability for long, high-resolution surveillance videos. Furthermore, many existing datasets used for training are either staged (e.g., movie scenes) or weakly labeled, failing to represent the complexities of real-world surveillance scenarios.

Methodology

The authors propose FuseMamba-VD, an efficient architecture designed for supervised violence detection that leverages State-Space Models (SSMs). The core innovation lies in a dual-branch design combined with a Gated Class Token Fusion (GCTF) mechanism.

1. Architecture Overview

The model is built upon the VideoMamba backbone, utilizing bidirectional state-space modules. It consists of four primary components:

Cropping Module: To focus computational resources on relevant human interactions, the system extracts regions encompassing all detected people in each frame, based on the observation that violence typically involves multiple individuals.
Dual-Branch Pipeline: Two parallel pipelines process the video input ( $X \in \mathbb{R}^{3 \times T \times H \times W}$ $X \in R^{3 \times T \times H \times W}$ ) after tokenization into spatiotemporal patches:
- Branch-1 (Spatial-First Scanning): Tokens are reorganized to prioritize spatial layout within frames before concatenating across time. This branch is designed to capture fine-grained local spatial cues.
- Branch-2 (Temporal-First Scanning): Tokens maintain their natural temporal order and are stacked along the spatial dimension. This branch focuses on dynamic motion patterns and global temporal dependencies.
- Both branches utilize pre-trained VideoMamba weights and employ bi-directional scanning.

2. Gated Class Token Fusion (GCTF)

A novel continuous fusion mechanism is introduced to integrate information between the two branches. Unlike late-fusion approaches, GCTF operates at every layer of the network:

Each branch produces a Class Token ($CLS$) at every layer.
A learnable gating mechanism adaptively merges the spatial branch's $CLS$ token ( $CLS^1$ ) into the temporal branch's $CLS$ token ( $CLS^2$ ).
The fusion is defined as: $CLS^2_{fused} = \sigma' \odot CLS^2 + (1 - \sigma') \odot CLS^1$ , where $\sigma'$ is a sigmoid-activated learnable parameter.
This allows for progressive refinement of spatial and temporal representations without overcommitting to either modality early in the network.

3. Final Fusion

At the final layer, the $CLS$ tokens from both branches are concatenated and passed through a fully connected projection layer to produce the final classification probability.

Key Contributions

The paper outlines three primary contributions:

Dual-Branch State-Space Architecture: The first state-space-based dual-branch design for supervised video violence detection. It explicitly decouples spatial and temporal reasoning while enabling continuous semantic interaction via class tokens, contrasting with prior models that rely on late fusion or heavy attention mechanisms.
Gated Class Token Fusion (GCTF): A continuous, layer-wise fusion mechanism that adaptively integrates branch-specific class tokens. This enables gradual alignment of features and prevents early overcommitment to a single branch.
Integrated Surveillance Benchmark and Efficiency: The authors curated a new benchmark by merging four strongly labeled open-source datasets (RWF-2000, RLVS, SURV, and VioPeru) with strict controls to prevent data leakage between training and testing sets. They demonstrate that FuseMamba-VD achieves state-of-the-art performance while substantially reducing parameters and FLOPs compared to prior work.

Experimental Results

The model was evaluated on the newly constructed combined benchmark and the recently introduced DVD dataset.

Combined Benchmark: FuseMamba-VD achieved a Top-1 accuracy of 95.85%, outperforming baselines including SlowFast, VideoSwin, UniFormer-V2, CUE-Net, and various VideoMamba variants. It achieved F1 scores of 95.89% (violent) and 95.81% (non-violent).
DVD Dataset: On this more challenging dataset with real-world variability and class imbalance, the model reached 74.13% Top-1 accuracy, surpassing all baselines (which scored $\le$ 71%).
Efficiency: Despite outperforming CUE-Net (the closest competitor in accuracy), FuseMamba-VD uses significantly fewer resources:
- Parameters: 154.3M vs. 354M (CUE-Net).
- FLOPs: 1912 GFLOPs vs. 5906 GFLOPs.
- Inference Speed: On an NVIDIA H100 GPU, FuseMamba-VD achieves 343.53 FPS, a $\approx$ 4.7 $\times$ speed-up over CUE-Net (72.64 FPS).
Ablation Studies: Experiments confirmed that the dual-branch design is superior to single-branch variants. Furthermore, continuous lateral connections at every block outperformed configurations where connections were applied only at specific layers (beginning, middle, or end).

Significance and Claims

The paper claims that FuseMamba-VD offers an optimal balance between accuracy and computational efficiency, demonstrating the promise of State-Space Models (SSMs) for scalable, resource-efficient video violence detection. By effectively separating spatial and temporal reasoning while maintaining continuous interaction through a gating mechanism, the model improves sensitivity to subtle motion differences indicative of violence. The authors position this work as a step toward practical, real-world surveillance applications where both high accuracy and low computational cost are critical. The code and pre-trained models are made available to the community.