🤖 AI

CoLoRSMamba: Conditional LoRA-Steered Mamba for Supervised Multimodal Violence Detection

The paper introduces CoLoRSMamba, a directional Video-to-Audio multimodal architecture that leverages CLS-guided conditional LoRA to dynamically adapt AudioMamba's state-space parameters based on visual context, achieving state-of-the-art violence detection accuracy and efficiency on curated audio-filtered NTU-CCTV and DVD datasets.

Original authors: Damith Chamalke Senadeera, Dimitrios Kollias, Gregory Slabaugh

Published 2026-04-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Damith Chamalke Senadeera, Dimitrios Kollias, Gregory Slabaugh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a security guard watching a live feed of a busy city square. Your job is to spot a fight before it gets out of hand.

Sometimes, the video is blurry, people are blocking the view, or the action happens too fast to see clearly. But if you listen closely, you might hear a scream, a crash, or a gunshot that gives you the answer the camera missed.

This paper introduces a new AI system called CoLoRSMamba that acts like a super-smart security guard who knows exactly how to balance what it sees with what it hears.

Here is the breakdown of how it works, using simple analogies:

1. The Problem: The "Noisy Crowd"

In real life, video is usually the most reliable clue. You can see a punch; you can't always hear it over the wind or traffic. However, audio can sometimes be a liar. Imagine a scene where a movie is playing on a loudspeaker nearby. The audio screams "Violence!" but the video shows a peaceful park. If an AI trusts the audio too much, it will make mistakes.

Most old AI systems either:

Ignored the audio (sticking only to the video).
Treated audio and video as equal partners (like two people shouting at each other), which gets confusing when the audio is just background noise.

2. The Solution: The "Conductor and the Orchestra"

The authors built a system where Video is the Conductor and Audio is the Orchestra.

The Conductor (Video): The video part of the AI (called VideoMamba) watches the scene. It knows the context: "Is this a crowded street? Is someone running?"
The Orchestra (Audio): The audio part (called AudioMamba) listens to the sounds.
The Magic Trick: Instead of the audio just shouting its opinion, the Video Conductor holds a baton. At every single moment, the Conductor tells the Audio Orchestra: "Hey, that sound you just heard? It's probably just a car backfiring, so ignore it." OR "That sound? It's a scream, so pay extra attention!"

This is called a Directional Video → Audio design. The video leads, and the audio follows its lead.

3. The Secret Sauce: "Conditional LoRA-Steering"

This is the technical part, but think of it like adjusting the volume knobs on a radio in real-time.

Usually, AI models have fixed "knobs" (parameters) that decide how to process sound. They are set once and stay that way.

CoLoRSMamba changes the rules. It uses a special mechanism (called Conditional LoRA) that lets the Video Conductor reach over and twist the knobs on the Audio system instantly.
If the video sees a calm scene, the Conductor turns the audio sensitivity down so it doesn't get spooked by loud noises.
If the video sees a chaotic scene, the Conductor turns the audio sensitivity up to catch subtle screams.

It's like having a smart assistant who doesn't just listen to the music, but changes the equalizer settings based on what they see happening in the room.

4. The "Audio Filter" (Cleaning the Data)

The researchers realized that to test this fairly, they couldn't use datasets where the audio was missing or completely unrelated (like a documentary with a voiceover).

They went through thousands of videos and filtered out any clip where the audio was silent, missing, or just background music.
They kept only the clips where the sound actually matched what was happening on screen. This ensured they were testing the AI's ability to understand the connection, not just guess.

5. The Results: Smarter and Faster

When they tested CoLoRSMamba:

It got better scores than other AI models that only looked at video or only listened to audio.
It was more efficient. It achieved these high scores with fewer "brain cells" (parameters) and less computing power than massive, heavy models.
It fixed mistakes: In many cases, the video was confused (e.g., "Is that a fight or a game?"), but the audio clarified it ("That's a gunshot!"). The system used the audio to fix the video's confusion.

Summary

CoLoRSMamba is a violence detection system that treats video as the boss and audio as a helpful assistant. Instead of letting the audio shout over the video, it lets the video decide how the audio should be heard. By using a clever "knob-twisting" technique, it adapts to the scene in real-time, making it incredibly accurate at spotting fights in noisy, real-world environments without needing a supercomputer to run it.

1. Problem Statement

Automated violence detection is critical for surveillance and public safety but faces significant challenges in real-world scenarios:

Visual Ambiguity: Violent events are often small, blurred, occluded, or embedded in crowded scenes, making pixel-only analysis unreliable.
Multimodal Asymmetry: While audio (screams, impacts, breaking glass) provides crucial cues, real-world soundscapes are often noisy, silent, or unrelated to the visual scene (e.g., off-screen music). Treating audio and video as equal partners can lead to performance degradation when audio is misleading.
Weak Supervision & Benchmark Flaws: Existing datasets often lack explicit clip-level labels or contain clips with missing/irrelevant audio, penalizing multimodal models unfairly. Most prior work relies on weak supervision (video-level labels) or heavy cross-attention mechanisms that are computationally expensive.

The authors propose a solution that leverages the reliability of visual context to dynamically control the contribution of the audio stream, specifically within the efficient State-Space Model (SSM) architecture (Mamba).

2. Methodology: CoLoRSMamba

The proposed architecture is a directional Video $\to$ Audio multimodal framework built upon VideoMamba and AudioMamba.

Core Architecture

Backbones: The system uses two parallel encoders based on bidirectional Mamba blocks (Selective State-Space Models).
- VideoMamba: Acts as the semantic anchor. It processes the visual stream and produces a CLS token at each layer.
- AudioMamba: Acts as the guided branch. It processes the log-mel spectrogram of the audio.
Conditional LoRA Steering (CoLoRS):
- Instead of standard feature-level fusion or token-level cross-attention, the method applies Conditional Low-Rank Adaptation (LoRA) directly to the operator generators of the AudioMamba branch.
- Mechanism: At each layer $\ell$ $ℓ$ , the normalized CLS token from the VideoMamba branch ( $c^{\ell}_{v,i}$ $c_{v, i}^{ℓ}$ ) generates two conditioning signals:
  1. A channel-wise modulation vector $s(z)$ .
  2. A scalar stabilization gate $g(z)$ .
- These signals dynamically modulate the low-rank factors ( $U, V$ ) of the linear projections that generate the selective SSM parameters ( $\Delta, B, C$ ) in the AudioMamba branch.
- Significance: This allows the visual context to reshape the temporal dynamics of the audio branch (e.g., how quickly the state updates, which frequencies are admitted) on a per-clip basis, without the computational cost of dense cross-attention.

Training Objective

The model is trained using a combination of two losses:

Binary Classification Loss: Standard cross-entropy for violent/non-violent classification.
Symmetric AV-InfoNCE Loss: A contrastive loss that aligns the clip-level embeddings of matched audio and video pairs in a shared space, regularizing the multimodal representation.

Dataset Curation

To ensure fair evaluation, the authors curated audio-filtered subsets of the NTU-CCTV and DVD datasets. They applied a strict protocol to discard clips with missing audio, silence, or off-scene sounds (e.g., dubbed music), ensuring that only clips with scene-relevant audio were used for training and testing.

3. Key Contributions

First Strongly Supervised Multimodal Mamba Detector: Introduces a directional Video $\to$ Audio architecture where VideoMamba anchors the semantics and AudioMamba is guided via conditional updates.
Operator-Level Conditioning: Proposes a novel Conditional LoRA mechanism that conditions the generators of SSM parameters ( $\Delta, B, C$ ) rather than the features themselves. This is the first use of input-conditioned LoRA as a cross-modal steering signal inside a selective state-space model.
Robust Benchmarking: Created audio-filtered subsets of NTU-CCTV and DVD to eliminate bias against the audio modality, enabling a fair assessment of cross-modal learning.
Efficiency: Achieves state-of-the-art performance with fewer parameters and FLOPs compared to larger Transformer-based multimodal models.

4. Experimental Results

The model was evaluated on the curated NTU-CCTV and DVD subsets, outperforming audio-only, video-only, and multimodal baselines (including ResNet, Transformers, and other Mamba variants).

NTU-CCTV Performance:
- Accuracy: 88.63% (vs. 86.44% for the next best, LAVISH).
- F1-V (Violent): 86.24%.
- Macro F1: 88.28%.
DVD Performance:
- Accuracy: 75.77% (vs. 74.40% for LAVISH).
- F1-V: 72.94%.
- Macro F1: 75.51%.
Efficiency Trade-off: CoLoRSMamba (158M parameters, 1,810 GFLOPs) outperforms larger models like LAVISH (238M params) and MBT (174M params) while using significantly fewer computational resources.
Ablation Insights:
- Directionality: Video $\to$ Audio steering significantly outperforms Audio $\to$ Video or criss-cross designs, confirming video as the more reliable anchor.
- Operator vs. Feature: Conditioning the operator (generating $\Delta, B, C$ ) yields higher accuracy than conditioning features (FiLM) or using static LoRA.
- Stabilization Gate: The gating mechanism is crucial for stabilizing the conditioning, preventing the model from being misled by noisy audio.
- Audio Contribution: Analysis shows audio helps correct video-only errors 2.67 times more often than it introduces new errors, particularly in ambiguous visual scenes (e.g., crowds) where audio signatures (gunshots) are decisive.

5. Significance and Conclusion

CoLoRSMamba demonstrates that directional, operator-level conditioning is a superior strategy for multimodal violence detection compared to symmetric fusion or heavy attention mechanisms. By treating video as the primary anchor and using it to dynamically steer the temporal processing of audio, the model effectively handles the "noisy audio" problem inherent in real-world surveillance.

The work highlights two critical lessons for future multimodal safety systems:

Data Curation: Strict filtering of audio quality is essential for fair evaluation and model learning.
Architecture Design: In safety-critical tasks where modalities have unequal reliability, fusion mechanisms should respect this asymmetry (Video $\to$ Audio) and operate at the level of model dynamics (operators) rather than just feature re-weighting.

This approach offers a highly efficient, state-of-the-art solution for deploying violence detection in large-scale camera networks where computational resources and robustness to environmental noise are paramount.