💻 computer science

UniSkip-Mamba: A Frequency-Aware State Space Model for Audio-Visual Temporal Forgery Localization

This paper proposes UniSkip-Mamba, a frequency-aware State Space Model that enhances Audio-Visual Temporal Forgery Localization by selectively focusing on discriminative low-to-mid frequency patterns while filtering out high-frequency noise, thereby achieving state-of-the-art accuracy and significantly faster inference.

Original authors: Cangjin Yu, Quan Zhang, Dan Jiang, Ke Zhang

Published 2026-07-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Cangjin Yu, Quan Zhang, Dan Jiang, Ke Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery in a crowded, noisy room. The room is filled with people talking, music playing, and lights flashing. In the world of digital media, this "room" is a video, and the "people" are the pixels and sound waves that make up the story. For a long time, computers have been getting very good at spotting if a video is fake (a "Deepfake"), but they often struggle with a specific, tricky job: finding exactly when the fake part starts and stops. It's like knowing a song is a cover version but not being able to point to the exact second the singer changed their voice. This field, called Audio-Visual Temporal Forgery Localization, is crucial because in real life—like in a courtroom or for social media safety—we need to know the precise boundaries of a lie, not just that a lie exists.

To understand how computers do this, think of a video as a complex song. Just like music, a video has different "frequencies." Some parts are slow, deep, and steady (low frequencies), like the bass drum or a calm conversation. Other parts are fast, sharp, and jittery (high frequencies), like the crackle of a record or a sudden, jerky camera shake. Traditional computer models try to listen to every sound in the song, from the deepest bass to the highest squeak. But here's the problem: in the digital world, those fast, high-pitched sounds are often just static noise or glitches from compression, not the actual clues that reveal a forgery. If a detective tries to solve a case by focusing on the background static, they might get distracted and miss the real evidence.

This is where a new study comes in with a clever twist. The researchers, Cangjin Yu, Quan Zhang, Dan Jiang, and Ke Zhang, built a new kind of digital detective called UniSkip-Mamba. Instead of listening to every single sound in the video, they figured out that the "smoking gun" clues for fake videos are mostly hiding in the slow, steady, low-to-mid-range frequencies. The fast, jittery high-frequency stuff? It's mostly just noise that confuses the computer. So, they designed their system to "skip" over the noisy parts, much like a DJ who knows to ignore the static on a record and focus on the beat.

The paper's main discovery is that by intentionally ignoring the high-frequency noise, the computer actually gets better at its job. They tested this on two huge datasets of fake videos, LAV-DF and AV-Deepfake1M. The results were impressive: their new method, UniSkip-Mamba, didn't just find the fakes; it pinpointed the exact start and end times with much higher accuracy than previous top methods. On one dataset, it improved the accuracy score by nearly 10%, and on the other, by over 14%. Even more surprisingly, by skipping the noise, the system became 6 times faster at making decisions.

The researchers argue against the old way of doing things, where computers try to process every single frame and sound wave in detail. They show that this "dense" approach is actually a weakness because it makes the computer too sensitive to tiny, meaningless glitches. Instead, their "Skip-Scanning" method acts like a smart filter. It groups the video frames together and scans them in a way that naturally tunes out the high-frequency static while keeping the important, low-frequency patterns that reveal the forgery. They also found that combining audio and video in a specific, flexible way—rather than just sticking them side-by-side rigidly—helps the computer catch fakes where the lips and voice are out of sync by even a tiny bit.

In short, this paper suggests that sometimes, to see the truth clearly, you have to stop looking at everything. By building a system that knows exactly which frequencies matter and which ones are just noise, the researchers created a tool that is not only faster and more accurate but also tougher against real-world problems like blurry or compressed videos. It's a reminder that in the age of AI, sometimes the best way to find a lie is to tune out the noise and listen to the rhythm.

Technical Summary: UniSkip-Mamba

Problem Statement

The proliferation of Artificial Intelligence Generated Content (AIGC) has led to sophisticated multimedia manipulation, creating urgent needs for Audio-Visual Temporal Forgery Localization (AV-TFL). While existing deepfake detection methods have advanced, they predominantly rely on binary classification, failing to identify the precise temporal boundaries of forged segments. This limitation hinders practical applications in judicial forensics and content moderation.

Current AV-TFL approaches face two primary challenges:

Inefficient Temporal Modeling: State-of-the-art methods often rely on Transformer backbones, which suffer from quadratic computational complexity ( $O(T^2)$ ), creating bottlenecks for long video sequences.
Indiscriminate Frequency Processing: Existing methods process all frequency components equally. Systematic analysis reveals that forgery-discriminative patterns concentrate in low-to-mid frequency bands (normalized frequency 0–0.15), while high-frequency components (>0.15) primarily introduce noise (e.g., compression artifacts, frame-level variations). Current models overfit this high-frequency noise, leading to poor robustness under real-world data degradation.
Rigid Multimodal Fusion: Most methods use channel-wise concatenation, forcing strict frame-level alignment between audio and visual streams. This prevents the detection of asynchronous artifacts where visual and auditory cues are misaligned by temporal offsets.

Methodology: UniSkip-Mamba

The authors propose UniSkip-Mamba, a frequency-aware State Space Model (SSM) framework designed to address these challenges through three core components:

1. Unified Multimodal Sequence Fusion

Instead of channel-wise concatenation, the framework serializes audio and visual features into a single temporal sequence.

Mechanism: Visual features are projected to match the audio dimension, then concatenated temporally ( $F_{unified} = [F_v, F_a]$ ).
Modality Embeddings: Learnable embeddings are added to explicitly inform the model of each token's origin (audio vs. visual).
Benefit: This transforms cross-modal correlation into a long-range dependency problem, allowing the model to detect asynchronous forgeries with arbitrary temporal offsets, which rigid alignment methods miss.

2. Efficient Skip-Scanning Mamba Blocks

The backbone utilizes the Mamba architecture, which offers linear complexity ( $O(T)$ ), but introduces a novel Skip-Scanning mechanism via a "Group-Scan-Merge" strategy.

Grouping: The input sequence is split into $p$ interleaved groups (where $p$ is the stride).
Parallel Scanning: Each group is processed independently by a Bidirectional Mamba block. This reduces the effective sequence length for the recurrent state update, accelerating inference and reducing sequential dependency depth.
Merging: Processed sub-sequences are interleaved back to restore full temporal resolution.
Frequency-Aware Regularization: By selecting a stride of $p=2$ , the mechanism acts as a structural low-pass filter. It naturally attenuates high-frequency noise (which the authors identify as non-discriminative) while preserving the critical low/mid-frequency patterns (0–0.15) where forgery signals reside.

3. Hierarchical Architecture

The model employs a hierarchical design with a Stem (operating at full resolution to capture fine-grained micro-artifacts) and Branches (progressively downsampling the sequence to capture long-range, coarse-grained dependencies).

Key Contributions

Frequency-Domain Insight: The paper provides systematic evidence that discriminative forgery information is concentrated in the 0–0.15 normalized frequency band, while high-frequency components act as noise. Removing high frequencies can actually improve performance by +1.4%.
UniSkip-Mamba Framework: A novel SSM framework that combines Unified Sequence Representation (for flexible multimodal fusion) with Skip-Scanning Mamba Blocks (for frequency-aware regularization and efficiency).
Theoretical Justification: The authors derive that Skip-Scanning with stride $p=2$ aligns with the Nyquist-Shannon sampling theorem relative to the discriminative frequency band, providing a theoretical basis for why this stride optimizes the trade-off between accuracy, robustness, and speed.
Architectural Synergy: The paper demonstrates that the Unified Sequence representation requires the linear complexity and position-agnostic state propagation of Mamba to succeed; applying this fusion strategy to Transformer backbones results in performance degradation due to quadratic complexity and position encoding mismatches.

Experimental Results

The framework was evaluated on two major benchmarks: LAV-DF and AV-Deepfake1M.

LAV-DF: UniSkip-Mamba (stride=2) achieved 63.4% AP@0.95, a 9.8% absolute improvement over the previous best method (UniCaCLF). It also showed a 2× inference speedup compared to dense scanning.
AV-Deepfake1M: The model achieved 63.58% mAP, a 14.32% improvement over the previous SOTA (DiMoDif). Notably, it showed massive gains at high IoU thresholds (e.g., +23.77% at AP@0.95), indicating superior precision in boundary localization.
Robustness: Under various cross-modal perturbations (noise, compression, blur), Skip-Scanning models (stride 2 and 4) consistently outperformed dense scanning (stride 1) by 5–8% mAP. In extreme degradation scenarios, the stride-4 model maintained 1.4× better resilience than the baseline.
Efficiency: The framework achieves 6× faster inference compared to Transformer-based baselines (UMMAFormer) and scales significantly better with batch size and sequence length.

Significance and Claims

The paper claims that UniSkip-Mamba establishes a new state-of-the-art for AV-TFL by fundamentally rethinking temporal modeling from a frequency-domain perspective.

Principled Regularization: The authors argue that Skip-Scanning is not merely an efficiency trick but a "principled regularization strategy." By aligning the scanning stride with the discriminative frequency band, the model inherently filters noise without losing representational completeness.
Robustness: The frequency-aware design provides provable robustness against real-world data degradation, as high-frequency perturbations (common in compressed or noisy media) are naturally down-weighted.
Efficiency vs. Accuracy: The work demonstrates that linear complexity SSMs can overcome the efficiency bottlenecks of Transformers while simultaneously improving accuracy through better long-range modeling and noise suppression.

The authors conclude that their approach offers a practical, robust solution for large-scale multimedia forensics, though they acknowledge a limitation: extremely short-duration, single-frame glitches residing purely in the high-frequency band might be attenuated by the skip-scanning mechanism.