🤖 AI

VideoSEMA: a scalable and efficient Mamba-like attention for video understanding

The paper introduces VideoSEMA, a scalable and efficient video classification model that combines a Mamba-like split space-time attention mechanism with softmax temporal attention, demonstrating superior accuracy and resolution robustness compared to existing Vision Transformer and Mamba models on K400 and SSv2 benchmarks.

Original authors: Nhat Thanh Tran, Fanghui Xue, Shuai Zhang, Jiancheng Lyu, Yunling Zheng, Yingyong Qi, Jack Xin

Published 2026-07-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Nhat Thanh Tran, Fanghui Xue, Shuai Zhang, Jiancheng Lyu, Yunling Zheng, Yingyong Qi, Jack Xin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to understand a movie. You can't just show it a single photo; you have to show it a whole stream of images changing over time. This is the world of video understanding, a branch of computer science where machines learn to recognize actions, like someone peeling a potato or a car turning a corner. For a long time, the best way to do this was to use a giant, hungry brain called a Transformer. Think of a Transformer like a super-organized librarian who reads every single page of every book in the library at once to find connections. It's incredibly smart, but it gets overwhelmed and slow when the library (the video) gets too big or the books (the frames) get too detailed.

To fix this, scientists recently invented a new kind of brain called Mamba. If the Transformer is a librarian reading everything at once, Mamba is a detective who walks down the hallway, remembering clues as they go, which is much faster. However, even Mamba can struggle when the video is high-definition or very long. The big question for researchers is: Can we build a video brain that is as smart as the heavy Transformers but as fast and efficient as the new Mamba detectives? This is exactly the puzzle the paper "VideoSEMA" tries to solve.

The authors of this paper, working with researchers from Qualcomm AI Research and the University of California, Irvine, have built a new model called VideoSEMA. They didn't just throw a new ingredient into the pot; they created a clever split-system. Imagine watching a movie where you have two different ways of paying attention. First, you look at a single frame (a still picture) and use a special "Mamba-like" eye called SEMA. This eye is smart enough to look at a small window of details (like the texture of a potato skin) while also taking a quick, global average of the whole scene, all without getting tired. Then, to understand how the story moves, the model uses a standard "soft" attention to look at how things change from one frame to the next.

The paper shows that this split approach is not just a lucky guess; they actually proved mathematically that under certain conditions, this split method is just as good as looking at the entire video all at once. When they tested VideoSEMA on famous video datasets like K400 (which has 400 types of human actions) and SSv2 (which focuses on tricky, fine-grained movements), the results were impressive. On the K400 dataset, VideoSEMA beat much larger and heavier models, including other Mamba-based models and big Transformers, while using fewer computer resources. For example, at a resolution of 16 × 224², VideoSEMA achieved a 82.4% accuracy, beating the previous Mamba model (VideoMamba) which scored 80.8%, even though VideoSEMA had fewer parameters.

One of the most exciting findings is how VideoSEMA handles high-definition videos. The researchers tested what happens when they fed the model videos with much higher resolutions, scaling up from 224² to 1024² (which is a huge jump in detail), without retraining the model. The old Mamba model (VideoMamba) crashed hard, dropping its accuracy from 80.8% down to 40.8%. In contrast, VideoSEMA held its ground much better, dropping only to 54.6%. This suggests that VideoSEMA is much more robust when the visual details get complicated. They also found that on the SSv2 dataset, VideoSEMA could achieve better results with just 8 frames of video than other models needed 16 frames to achieve, proving it is very efficient at grabbing the right information quickly.

The paper also rules out some ideas. They tested using a simple 3D convolution (a standard way of looking at video) and a pure Mamba block for the time-based part of the video, but found that neither worked as well as using a standard attention mechanism for the time dimension. They showed that while Mamba is great for space, it wasn't the best choice for handling the time dimension in this specific setup. The authors are careful to note that while their model is very promising, it is still a work in progress. They suggest that for even longer videos, they might need to add "sparse" or "dilated" attention in the future to handle the extra time without slowing down. But for now, VideoSEMA stands as a strong, efficient, and scalable new way for computers to understand the moving world around us.

Technical Summary of VideoSEMA: A Scalable and Efficient Mamba-like Attention for Video Understanding

Problem Statement
Understanding complex spatiotemporal patterns in videos is a fundamental challenge in computer vision, critical for applications ranging from action classification to world modeling for autonomous systems. While recent advances leveraging Transformer-based architectures and large foundation models have achieved state-of-the-art performance, they predominantly rely on Vision Transformer (ViT) backbones. These models incur prohibitively high computational costs for large video inputs due to the quadratic complexity of full space-time attention. Although State Space Models (SSMs) like Mamba have been explored to reduce costs for long sequences, and linear attention offers another efficiency pathway, there remains a need for a backbone that efficiently captures both spatial and temporal features without the heavy computational burden of full joint attention or the limitations of existing Mamba implementations in video contexts.

Methodology: VideoSEMA
The authors propose VideoSEMA, a novel video classification model built on a split space-time attention framework. The architecture decouples spatial and temporal processing to achieve scalability and efficiency:

Spatial Processing (SEMA Block): Instead of full global attention or standard Mamba recurrence, the model employs a Mamba-like attention (SEMA) block for spatial dimensions. SEMA mimics the exponential forgetting property of Mamba by combining a local window attention with a global averaging operation. This design allows the model to access global information efficiently while maintaining linear computational complexity relative to the spatial dimension ( $O(N)$ ), avoiding the $O(N^2)$ cost of full softmax attention.
Temporal Processing: The model applies a standard softmax temporal attention across the frame sequence. The authors argue that for moderate video lengths (e.g., 16–32 frames), global softmax attention in the temporal dimension is computationally affordable and effective, obviating the need for linear approximations in time.
Architecture: The model tokenizes input videos using 3D convolutions, appends a learnable classifier token, and adds positional and temporal embeddings. The core processing involves alternating layers of spatial SEMA and temporal attention, followed by hierarchical downsampling via convolutional layers.
Theoretical Equivalence: The paper provides a theoretical proof demonstrating that under specific rank conditions, this split space-time attention is mathematically equivalent to full space-time attention. Specifically, the split attention acts as an exact marginal-conditional decomposition of the full attention matrix, provided the temporal and spatial score matrices satisfy certain rank constraints relative to the model's head dimensions.

Key Contributions

Novel Architecture: Development of the first video model utilizing a Mamba-like scalable and efficient image backbone (SEMA) within a split space-time attention framework.
Theoretical Insight: A proof establishing the equivalence between split space-time attention and full space-time attention under specific rank conditions, validating the theoretical soundness of the decomposition.
Efficiency and Performance: Demonstration that VideoSEMA outperforms heavier Vision Transformer and Mamba-based video models on standard benchmarks while maintaining a significantly lower parameter count and FLOPs.
Resolution Robustness: Evidence that VideoSEMA degrades much more gracefully than VideoMamba when input resolution scales up (from $224^2$ to $1024^2$ ) without fine-tuning.

Experimental Results
The model was evaluated on the Kinetics-400 (K400) and Something-Something v2 (SSv2) datasets:

K400 Performance: VideoSEMA achieves superior Top-1 and Top-5 accuracy compared to models with comparable or larger parameter sizes and FLOPs. For instance, at $16 \times 224^2$ resolution, VideoSEMA (31M parameters) achieves 82.4% Top-1 accuracy, outperforming VideoMamba-S (26M parameters, 80.8%) and significantly exceeding larger models like TimeSformer-L (121M parameters, 80.7%).
SSv2 Performance: VideoSEMA leads in Top-1 accuracy among models of similar parameter sizes. Notably, it outperforms VideoMamba-S by ~2% using only 8 frames and outperforms VideoMAE by 0.5% using 16 frames.
Ablation Studies: Experiments confirm that the combination of SEMA for space and softmax attention for time yields the best trade-off between accuracy and efficiency, outperforming variants using 3D convolutions or Mamba for temporal processing.
Scalability: When tested on higher resolutions ( $512^2$ and $1024^2$ ) without fine-tuning, VideoSEMA maintains robustness, whereas VideoMamba suffers a drastic accuracy drop (e.g., dropping to 40.8% at $1024^2$ compared to VideoSEMA's 54.6%).

Significance and Claims
The paper claims that VideoSEMA represents a significant step forward in efficient video understanding by successfully adapting Mamba-like principles to the spatial domain while retaining effective temporal modeling. The authors emphasize that this is the first work to explore Mamba-like backbones specifically for video applications. The model's ability to outperform larger, more computationally expensive architectures suggests that the split space-time approach, grounded in the theoretical equivalence to full attention, offers a highly efficient alternative for capturing spatiotemporal dependencies. Furthermore, the model's robustness to high-resolution inputs without fine-tuning highlights its potential for scalable deployment in real-world scenarios where video quality varies. The authors suggest future work could extend this framework to longer videos using dilated or sparse temporal attention.

Technical Summary of VideoSEMA: A Scalable and Efficient Mamba-like Attention for Video Understanding

More like this